推理框架SGLang安装与调试

SGLang的安装和调试

SGLang是一个用于大型语言模型和视觉语言模型的快速服务框架。
SGLang Docs
SGLang源码安装手册

我的环境

Linux + conda + cuda12.6 + python3.12 + 4* GeForce RTX 3090

安装

（安装torch2.5是为了和sglang对齐，cuda向后兼容）

git clone git@github.com:gitover22/sglang.git
cd sglang
conda create -n sglang
conda activate sglang
conda install python=3.12
pip install --upgrade pip
conda install pytorch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 pytorch-cuda=12.4 -c pytorch -c nvidia 
pip install -e "python[all]" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

调试

程序路径：examples/frontend_language/quick_start/local_example_chat.py
配置launch.json:

{"configurations": [{"name": "Python Debugger: Current File","type": "debugpy","request": "launch","program": "${file}","console": "integratedTerminal"}]
}

可以愉快的进行debug啦！

重构代码后执行以下命令重新安装sglang

pip install -e "python[all]"