vLLM 部署 Qwen3

参考链接：https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#pre-built-wheels

CUDA：12.2

显存：40GB

Python 包管理：conda

LLM：Qwen3-8B

1）创建 conda 环境

# 创建 conda 虚拟环境，环境名称为 vllm，python 的版本为 3.10
conda create -n vllm python=3.10

2）切换 vllm 环境

conda activate vllm

3）安装 vllm

pip install -U vllm \
    --pre \
    --extra-index-url https://wheels.vllm.ai/nightly

参考链接：https://qwen.readthedocs.io/zh-cn/latest/deployment/vllm.html#

vllm serve Qwen/Qwen3-8B

curl

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-8B",
  "messages": [
    {"role": "user", "content": "现在你的身份是刘备，而我是关羽，请在这个背景下完成对话。大哥，我等何日光复大汉"}
  ],
  "temperature": 0.6,
  "top_p": 0.95,
  "top_k": 20,
  "max_tokens": 32768
}'

python

from openai import OpenAI
# Set OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_response = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[
        {"role": "user", "content": "现在你的身份是刘备，而我是关羽，请在这个背景下完成对话。大哥，我等何日光复大汉"},
    ],
    max_tokens=32768,
    temperature=0.6,
    top_p=0.95,
    extra_body={
        "top_k": 20,
    },
)
print("Chat response:", chat_response)