VLLM 格式化LLM输出
文章目录
- 前言
- guided_json
- guided_choice
- guided_regex
- guided_grammar
- 总结
前言
vllm OpenAI Compatible Server 提供了格式化LLM输出的能力,默认的格式化解码后端应该是outlines
目前提供了四个参数来控制格式化输出,分别是:
guided_json: 按照给定的json schema输出
guided_choice: 从给定的选项里面选一个
guided_regex: 按照给定的正则表达式输出
guided_grammar: 按照给定的 扩展巴科斯范式(EBNF)格式 的上下文无关语法输出(我也不懂)
下面我们直接看看如何使用这四个参数,控制LLM的输出
import json
from openai import OpenAIdef chatgpt_base(system_prompt, user_prompt):api_key = "empty"base_url = "http://localhost:8000/v1"model = "Qwen1.5-14B-Chat-AWQ"client = OpenAI(api_key=api_key, base_url=base_url)completion = client.chat.completions.create(model=model,temperature=0,messages=[{"role": "system", "content": system_prompt},{"role": "user", "content": user_prompt},],)return completion.choices[0].message.content
guided_json
from pydantic import BaseModelclass Topic(BaseModel):问题: str答案: strdef chatgpt_guide_json(system_prompt, user_prompt):api_key = "empty"base_url = "http://localhost:8000/v1"model = "Qwen1.5-14B-Chat-AWQ"client = OpenAI(api_key=api_key, base_url=base_url)completion = client.chat.completions.create(model=model,temperature=0,messages=[{"role": "system", "content": system_prompt},{"role": "user", "content": user_prompt},],extra_body={"guided_json": Topic.model_json_schema()},)return completion.choices[0].message.contentsystem_prompt = "You are a helpful assistant."
user_prompt = """
请你生成一对和python相关的问题和答案
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_json(system_prompt, user_prompt)
print("guide json reponse: ", guide_reponse)
输出:
base response: 问题:如何在Python中安装一个新的库?答案:在Python中,你可以使用pip工具来安装新的库。首先,你需要确保pip已安装。然后,打开命令行或终端,输入以下命令来安装所需的库:```pip install 库名```例如,如果你想安装requests库,你可以输入:```pip install requests```这将从Python Package Index (PyPI)下载并安装requests库及其依赖项。
--------------------
guide json reponse: { "问题": "如何在Python中安装一个新的库?", "答案": "在Python中,你可以使用pip工具来安装新的库。例如,如果你想安装requests库,你可以在命令行中输入:\">> pip install requests\"。这将会从Python Package Index (PyPI) 下载并安装requests库。" }
可以看到,即使我们不显式地在prompt中告诉LLM要返回JSON格式,我们拿到的响应竟还是JSON,并且符合我们给的格式。
我们还能给每个字段添加解释,如:
class Topic(BaseModel):问题: str = Field(description="问题")答案: str = Field(description="答案")
guided_choice
def chatgpt_guide_choice(system_prompt, user_prompt):api_key = "empty"base_url = "http://localhost:8000/v1"model = "Qwen1.5-14B-Chat-AWQ"client = OpenAI(api_key=api_key, base_url=base_url)completion = client.chat.completions.create(model=model,temperature=0,messages=[{"role": "system", "content": system_prompt},{"role": "user", "content": user_prompt},],extra_body={"guided_choice": ["Positive", "Negative"]},)return completion.choices[0].message.contentsystem_prompt = "You are a helpful assistant."
user_prompt = """
Is the following review positive or negative?Review: 今天天气真不错,好想出去吃大餐
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_choice(system_prompt, user_prompt)
print("guide choice reponse: ", guide_reponse)
输出:
base response: 这条评论并不是在评价某个产品或服务,而是在描述天气并表达了想出去吃大餐的愿望。不过,如果要从情感色彩来看,这条评论是积极的,因为它提到了好天气,并且表达了积极的愿望。
--------------------
guide choice reponse: Positive
注意:guide_choice 无法输出选项中的多个答案,即无法处理多标签任务
guided_regex
def chatgpt_guide_regex(system_prompt, user_prompt):api_key = "empty"base_url = "http://localhost:8000/v1"model = "Qwen1.5-14B-Chat-AWQ"client = OpenAI(api_key=api_key, base_url=base_url)completion = client.chat.completions.create(model=model,temperature=0,messages=[{"role": "system", "content": system_prompt},{"role": "user", "content": user_prompt},],extra_body={"guided_regex": r"((25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)"},)return completion.choices[0].message.contentsystem_prompt = "You are a helpful assistant."
user_prompt = """
What is the IP address of the Google DNS servers?
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_regex(system_prompt, user_prompt)
print("guide regex reponse: ", guide_reponse)
输出:
base response: The IP addresses for the Google Public DNS servers are as follows:- 8.8.8.8 (primary server)
- 8.8.4.4 (secondary server)These can be used as your DNS servers to take advantage of Google's DNS service.
--------------------
guide regex reponse: 1.8.8.88
从这个输出结果来看,使用格式化输出似乎会导致LLM效果下降?
guided_grammar
arithmetic_grammar = r"""?start: expression?expression: term (("+" | "-") term)*?term: factor (("*" | "/") factor)*?factor: NUMBER| "-" factor| "(" expression ")"%import common.NUMBER
"""def chatgpt_guide_grammar(system_prompt, user_prompt):api_key = "empty"base_url = "http://localhost:8000/v1"model = "Qwen1.5-14B-Chat-AWQ"client = OpenAI(api_key=api_key, base_url=base_url)completion = client.chat.completions.create(model=model,temperature=0,messages=[{"role": "system", "content": system_prompt},{"role": "user", "content": user_prompt},],extra_body={"guided_grammar": arithmetic_grammar},)return completion.choices[0].message.contentsystem_prompt = "You are a helpful assistant."
user_prompt = """
Alice had 4 apples and Bob ate 2. Write an expression for Alice's apples:
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_grammar(system_prompt, user_prompt)
print("guide grammar reponse: ", guide_reponse)
输出:
base response: If Alice originally had 4 apples and Bob ate 2 of them, the expression for the number of apples Alice has left would be:\[ 4 - 2 \]So, Alice now has:\[ 4 - 2 = 2 \]Therefore, the expression for the number of apples Alice has left is \( 4 - 2 \).
--------------------
guide grammar reponse: (4-2)
这个咱也不懂,就不乱讲了,各位同学可以自行探索
感兴趣的同学可以看看:EBNF
总结
输出JSON还可以通过 response_format
控制,具体介绍可以查看vllm官方文档
这几个例子,也可以通过 outlines 仓库学习具体的用法