当前位置：首页 > news >正文

VLLM 格式化LLM输出

news 2024/11/28 10:12:39

文章目录

- 前言
- guided_json
- guided_choice
- guided_regex
- guided_grammar
- 总结

前言

vllm OpenAI Compatible Server 提供了格式化LLM输出的能力，默认的格式化解码后端应该是outlines

目前提供了四个参数来控制格式化输出，分别是：

guided_json: 按照给定的json schema输出
guided_choice: 从给定的选项里面选一个
guided_regex: 按照给定的正则表达式输出
guided_grammar: 按照给定的 扩展巴科斯范式（EBNF）格式 的上下文无关语法输出（我也不懂）

下面我们直接看看如何使用这四个参数，控制LLM的输出

import json
from openai import OpenAIdef chatgpt_base(system_prompt, user_prompt):api_key = "empty"base_url = "http://localhost:8000/v1"model = "Qwen1.5-14B-Chat-AWQ"client = OpenAI(api_key=api_key, base_url=base_url)completion = client.chat.completions.create(model=model,temperature=0,messages=[{"role": "system", "content": system_prompt},{"role": "user", "content": user_prompt},],)return completion.choices[0].message.content

guided_json

from pydantic import BaseModelclass Topic(BaseModel):问题: str答案: strdef chatgpt_guide_json(system_prompt, user_prompt):api_key = "empty"base_url = "http://localhost:8000/v1"model = "Qwen1.5-14B-Chat-AWQ"client = OpenAI(api_key=api_key, base_url=base_url)completion = client.chat.completions.create(model=model,temperature=0,messages=[{"role": "system", "content": system_prompt},{"role": "user", "content": user_prompt},],extra_body={"guided_json": Topic.model_json_schema()},)return completion.choices[0].message.contentsystem_prompt = "You are a helpful assistant."
user_prompt = """
请你生成一对和python相关的问题和答案
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_json(system_prompt, user_prompt)
print("guide json reponse: ", guide_reponse)

输出：

base response:  问题：如何在Python中安装一个新的库？答案：在Python中，你可以使用pip工具来安装新的库。首先，你需要确保pip已安装。然后，打开命令行或终端，输入以下命令来安装所需的库：```pip install 库名```例如，如果你想安装requests库，你可以输入：```pip install requests```这将从Python Package Index (PyPI)下载并安装requests库及其依赖项。
--------------------
guide json reponse:  { "问题": "如何在Python中安装一个新的库？", "答案": "在Python中，你可以使用pip工具来安装新的库。例如，如果你想安装requests库，你可以在命令行中输入：\">> pip install requests\"。这将会从Python Package Index (PyPI) 下载并安装requests库。" }

可以看到，即使我们不显式地在prompt中告诉LLM要返回JSON格式，我们拿到的响应竟还是JSON，并且符合我们给的格式。

我们还能给每个字段添加解释，如：

class Topic(BaseModel):问题: str = Field(description="问题")答案: str = Field(description="答案")

guided_choice

def chatgpt_guide_choice(system_prompt, user_prompt):api_key = "empty"base_url = "http://localhost:8000/v1"model = "Qwen1.5-14B-Chat-AWQ"client = OpenAI(api_key=api_key, base_url=base_url)completion = client.chat.completions.create(model=model,temperature=0,messages=[{"role": "system", "content": system_prompt},{"role": "user", "content": user_prompt},],extra_body={"guided_choice": ["Positive", "Negative"]},)return completion.choices[0].message.contentsystem_prompt = "You are a helpful assistant."
user_prompt = """
Is the following review positive or negative?Review: 今天天气真不错，好想出去吃大餐
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_choice(system_prompt, user_prompt)
print("guide choice reponse: ", guide_reponse)

输出：

base response:  这条评论并不是在评价某个产品或服务，而是在描述天气并表达了想出去吃大餐的愿望。不过，如果要从情感色彩来看，这条评论是积极的，因为它提到了好天气，并且表达了积极的愿望。
--------------------
guide choice reponse:  Positive

注意：guide_choice 无法输出选项中的多个答案，即无法处理多标签任务

guided_regex

def chatgpt_guide_regex(system_prompt, user_prompt):api_key = "empty"base_url = "http://localhost:8000/v1"model = "Qwen1.5-14B-Chat-AWQ"client = OpenAI(api_key=api_key, base_url=base_url)completion = client.chat.completions.create(model=model,temperature=0,messages=[{"role": "system", "content": system_prompt},{"role": "user", "content": user_prompt},],extra_body={"guided_regex": r"((25[0-5]|2[0-4]\d|[01]?\d\d?)\.){3}(25[0-5]|2[0-4]\d|[01]?\d\d?)"},)return completion.choices[0].message.contentsystem_prompt = "You are a helpful assistant."
user_prompt = """
What is the IP address of the Google DNS servers? 
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_regex(system_prompt, user_prompt)
print("guide regex reponse: ", guide_reponse)

输出：

base response:  The IP addresses for the Google Public DNS servers are as follows:- 8.8.8.8 (primary server)
- 8.8.4.4 (secondary server)These can be used as your DNS servers to take advantage of Google's DNS service.
--------------------
guide regex reponse:  1.8.8.88

从这个输出结果来看，使用格式化输出似乎会导致LLM效果下降？

guided_grammar

arithmetic_grammar = r"""?start: expression?expression: term (("+" | "-") term)*?term: factor (("*" | "/") factor)*?factor: NUMBER| "-" factor| "(" expression ")"%import common.NUMBER
"""def chatgpt_guide_grammar(system_prompt, user_prompt):api_key = "empty"base_url = "http://localhost:8000/v1"model = "Qwen1.5-14B-Chat-AWQ"client = OpenAI(api_key=api_key, base_url=base_url)completion = client.chat.completions.create(model=model,temperature=0,messages=[{"role": "system", "content": system_prompt},{"role": "user", "content": user_prompt},],extra_body={"guided_grammar": arithmetic_grammar},)return completion.choices[0].message.contentsystem_prompt = "You are a helpful assistant."
user_prompt = """
Alice had 4 apples and Bob ate 2. Write an expression for Alice's apples:
"""
response = chatgpt_base(system_prompt, user_prompt)
print("base response: ", response)
print("----" * 5)
guide_reponse = chatgpt_guide_grammar(system_prompt, user_prompt)
print("guide grammar reponse: ", guide_reponse)

输出：

base response:  If Alice originally had 4 apples and Bob ate 2 of them, the expression for the number of apples Alice has left would be:\[ 4 - 2 \]So, Alice now has:\[ 4 - 2 = 2 \]Therefore, the expression for the number of apples Alice has left is \( 4 - 2 \).
--------------------
guide grammar reponse: (4-2)