当前位置：首页 > news >正文

使用Facebook Messenger数据进行AI模型微调的完整指南

news 2025/1/16 16:10:42

引言

在这个数字化时代，个性化的AI助手已经成为热门话题。而微调模型是实现这一目标的核心方法之一。本文将介绍如何从Facebook Messenger中提取对话数据，并使用这些数据对AI模型进行微调，从而创建更符合个人需求的AI助手。

主要内容

1. 下载数据

要开始使用，首先需要下载Facebook Messenger的数据。可以按照此处说明下载数据，确保选择JSON格式。

我们提供了一个示例数据集，可以通过以下代码下载和解压：

import zipfile
import requestsdef download_and_unzip(url: str, output_path: str = "file.zip") -> None:file_id = url.split("/")[-2]download_url = f"https://drive.google.com/uc?export=download&id={file_id}"response = requests.get(download_url)if response.status_code != 200:print("Failed to download the file.")returnwith open(output_path, "wb") as file:file.write(response.content)print(f"File {output_path} downloaded.")with zipfile.ZipFile(output_path, "r") as zip_ref:zip_ref.extractall()print(f"File {output_path} has been unzipped.")url = "https://drive.google.com/file/d/1rh1s1o2i7B-Sk1v9o8KNgivLVGwJ-osV/view?usp=sharing"
download_and_unzip(url)

2. 创建聊天加载器

我们可以使用FolderFacebookMessengerChatLoader或SingleFileFacebookMessengerChatLoader加载数据：

from langchain_community.chat_loaders.facebook_messenger import (FolderFacebookMessengerChatLoader,SingleFileFacebookMessengerChatLoader,
)loader = SingleFileFacebookMessengerChatLoader(path="./hogwarts/inbox/HermioneGranger/messages_Hermione_Granger.json",
)chat_session = loader.load()[0]
print(chat_session["messages"][:3])

3. 准备微调

合并消息并将指定发送者的消息转换为“AIMessage”类：

from langchain_community.chat_loaders.utils import (map_ai_messages,merge_chat_runs,
)merged_sessions = merge_chat_runs(chat_sessions)
alternating_sessions = list(map_ai_messages(merged_sessions, "Harry Potter"))from langchain_community.adapters.openai import convert_messages_for_finetuning
training_data = convert_messages_for_finetuning(alternating_sessions)
print(f"Prepared {len(training_data)} dialogues for training")

4. 微调模型

确保安装了openai库并设置了OPENAI_API_KEY：

import json
import time
from io import BytesIO
import openaimy_file = BytesIO()
for m in training_examples:my_file.write((json.dumps({"messages": m}) + "\n").encode("utf-8"))my_file.seek(0)
training_file = openai.files.create(file=my_file, purpose="fine-tune")status = openai.files.retrieve(training_file.id).status
start_time = time.time()
while status != "processed":print(f"Status=[{status}]... {time.time() - start_time:.2f}s", end="\r", flush=True)time.sleep(5)status = openai.files.retrieve(training_file.id).status
print(f"File {training_file.id} ready after {time.time() - start_time:.2f} seconds.")

5. 在LangChain中使用

使用生成的模型ID直接在ChatOpenAI类中：

from langchain_openai import ChatOpenAImodel = ChatOpenAI(model=job.fine_tuned_model,temperature=1,
)from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplateprompt = ChatPromptTemplate.from_messages([("human", "{input}"),]
)chain = prompt | model | StrOutputParser()