跳到主要内容

评估提示

结合 MLflow 提示词注册表 (Prompt Registry) 与 MLflow LLM 评估 (LLM Evaluation)，您可以跨不同模型和数据集评估提示词性能，并将评估结果记录在中心化注册表中。您还可以通过评估过程中记录的追踪 (traces) 来检查模型输出，从而深入了解模型对不同提示词的响应方式。

MLflow 提示词评估的主要优势
  • 高效评估：MLflow 的 LLM 评估 API 提供了一种简单且一致的方法来跨不同模型和数据集评估提示词，无需编写样板代码。
  • 比较结果：在 MLflow UI 中轻松比较评估结果。
  • 追踪结果：在 MLflow 实验中跟踪评估结果，以维护提示词性能和不同评估设置的历史记录。
  • 追踪 (Tracing)：通过评估期间生成的追踪信息，深入检查推理过程中的模型行为。

快速入门​

1. 安装所需库​

首先安装 MLflow 和 OpenAI SDK。如果您使用其他 LLM 提供商，请安装相应的 SDK。

bash
pip install 'mlflow>=3.0.0' openai -qU

同时设置 OpenAI API 密钥（或其他 LLM 提供商，如 Anthropic）。

python
import os
from getpass import getpass

os.environ["OPENAI_API_KEY"] = getpass("Enter your OpenAI API key: ")

1. 创建提示词​

Create Prompt UI

  1. 在终端运行 mlflow server 以启动 MLflow UI。
  2. 导航至 MLflow UI 中的 Prompts（提示词）选项卡。
  3. 点击 Create Prompt（创建提示词）按钮。
  4. 填写提示词详细信息，例如名称、提示词模板文本和提交消息（可选）。
  5. 点击 Create（创建）以注册该提示词。

2. 准备评估数据​

下面我们创建一个小型摘要数据集用于演示。

python
import pandas as pd

eval_data = [
    {
        "inputs": {
            "sentences": "Artificial intelligence has transformed how businesses operate in the 21st century. Companies are leveraging AI for everything from customer service to supply chain optimization. The technology enables automation of routine tasks, freeing human workers for more creative endeavors. However, concerns about job displacement and ethical implications remain significant. Many experts argue that AI will ultimately create more jobs than it eliminates, though the transition may be challenging.",
        },
        "expectations": {
            "summary": "AI has revolutionized business operations through automation and optimization, though ethical concerns about job displacement persist alongside predictions that AI will ultimately create more employment opportunities than it eliminates.",
        },
    },
    {
        "inputs": {
            "sentences": "Climate change continues to affect ecosystems worldwide at an alarming rate. Rising global temperatures have led to more frequent extreme weather events including hurricanes, floods, and wildfires. Polar ice caps are melting faster than predicted, contributing to sea level rise that threatens coastal communities. Scientists warn that without immediate and dramatic reductions in greenhouse gas emissions, many of these changes may become irreversible. International cooperation remains essential but politically challenging.",
        },
        "expectations": {
            "summary": "Climate change is causing accelerating environmental damage through extreme weather events and melting ice caps, with scientists warning that without immediate reduction in greenhouse gas emissions, many changes may become irreversible.",
        },
    },
    {
        "inputs": {
            "sentences": "The human genome project was completed in 2003 after 13 years of international collaborative research. It successfully mapped all of the genes of the human genome, approximately 20,000-25,000 genes in total. The project cost nearly $3 billion but has enabled countless medical advances and spawned new fields like pharmacogenomics. The knowledge gained has dramatically improved our understanding of genetic diseases and opened pathways to personalized medicine. Today, a complete human genome can be sequenced in under a day for about $1,000.",
        },
        "expectations": {
            "summary": "The Human Genome Project, completed in 2003, mapped approximately 20,000-25,000 human genes at a cost of $3 billion, enabling medical advances, improving understanding of genetic diseases, and establishing the foundation for personalized medicine.",
        },
    },
    {
        "inputs": {
            "sentences": "Remote work adoption accelerated dramatically during the COVID-19 pandemic. Organizations that had previously resisted flexible work arrangements were forced to implement digital collaboration tools and virtual workflows. Many companies reported surprising productivity gains, though concerns about company culture and collaboration persisted. After the pandemic, a hybrid model emerged as the preferred approach for many businesses, combining in-office and remote work. This shift has profound implications for urban planning, commercial real estate, and work-life balance.",
        },
        "expectations": {
            "summary": "The COVID-19 pandemic forced widespread adoption of remote work, revealing unexpected productivity benefits despite collaboration challenges, and resulting in a hybrid work model that impacts urban planning, real estate, and work-life balance.",
        },
    },
]

3. 定义预测函数​

定义一个接收输入 DataFrame 并返回预测列表的函数。

MLflow 会将输入列（本示例中仅为 inputs）传递给函数。输出字符串将与 targets 列进行比较，以评估模型。

python
import mlflow
import openai


def predict_fn(sentences: str) -> str:
    # Load the latest version of the registered prompt
    prompt = mlflow.genai.load_prompt("prompts:/summarization-prompt@latest")
    completion = openai.OpenAI().chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": prompt.format(sentences=sentences, num_sentences=1),
            }
        ],
    )
    return completion.choices[0].message.content

4. 运行评估​

运行 mlflow.genai.evaluate() API，利用准备好的数据和提示词来评估模型。在本例中，我们将使用以下两个内置指标。

python
from typing import Literal
from mlflow.genai.judges import make_judge

answer_similarity = make_judge(
    name="answer_similarity",
    instructions=(
        "Evaluated on the degree of semantic similarity of the provided output to the expected answer.\n\n"
        "Output: {{ outputs }}\n\n"
        "Expected: {{ expectations }}"
        "Return 'yes' if the output is similar to the expected answer, otherwise return 'no'."
    ),
    model="openai:/gpt-5-mini",
    feedback_value_type=Literal["yes", "no"],
)

results = mlflow.genai.evaluate(
    data=eval_data,
    predict_fn=predict_fn,
    scorers=[answer_similarity],
)

5. 查看结果​

您可以在 MLflow UI 中查看评估结果。导航至 Experiments（实验）选项卡，选择 Evaluations（评估）选项卡，然后点击评估运行记录以查看评估结果。

Evaluation Results