CrewAI 跟踪

CrewAI 是一个开源框架，用于编排角色扮演的自主 AI 代理。

MLflow Tracing 为 CrewAI（一个用于构建多代理应用程序的开源框架）提供了自动跟踪功能。通过调用 mlflow.crewai.autolog() 函数为 CrewAI 启用自动跟踪，MLflow 将捕获 CrewAI 工作流执行的嵌套跟踪，并将其记录到活动的 MLflow 实验中。

import mlflow

mlflow.crewai.autolog()

MLflow trace 会自动捕获有关 CrewAI 代理的以下信息：

• 执行每个任务的任务和代理
• 每次 LLM 调用及其输入提示、完成响应和各种元数据
• 内存加载和写入操作
• 每个操作的延迟
• 如果抛出任何异常

注意

目前，MLflow CrewAI 集成仅支持同步任务执行的跟踪。目前不支持异步任务和 kickoff。

示例用法

首先，为 CrewAI 启用自动跟踪，并可选地创建一个 MLflow 实验来写入跟踪。这有助于更好地组织您的跟踪。

import mlflow

# Turn on auto tracing by calling mlflow.crewai.autolog()
mlflow.crewai.autolog()


# Optional: Set a tracking URI and an experiment
mlflow.set_tracking_uri("https://:5000")
mlflow.set_experiment("CrewAI")

接下来，使用 CrewAI 定义一个多代理工作流。下面的示例定义了一个使用 Web 搜索功能作为工具的旅行规划代理。

from crewai import Agent, Crew, Task
from crewai.knowledge.source.string_knowledge_source import StringKnowledgeSource
from crewai_tools import SerperDevTool, WebsiteSearchTool

from textwrap import dedent

content = "Users name is John. He is 30 years old and lives in San Francisco."
string_source = StringKnowledgeSource(
    content=content, metadata={"preference": "personal"}
)

search_tool = WebsiteSearchTool()


class TripAgents:
    def city_selection_agent(self):
        return Agent(
            role="City Selection Expert",
            goal="Select the best city based on weather, season, and prices",
            backstory="An expert in analyzing travel data to pick ideal destinations",
            tools=[
                search_tool,
            ],
            verbose=True,
        )

    def local_expert(self):
        return Agent(
            role="Local Expert at this city",
            goal="Provide the BEST insights about the selected city",
            backstory="""A knowledgeable local guide with extensive information
        about the city, it's attractions and customs""",
            tools=[search_tool],
            verbose=True,
        )


class TripTasks:
    def identify_task(self, agent, origin, cities, interests, range):
        return Task(
            description=dedent(
                f"""
                Analyze and select the best city for the trip based
                on specific criteria such as weather patterns, seasonal
                events, and travel costs. This task involves comparing
                multiple cities, considering factors like current weather
                conditions, upcoming cultural or seasonal events, and
                overall travel expenses.
                Your final answer must be a detailed
                report on the chosen city, and everything you found out
                about it, including the actual flight costs, weather
                forecast and attractions.

                Traveling from: {origin}
                City Options: {cities}
                Trip Date: {range}
                Traveler Interests: {interests}
            """
            ),
            agent=agent,
            expected_output="Detailed report on the chosen city including flight costs, weather forecast, and attractions",
        )

    def gather_task(self, agent, origin, interests, range):
        return Task(
            description=dedent(
                f"""
                As a local expert on this city you must compile an
                in-depth guide for someone traveling there and wanting
                to have THE BEST trip ever!
                Gather information about key attractions, local customs,
                special events, and daily activity recommendations.
                Find the best spots to go to, the kind of place only a
                local would know.
                This guide should provide a thorough overview of what
                the city has to offer, including hidden gems, cultural
                hotspots, must-visit landmarks, weather forecasts, and
                high level costs.
                The final answer must be a comprehensive city guide,
                rich in cultural insights and practical tips,
                tailored to enhance the travel experience.

                Trip Date: {range}
                Traveling from: {origin}
                Traveler Interests: {interests}
            """
            ),
            agent=agent,
            expected_output="Comprehensive city guide including hidden gems, cultural hotspots, and practical travel tips",
        )


class TripCrew:
    def __init__(self, origin, cities, date_range, interests):
        self.cities = cities
        self.origin = origin
        self.interests = interests
        self.date_range = date_range

    def run(self):
        agents = TripAgents()
        tasks = TripTasks()

        city_selector_agent = agents.city_selection_agent()
        local_expert_agent = agents.local_expert()

        identify_task = tasks.identify_task(
            city_selector_agent,
            self.origin,
            self.cities,
            self.interests,
            self.date_range,
        )
        gather_task = tasks.gather_task(
            local_expert_agent, self.origin, self.interests, self.date_range
        )

        crew = Crew(
            agents=[city_selector_agent, local_expert_agent],
            tasks=[identify_task, gather_task],
            verbose=True,
            memory=True,
            knowledge={
                "sources": [string_source],
                "metadata": {"preference": "personal"},
            },
        )

        result = crew.kickoff()
        return result


trip_crew = TripCrew("California", "Tokyo", "Dec 12 - Dec 20", "sports")
result = trip_crew.run()

Token 用量

MLflow >= 3.5.0 支持 CrewAI 的 token 使用情况跟踪。每次 LLM 调用的 token 使用情况将记录在 mlflow.chat.tokenUsage 属性中。跟踪中的总 token 使用情况可在跟踪信息对象的 token_usage 字段中找到。

import json
import mlflow

mlflow.crewai.autolog()

# Run the tool calling agent defined in the previous section
trip_crew = TripCrew("California", "Tokyo", "Dec 12 - Dec 20", "sports")
result = trip_crew.run()

# Get the trace object just created
last_trace_id = mlflow.get_last_active_trace_id()
trace = mlflow.get_trace(trace_id=last_trace_id)

# Print the token usage
total_usage = trace.info.token_usage
print("== Total token usage: ==")
print(f"  Input tokens: {total_usage['input_tokens']}")
print(f"  Output tokens: {total_usage['output_tokens']}")
print(f"  Total tokens: {total_usage['total_tokens']}")

== Total token usage: ==
  Input tokens: 32870
  Output tokens: 1826
  Total tokens: 34696

== Detailed usage for each LLM call: ==
LLM.call_1:
  Input tokens: 295
  Output tokens: 63
  Total tokens: 358
LLM.call_2:
  Input tokens: 465
  Output tokens: 65
  Total tokens: 530
LLM.call_3:
  Input tokens: 4445
  Output tokens: 76
  Total tokens: 4521

... (other modules)

禁用自动跟踪

可以通过调用 mlflow.crewai.autolog(disable=True) 或 mlflow.autolog(disable=True) 来全局禁用 CrewAI 的自动跟踪。