Experiment with a Customer Support Agent

This guide shows you how to create and evaluate agents with Phoenix to improve performance.

Google Colab

colab.research.google.com

We’ll go through the following steps:

• Create a customer support agent using a router template
• Trace the agent activity, including function calling
• Create a dataset to benchmark performance
• Evaluate agent performance using code, human annotation, and LLM as a judge
• Experiment with different prompts and models

Notebook Walkthrough

We will go through key code snippets on this page. To follow the full tutorial, check out the Colab notebook above.

Customer Support Agent Architecture

We’ll be creating a customer support agent using function calling following the architecture below:

Create Tools and Agent

We have 6 functions that we will define for our agent:

1. product_comparison
2. product_search
3. customer_support
4. track_package
5. product_details
6. apply_discount_code

See the notebook for complete tool function definitions.

tools = [
    {
        "type": "function",
        "function": {
            "name": "product_comparison",
            "description": "Compare features of two products.",
            "parameters": {
                "type": "object",
                "properties": {
                    "product_a_id": {
                        "type": "string",
                        "description": "The unique identifier of Product A.",
                    },
                    "product_b_id": {
                        "type": "string",
                        "description": "The unique identifier of Product B.",
                    },
                },
                "required": ["product_a_id", "product_b_id"],
            },
        },
    },
    # Continued in notebook

We define a function below called run_prompt, which uses the chat completion call from OpenAI with functions

def run_prompt(input):
    client = openai.Client()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        tools=tools,
        tool_choice="auto",
        messages=[
            {
                "role": "system",
                "content": " ",
            },
            {
                "role": "user",
                "content": input,
            },
        ],
    )

    if (
        hasattr(response.choices[0].message, "tool_calls")
        and response.choices[0].message.tool_calls is not None
        and len(response.choices[0].message.tool_calls) > 0
    ):
        tool_calls = response.choices[0].message.tool_calls
    else:
        tool_calls = []

    if response.choices[0].message.content is None:
        response.choices[0].message.content = ""
    if response.choices[0].message.content:
        return response.choices[0].message.content
    else:
        return tool_calls

Generate Synthetic Dataset of Questions

Now that we have a basic agent, let’s generate a dataset of questions and run the prompt against this dataset! Using the template below, we’re going to generate a dataframe of 25 questions we can use to test our customer support agent.

GEN_TEMPLATE = """
You are an assistant that generates complex customer service questions.
The questions should often involve:

Multiple Categories: Questions that could logically fall into more than one category (e.g., combining product details with a discount code).
Vague Details: Questions with limited or vague information that require clarification to categorize correctly.
Mixed Intentions: Queries where the customer’s goal or need is unclear or seems to conflict within the question itself.
Indirect Language: Use of indirect or polite phrasing that obscures the direct need or request (e.g., using "I was wondering if..." or "Perhaps you could help me with...").
For specific categories:

Track Package: Include vague timing references (e.g., "recently" or "a while ago") instead of specific dates.
Product Comparison and Product Search: Include generic descriptors without specific product names or IDs (e.g., "high-end smartphones" or "energy-efficient appliances").
Apply Discount Code: Include questions about discounts that might apply to hypothetical or past situations, or without mentioning if they have made a purchase.
Product Details: Ask for comparisons or details that involve multiple products or categories ambiguously (e.g., "Tell me about your range of electronics that are good for home office setups").

Examples of More Challenging Questions
"There's an issue with one of the items I think I bought last month—what should I do?"
"I need help with something I ordered, or maybe I'm just looking for something new. Can you help?"

Some questions should be straightforward uses of the provided functions

Respond with a list, one question per line. Do not include any numbering at the beginning of each line. Do not include any category headings.
Generate 25 questions. Be sure there are no duplicate questions.
"""

resp = model(GEN_TEMPLATE)
split_response = resp.strip().split("\n")

questions_df = pd.DataFrame(split_response, columns=["question"])

Now let’s use this dataset and run it against the router prompt above!

response_df = questions_df.copy(deep=True)
response_df["response"] = response_df["question"].apply(run_prompt)
response_df["response"] = response_df["response"].astype(str)

Evaluating your Agent

Now that we have a set of test cases, we can create evaluators to measure performance. This way, we don’t have to manually inspect every single trace to see if the LLM is doing the right thing.

Here, we are defining our evaluation templates to judge whether the router selected a function correctly, whether it selected the right function, and whether it filled the arguments correctly.

ROUTER_EVAL_TEMPLATE = """ You are comparing a response to a question, and verifying whether that response should have made a function call instead of responding directly. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {question}
    ************
    [LLM Response]: {response}
    ************
    [END DATA]

Compare the Question above to the response. You must determine whether the response
decided to call the correct function.
"""
# See full eval template in the notebook

FUNCTION_SELECTION_EVAL_TEMPLATE = """You are comparing a function call response to a question and trying to determine if the generated call is correct. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {question}
    ************
    [LLM Response]: {response}
    ************
    [END DATA]

Compare the Question above to the function call. You must determine whether the function call
will return the answer to the Question. Please focus on whether the very specific
question can be answered by the function call.
"""
# See full eval template in the notebook

PARAMETER_EXTRACTION_EVAL_TEMPLATE = """ You are comparing a function call response to a question and trying to determine if the generated call has extracted the exact right parameters from the question. Here is the data:
    [BEGIN DATA]
    ************
    [Question]: {question}
    ************
    [LLM Response]: {response}
    ************
    [END DATA]

Compare the parameters in the generated function against the JSON provided below.
The parameters extracted from the question must match the JSON below exactly.
"""
# See full eval template in the notebook

Let’s run evaluations using Phoenix’s async_evaluate_dataframe function for our responses dataframe we generated above!

from phoenix.evals import LLM, ClassificationEvaluator, async_evaluate_dataframe

llm = LLM(provider="openai", model="gpt-4o")

router_evaluator = ClassificationEvaluator(
    name="router_eval",
    prompt_template=ROUTER_EVAL_TEMPLATE,
    llm=llm,
    choices={"incorrect": 0.0, "correct": 1.0},
)

function_selection_evaluator = ClassificationEvaluator(
    name="function_selection_eval",
    prompt_template=FUNCTION_SELECTION_EVAL_TEMPLATE,
    llm=llm,
    choices={"incorrect": 0.0, "correct": 1.0},
)

parameter_extraction_evaluator = ClassificationEvaluator(
    name="parameter_extraction_eval",
    prompt_template=PARAMETER_EXTRACTION_EVAL_TEMPLATE,
    llm=llm,
    choices={"incorrect": 0.0, "correct": 1.0},
)

evals_df = await async_evaluate_dataframe(
    dataframe=response_df,
    evaluators=[router_evaluator, function_selection_evaluator, parameter_extraction_evaluator],
    concurrency=10,
)
router_eval_df = pd.DataFrame(evals_df["router_eval_score"].tolist(), index=evals_df.index)
function_selection_eval_df = pd.DataFrame(
    evals_df["function_selection_eval_score"].tolist(), index=evals_df.index
)
parameter_extraction_eval_df = pd.DataFrame(
    evals_df["parameter_extraction_eval_score"].tolist(), index=evals_df.index
)

Create and Run an Experiment

With our dataset of questions we generated above, we can use our experiments feature to track changes across models, prompts, parameters for our agent.

from uuid import uuid1

px_client = Client()

dataset = px_client.datasets.create_dataset(
    dataframe=questions_df,
    name="agents-cookbook-" + str(uuid1()),
    input_keys=["question"],
)

from phoenix.evals import LLM, ClassificationEvaluator, bind_evaluator

llm = LLM(provider="openai", model="gpt-4o")
choices = {"correct": 1.0, "incorrect": 0.0}

routing_eval = bind_evaluator(
    ClassificationEvaluator(
        name="routing_eval",
        prompt_template=ROUTER_EVAL_TEMPLATE,
        llm=llm,
        choices=choices,
    ),
    input_mapping={
        "question": "input.question",
        "response": lambda x: str(x["output"]),
    },
)

function_call_eval = bind_evaluator(
    ClassificationEvaluator(
        name="function_call_eval",
        prompt_template=FUNCTION_SELECTION_EVAL_TEMPLATE,
        llm=llm,
        choices=choices,
    ),
    input_mapping={
        "question": "input.question",
        "response": lambda x: str(x["output"]),
    },
)

parameter_extraction_eval = bind_evaluator(
    ClassificationEvaluator(
        name="parameter_extraction_eval",
        prompt_template=PARAMETER_EXTRACTION_EVAL_TEMPLATE,
        llm=llm,
        choices=choices,
    ),
    input_mapping={
        "question": "input.question",
        "response": lambda x: str(x["output"]),
    },
)

def prompt_gen_task(input):
    return run_prompt(input["question"])


experiment = run_experiment(
    dataset=dataset,
    task=prompt_gen_task,
    evaluators=[routing_eval, function_call_eval, parameter_extraction_eval],
    experiment_name="agents-cookbook",
)

View Results