Evaluation

Now that you have Phoenix up and running, and sent traces to your first project, the next step you can take is running evaluations of your application. Evaluations let you measure and monitor the quality of your application by scoring traces against metrics like accuracy, relevance, or custom checks.

1. Set environment variables to connect to your Phoenix instance:

   export PHOENIX_API_KEY="your-api-key"
   
   # Local (default, no API key required)
   export PHOENIX_HOST="http://localhost:6006"
   
   # Phoenix Cloud
   # export PHOENIX_HOST="https://app.phoenix.arize.com/s/your-space-name"
   
   # Self-hosted
   # export PHOENIX_HOST="https://your-phoenix-instance.com"
   

   You can find your host URL and API key in the Settings page of your Phoenix instance.

2. You’ll need to install the evals library that’s apart of Phoenix.

   Python

   pip install -q "arize-phoenix-evals>=2"
   pip install -q "arize-phoenix-client"
   

   TypeScript

   npm install @arizeai/phoenix-evals @arizeai/phoenix-client
   

3. Since, we are running our evaluations on our trace data from our first project, we’ll need to pull that data into our code.

   Python

   from phoenix.client import Client
   
   px_client = Client()
   primary_df = px_client.spans.get_spans_dataframe(project_identifier="my-llm-app")
   

   TypeScript

   import { getSpans } from "@arizeai/phoenix-client/spans";
   
   // Simple: Get first batch of spans (most common use case)
   const { spans } = await getSpans({
     project: { projectName: "my-llm-app" },
     limit: 100,
   });
   
   // Full: Get ALL spans with pagination
   const allSpans = [];
   let cursor: string | undefined;
   
   do {
     const result = await getSpans({
       project: { projectName: "my-llm-app" },
       cursor,
       limit: 100,
     });
     allSpans.push(...result.spans);
     cursor = result.nextCursor ?? undefined;
   } while (cursor);
   

4. In this example, we will define, create, and run our own evaluator. There’s a number of different evaluators you can run, but this quick start will go through an LLM as a Judge Model.

   1) Define your LLM Judge Model

   We’ll use OpenAI as our evaluation model for this example, but Phoenix also supports a number of other models.

   If you haven’t yet defined your OpenAI API Key from the previous step, let’s first add it to our environment.

   Python

   import os
   from getpass import getpass
   
   if not (openai_api_key := os.getenv("OPENAI_API_KEY")):
       openai_api_key = getpass("🔑 Enter your OpenAI API key: ")
   
   os.environ["OPENAI_API_KEY"] = openai_api_key
   
   from phoenix.evals.llm import LLM
   llm = LLM(model="gpt-4o", provider="openai")
   

   TypeScript

   // pnpm add @ai-sdk/openai
   
   import { createOpenAI } from "@ai-sdk/openai";
   
   const openai = createOpenAI({
     apiKey: process.env.OPENAI_API_KEY || "your-api-key-here",
   });
   
   const llm = openai("gpt-4o");
   

   2) Define your Evaluators

   We will set up a Q&A correctness Evaluator with the LLM of choice. I want to first define my LLM-as-a-Judge prompt template. Most LLM-as-a-judge evaluations can be framed as a classification task where the output is one of two or more categorical labels.

   Python

   CORRECTNESS_TEMPLATE = """ 
   You are given a question and an answer. Decide if the answer is fully correct. 
   Rules: The answer must be factually accurate, complete, and directly address the question. 
   If it is, respond with "correct". Otherwise respond with "incorrect". 
   [BEGIN DATA]
       ************
       [Question]: {attributes.llm.input_messages}
       ************
       [Answer]: {attributes.llm.output_messages}
   [END DATA]
   
   Your response must be a single word, either "correct" or "incorrect",
   and should not contain any text or characters aside from that word.
   "correct" means that the question is correctly and fully answered by the answer.
   "incorrect" means that the question is not correctly or only partially answered by the
   answer.
   """
   

   TypeScript

   const CORRECTNESS_TEMPLATE = `
   You are given a question and an answer. Decide if the answer is fully correct.
   Rules: The answer must be factually accurate, complete, and directly address the question.
   If it is, respond with "correct". Otherwise respond with "incorrect".
   
   [BEGIN DATA]
       ************
       [Question]: {{attributes.llm.input_messages}}
       ************
       [Answer]: {{attributes.llm.output_messages}}
   [END DATA]
   
   Your response must be a single word, either "correct" or "incorrect",
   and should not contain any text or characters aside from that word.
   "correct" means that the question is correctly and fully answered by the answer.
   "incorrect" means that the question is not correctly or only partially answered by the
   answer.
   `;
   

   Now we want to define our Classification Evaluator

   Python

   from phoenix.evals import create_classifier
   
   correctness_evaluator = create_classifier(
       name="correctness",
       prompt_template=CORRECTNESS_TEMPLATE,
       llm=llm,
       choices={"correct": 1.0, "incorrect": 0.0},
   )
   

   TypeScript

   import { createClassificationEvaluator } from "@arizeai/phoenix-evals";
   
   const correctnessEvaluator = createClassificationEvaluator({
     name: "correctness",
     promptTemplate: CORRECTNESS_TEMPLATE,
     model: llm,
     choices: { correct: 1.0, incorrect: 0.0 },
   });
   

5. Now that we have defined our evaluator, we’re ready to evaluate our traces.

   Python

   from phoenix.evals import evaluate_dataframe
   
   results_df = evaluate_dataframe(
       dataframe=primary_df,
       evaluators=[correctness_evaluator]
   )
   

6. You’ll now be able to log your evaluations in your project view.

   Python

   from phoenix.evals.utils import to_annotation_dataframe
   
   evaluations = to_annotation_dataframe(
       dataframe=results_df
   )
   client.log_span_annotations(
       dataframe=evaluations
   )
   

   TypeScript

   import { logSpanAnnotations } from "@arizeai/phoenix-client/annotations";
   
   await logSpanAnnotations({
     spanAnnotations: results
       .filter((r) => r.label !== undefined)
       .map((r) => ({
         spanId: r.spanId,
         name: "QA Correctness",
         label: r.label,
         score: r.score,
         annotatorKind: "LLM" as const,
       })),
   });
   

Next Steps

LLM as a Judge

Learn how LLM-based evaluation works and best practices

Pre-built Evaluators

Use pre-tested evaluators for hallucinations, relevance, toxicity, and more

Custom Evaluators

Build custom evaluators tailored to your use case

Datasets & Experiments

Run evaluations systematically with datasets and experiments