Classification

Classification helpers in @arizeai/phoenix-evals

Use the classification helpers when you want an LLM to choose from a fixed set of labels and return a structured explanation.

Create A Classifier Function

import { openai } from "@ai-sdk/openai";
import { createClassifierFn } from "@arizeai/phoenix-evals";

const classify = createClassifierFn({
  model: openai("gpt-4o-mini"),
  choices: { relevant: 1, irrelevant: 0 },
  promptTemplate:
    "Question: {{input}}\nContext: {{context}}\nAnswer: {{output}}\nLabel as relevant or irrelevant.",
});

const result = await classify({
  input: "What is Phoenix?",
  context: "Phoenix is an AI observability platform.",
  output: "Phoenix helps teams inspect traces and experiments.",
});

Decision Models

Any classification evaluator, including every built-in, can run on an AI SDK decision model such as OpenAI’s gpt-6-luna (Decisions API) or TypeSafe’s Jev. Decision models pick one of the labels directly and return a probability for each, instead of generating text. They are much faster and cheaper than a chat model judge. With OpenAI, use @ai-sdk/openai 4.0.89 or later.

import { openai } from "@ai-sdk/openai";
import { createCorrectnessEvaluator } from "@arizeai/phoenix-evals";

const correctness = createCorrectnessEvaluator({
  model: openai.decisionModel("gpt-6-luna"),
});

const result = await correctness.evaluate({
  input: "What is the capital of Australia?",
  output: "Sydney.",
});
// {
//   label: "incorrect",
//   score: 0,
//   metadata: {
//     probabilities: { correct: 0.02, incorrect: 0.98 },
//     confidence: 0.96,
//     modelId: "gpt-6-luna",
//   },
// }

How it differs from a language model:

• No explanation. Decision models don’t generate text, so explanation is always undefined.
• Probabilities in metadata. metadata.probabilities maps each label to its probability. metadata.confidence is included when the provider reports one (OpenAI and TypeSafe do).
• Prompt shape. The template is rendered the same way as for a language model, then sent as one plain-text state. Message roles are dropped and only text content is sent.
• Choices. Each provider sets its own limits on how many choices a question can have. A classification outside them fails with the provider’s error.
• Refusals. A decision model may refuse to classify, for example when asked to infer sensitive attributes rejected by the chosen model. On a refusal, the evaluator throws the AI SDK’s Experimental_DecisionRefusalError. In runExperiment, it’s recorded as that evaluation’s error.
• Plain model IDs. A string model ID is always treated as a language model. Use the provider’s decisionModel() to get a decision model.

Decision models can’t work through a problem step by step. They do well on judgments you can read off the text, such as relevance, faithfulness, tool selection, and toxicity. They’re unreliable for checking answers that need a calculation, such as unit conversions or arithmetic, and their probabilities can be confidently wrong on those. Use a language model for those evaluations.

Lower-Level API

Use generateClassification directly when you already have a rendered prompt and only need structured label generation. It also accepts a decision model, in which case the rendered prompt is sent as one plain-text state.

Source Map

• src/llm/createClassifierFn.ts
• src/llm/createClassificationEvaluator.ts
• src/llm/generateClassification.ts
• src/decision/decideClassification.ts
• src/decision/decisionPrompt.ts
• src/types/evals.ts