Precision / Recall / F-Score

Compute precision, recall, and F-beta scores for classification tasks

Overview

The PrecisionRecallFScore evaluator computes precision, recall, and F-beta scores for comparing predicted labels against expected labels. It supports both binary and multi-class classification with various averaging strategies, and is available in Python and TypeScript.

When to Use

Use the PrecisionRecallFScore evaluator when you need to:

• Evaluate classification performance - Measure how well your model predicts correct labels
• Compare label sequences - Assess predicted vs expected labels for multi-item outputs
• Binary classification metrics - Compute metrics for spam/ham, positive/negative, etc.
• Multi-class evaluation - Evaluate across multiple categories with different averaging strategies

This is a code-based evaluator that computes standard classification metrics. Both expected and output should be sequences of labels (strings or integers) — the full sequence across a dataset, not a single row’s label.

How These Metrics Work

Every prediction for a given class falls into one of four buckets: true positive (TP, correctly predicted that class), false positive (FP, predicted that class but it wasn’t), false negative (FN, was that class but predicted something else), and true negative (not that class and not predicted as it). Precision, recall, and F-beta are all built from TP, FP, and FN:

Metric	Formula	Answers
Precision	TP / (TP + FP)	Of everything predicted as this class, what fraction actually was?
Recall	TP / (TP + FN)	Of everything that actually is this class, what fraction did the model find?
F-beta	(1 + beta^2) * precision * recall / (beta^2 * precision + recall)	The weighted harmonic mean of precision and recall.

Precision and recall trade off against each other: a model that predicts the positive class more aggressively tends to raise recall (catches more true positives) at the cost of precision (more false alarms), and vice versa. Which one matters more is a property of the task, not the model:

• Spam filtering — a false positive (a real email marked as spam) is usually worse than a false negative (spam that slips through), so precision matters more.
• Medical screening / fraud detection — a false negative (a missed disease or fraudulent transaction) is usually worse than a false positive (an unnecessary follow-up), so recall matters more.

F-beta (the F-measure introduced by Van Rijsbergen, see References) combines them into a single number, where beta sets how much more recall is weighted than precision: beta = 1 (F1, the default) weights them equally, beta > 1 (e.g. F2) favors recall, and beta < 1 (e.g. F0.5) favors precision. Because it’s a harmonic mean, F-beta stays low if either precision or recall is low.

Averaging Strategies

Precision, recall, and F-beta are inherently per-class metrics. For multi-class classification, average controls how the per-class scores combine into a single number:

Strategy	Description	Good for
macro	Compute each class’s metric independently, then take the unweighted mean.	Surfacing whether a rare class is being ignored — every class counts equally regardless of how often it occurs.
weighted	Like macro, but each class’s metric is weighted by its support (how often it actually occurs).	Reflecting overall performance when class frequencies are meaningful and imbalance is expected.
micro	Pool every class’s TP, FP, and FN first, then compute one precision/recall/F-beta from the totals.	A single aggregate number. For single-label multi-class problems (exactly one predicted and one expected label per example), micro precision, recall, and F1 are all equal to overall accuracy.

macro and weighted compute F-beta per class first and then average the per-class F-beta values — they don’t derive F-beta from the averaged precision and recall. This matches scikit-learn’s precision_recall_fscore_support semantics (see References), so results are directly comparable to the equivalent sklearn.metrics call.

For numeric labels {0, 1} with the default "macro" average and no positive_label/positiveLabel configured, the evaluator automatically treats 1 as the positive class and computes binary (one-vs-rest) metrics instead of multi-class averaging. Configuring a non-default average skips this auto-detection, so an explicitly requested averaging strategy is never silently overridden by the shape of the data.

Supported Levels

This evaluator is not tied to specific tracing levels. It operates on lists of predicted and expected labels, making it useful for:

• Comparing model predictions against ground truth labels
• Evaluating classification outputs at any level where you have paired label sequences
• Batch evaluation of classification tasks in experiments

Input Requirements

The evaluator requires two inputs:

Field	Type	Description
expected	List[str | int]	List of expected/true labels
output	List[str | int]	List of predicted labels

Both sequences must have the same length and contain at least one element.

Constructor Arguments

Python

Argument	Type	Default	Description
beta	float	1.0	Weight of recall relative to precision (F1 by default)
average	str	"macro"	Averaging strategy: "macro", "micro", or "weighted"
positive_label	str | int	None	For binary classification, specify the positive class
zero_division	float	0.0	Value to use when a metric is undefined (0/0)

TypeScript

Option	Type	Default	Description
beta	number	1	Weight of recall relative to precision. Only on createFBetaEvaluator (F1 is createF1Evaluator, equivalent to beta: 1).
average	"macro" | "micro" | "weighted"	"macro"	Averaging strategy
positiveLabel	string | number	undefined	For binary classification, specify the positive class
zeroDivision	number	0	Value to use when a metric is undefined (0/0)

Output Interpretation

Python

The evaluator returns three Score objects:

Score Name	Description
precision	Ratio of true positives to predicted positives
recall	Ratio of true positives to actual positives
f1 (or f{beta})	Harmonic mean of precision and recall

All scores have direction = "maximize" (higher is better) and kind = "code" (code-based evaluator).

TypeScript

Each factory returns an evaluator whose .evaluate({ expected, output }) resolves to a single { score }:

Evaluator	score is
createPrecisionEvaluator()	Precision
createRecallEvaluator()	Recall
createF1Evaluator() / createFBetaEvaluator({ beta })	F1 / F-beta

createPrecisionRecallFScoreEvaluators() bundles matching { precision, recall, fScore } evaluators from one options object. Every evaluator has kind: "CODE" and optimizationDirection: "MAXIMIZE", and its name reflects the configuration, e.g. precision, recall_weighted, f1, f0_5_micro.

Usage Examples

Python

from phoenix.evals.metrics import PrecisionRecallFScore

# Create evaluator with default settings (F1, macro averaging)
evaluator = PrecisionRecallFScore()

# Inspect the evaluator's requirements
print(evaluator.describe())

# Multi-class evaluation
eval_input = {
    "expected": ["cat", "dog", "cat", "bird", "dog"],
    "output": ["cat", "cat", "cat", "bird", "dog"]
}

scores = evaluator.evaluate(eval_input)
for score in scores:
    print(f"{score.name}: {score.score:.3f}")
# precision: 0.889
# recall: 0.833
# f1: 0.822

TypeScript

import {
  createPrecisionRecallFScoreEvaluators,
} from "@arizeai/phoenix-evals/code";

// Matching precision/recall/F1 evaluators sharing one options object
// (F1, macro averaging by default)
const { precision, recall, fScore } = createPrecisionRecallFScoreEvaluators();

// Multi-class evaluation
const example = {
  expected: ["cat", "dog", "cat", "bird", "dog"],
  output: ["cat", "cat", "cat", "bird", "dog"],
};

console.log("precision:", (await precision.evaluate(example)).score?.toFixed(3));
console.log("recall:", (await recall.evaluate(example)).score?.toFixed(3));
console.log("f1:", (await fScore.evaluate(example)).score?.toFixed(3));
// precision: 0.889
// recall: 0.833
// f1: 0.822

Binary Classification

For binary classification, specify the positive label:

Python

from phoenix.evals.metrics import PrecisionRecallFScore

# Binary classification for spam detection
evaluator = PrecisionRecallFScore(positive_label="spam")

eval_input = {
    "expected": ["spam", "ham", "spam", "ham", "spam"],
    "output": ["spam", "spam", "ham", "ham", "spam"]
}

scores = evaluator.evaluate(eval_input)
for score in scores:
    print(f"{score.name}: {score.score:.3f}")
# precision: 0.667  (2 TP / 3 predicted spam)
# recall: 0.667     (2 TP / 3 actual spam)
# f1: 0.667

TypeScript

import {
  createPrecisionEvaluator,
  createRecallEvaluator,
  createF1Evaluator,
} from "@arizeai/phoenix-evals/code";

// Binary classification for spam detection
const precision = createPrecisionEvaluator({ positiveLabel: "spam" });
const recall = createRecallEvaluator({ positiveLabel: "spam" });
const f1 = createF1Evaluator({ positiveLabel: "spam" });

const example = {
  expected: ["spam", "ham", "spam", "ham", "spam"],
  output: ["spam", "spam", "ham", "ham", "spam"],
};

console.log("precision:", (await precision.evaluate(example)).score?.toFixed(3));
console.log("recall:", (await recall.evaluate(example)).score?.toFixed(3));
console.log("f1:", (await f1.evaluate(example)).score?.toFixed(3));
// precision: 0.667  (2 TP / 3 predicted spam)
// recall: 0.667     (2 TP / 3 actual spam)
// f1: 0.667

A runnable version of this example, plus F-beta and all three averaging strategies, lives in examples/classification_metrics_example.ts.

Using with Phoenix

Evaluating Traces

Run evaluations on traces collected in Phoenix and log results as annotations:

• Evaluating Phoenix Traces

Running Experiments

Use the PrecisionRecallFScore evaluator in Phoenix experiments:

• Using Evaluators in Experiments

The PrecisionRecallFScore evaluator is dataset-level (batch), not per-row: expected/output are the full sequence of labels across every example you want to score together. Collect every row’s expected and predicted label first, then call the evaluator once over the full arrays — don’t wire it in as a per-row experiment evaluator.

References

• Van Rijsbergen, C.J. (1979). Information Retrieval (2nd ed.). Butterworth-Heinemann. — origin of the F-measure and its beta parameter.
• Precision and recall — Wikipedia overview of the underlying concepts and terminology.
• sklearn.metrics.precision_recall_fscore_support — scikit-learn’s reference implementation; the average strategies here follow the same semantics.

API Reference

• Python: PrecisionRecallFScore
• TypeScript: Classification Metrics

Related

• Exact Match Evaluator - For exact string comparison
• Correctness Evaluator - For semantic correctness evaluation