> ## Documentation Index
> Fetch the complete documentation index at: https://arizeai-433a7140.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# arize-phoenix-evals

> Tooling to evaluate LLM applications including faithfulness, correctness, relevance, and more.

[![PyPI Version](https://img.shields.io/pypi/v/arize-phoenix-evals)](https://pypi.org/project/arize-phoenix-evals/)

Phoenix Evals provides:

* Built-in evaluators for common tasks (faithfulness, correctness, relevance, conciseness, and more)
* A unified `LLM` wrapper supporting OpenAI, Anthropic, Google, LiteLLM, and other providers
* Batch evaluation over pandas DataFrames with `async_evaluate_dataframe` (or sync `evaluate_dataframe`)
* Custom evaluator creation via `create_evaluator` decorator or `ClassificationEvaluator`
* Benchmark datasets for testing evaluator accuracy

## Installation

```sh theme={null}
pip install "arize-phoenix-evals>=3"
```

Install the LLM vendor SDK for your chosen provider:

```sh theme={null}
pip install openai  # or anthropic, google-generativeai, litellm, etc.
```

## Quick example

```python theme={null}
import os
from phoenix.evals import LLM, async_evaluate_dataframe
from phoenix.evals.metrics import DocumentRelevanceEvaluator

os.environ["OPENAI_API_KEY"] = "<your-openai-key>"

llm = LLM(provider="openai", model="gpt-4o")
relevance_evaluator = DocumentRelevanceEvaluator(llm=llm)

# DataFrame must have columns matching the evaluator's input schema
# DocumentRelevanceEvaluator expects: 'input' (query) and 'output' (document)
results_df = await async_evaluate_dataframe(
    dataframe=df,
    evaluators=[relevance_evaluator],
    concurrency=10,
)
```

## Core API

| Symbol                                                         | Description                                              |
| :------------------------------------------------------------- | :------------------------------------------------------- |
| `LLM(provider=..., model=...)`                                 | Unified LLM wrapper for all supported providers          |
| `evaluate_dataframe(dataframe, evaluators)`                    | Run evaluators over a DataFrame (sync)                   |
| `async_evaluate_dataframe(dataframe, evaluators)`              | Async variant with concurrency control                   |
| `create_evaluator(name, kind=...)`                             | Decorator to turn a function into an `Evaluator`         |
| `ClassificationEvaluator(name, prompt_template, llm, choices)` | Custom classification evaluator for LLM-as-a-judge tasks |
| `bind_evaluator(evaluator, input_mapping)`                     | Bind field mappings to an evaluator                      |

## Built-in evaluators

All built-in evaluators live in `phoenix.evals.metrics` and accept `llm: LLM` as their first argument:

| Evaluator                       | Input fields                                    | Labels                           |
| :------------------------------ | :---------------------------------------------- | :------------------------------- |
| `FaithfulnessEvaluator`         | `input`, `output`, `context`                    | faithful / unfaithful            |
| `CorrectnessEvaluator`          | `input`, `output`                               | correct / incorrect              |
| `DocumentRelevanceEvaluator`    | `input`, `output`                               | relevant / unrelated             |
| `ConcisenessEvaluator`          | `input`, `output`                               | concise / verbose / too\_concise |
| `RefusalEvaluator`              | `input`, `output`                               | refusal / no\_refusal            |
| `ToolSelectionEvaluator`        | `input`, `output`, `tool_name`, `expected_tool` | correct / incorrect              |
| `ToolInvocationEvaluator`       | `input`, `tool_name`, `tool_call_args`          | correct / incorrect              |
| `ToolResponseHandlingEvaluator` | `input`, `tool_response`, `output`              | good / bad                       |
| `exact_match`                   | `output`, `expected`                            | (code-based, no LLM)             |
| `MatchesRegex(pattern=...)`     | `output`                                        | (code-based, no LLM)             |

To learn more about LLM Evals, see the [evals quickstart](/docs/phoenix/evaluation/evals).

***

## Reference Documentation

<Card title="Full API Reference" icon="arrow-up-right-from-square" href="https://arize-phoenix.readthedocs.io/projects/evals/">
  Complete API documentation for evaluators, metrics, and LLM classification
</Card>