> ## Documentation Index
> Fetch the complete documentation index at: https://arizeai-433a7140.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# LLM as a Judge
> Evaluating tasks performed by LLMs can be difficult due to their complexity and the diverse criteria involved. Traditional methods like rule-based assessment or similarity metrics (e.g., ROUGE, BLEU) often fall short when applied to the nuanced and varied outputs of LLMs.
LLM as a Judge is a general evaluation concept that applies to both evaluation approaches in Phoenix. You can use it via the SDK (client-side) or configure LLM evaluators directly in the Phoenix UI (server-side).
For instance, an AI assistant’s answer to a question can be:
* not grounded in context
* repetitive, repetitive, repetitive
* grammatically incorrect
* excessively lengthy and characterized by an overabundance of words
* incoherent
The list of criteria goes on. And even if we had a limited list, each of these would be hard to measure
To overcome this challenge, the concept of "LLM as a Judge" employs an LLM to evaluate another's output, combining human-like assessment with machine efficiency.
## How It Works
Here’s the step-by-step process for using an LLM as a judge:
First, determine what you want to evaluate, be it faithfulness, toxicity, accuracy, or another characteristic. See our [pre-built evaluators](/docs/phoenix/evaluation/pre-built-metrics) for examples of what can be assessed.
Write a prompt template that will guide the evaluation. This template should clearly define what variables are needed from both the initial prompt and the LLM's response to effectively assess the output.
Choose the most suitable LLM from our available options for conducting your specific evaluations.
Execute the evaluations across your data. This process allows for comprehensive testing without the need for manual annotation, enabling you to iterate quickly and refine your LLM's prompts.
Using an LLM as a judge significantly enhances the scalability and efficiency of the evaluation process. By employing this method, you can run thousands of evaluations across curated data without the need for human annotation.
This capability will not only speed up the iteration process for refining your LLM's prompts but will also ensure that you can deploy your models to production with confidence.
## Using LLM as a Judge in Phoenix
Write custom LLM evaluators in Python or TypeScript. See also: [Configuring the LLM](/docs/phoenix/evaluation/how-to-evals/configuring-the-llm) for model selection and prompt setup.
Configure LLM evaluators in the Phoenix UI — no local code or API key setup required.
## Additional Resources
Arize AI
Arize AI