Using the Playground

Run, configure, and compare prompts in the Phoenix Prompt Playground, and run them over datasets as experiments.

Use the Prompt Playground to run a prompt against an LLM, change its model settings, compare prompt variants, and run prompts over a dataset.

​

Setup

1. Configure AI Providers and set the API key for your provider.
2. In the playground view, write a valid prompt for the LLM.
3. Click Run on the top right, or press mod + enter.

If the run is successful, the LLM output streams into the Output section.

Pick an LLM and setup the API Key for that provider to get started

​

Prompt Editor

Define the prompt template in the prompt editor, on the left side of the screen. For the template model, see Prompts Concepts.

1. Select the template language on the toolbar: mustache or f-string.
2. Type a variable placeholder in the prompt, for example {{question}} for mustache.
3. Fill each variable in the inputs section, or fill the variables from a dataset.

Each variable placeholder shows in the inputs section. In a dataset, each row supplies key/value pairs for the inputs.

Use the template language to create prompt template variables that can be applied during runtime

​

Model Configuration

Each prompt instance has its own LLM and invocation parameters.

1. Click the model configuration button at the top of the prompt editor.
2. Select the LLM and set the invocation parameters.
3. To keep this configuration across playground sessions, click save as default.

Switch models and modify invocation params

For OpenAI and Azure OpenAI models, select the OpenAI API type in the model configuration panel. Select Chat Completions (chat.completions.create) or Responses (responses.create) for the model and features that you need.

​

Reasoning / extended thinking

For models that support reasoning, the model configuration panel shows controls for how much the model thinks before it responds:

• Anthropic (Claude 3.7 Sonnet and later): set extended thinking to disabled, adaptive (the default), or enabled. With enabled, set an explicit budget_tokens value and a visibility toggle. You can also set the output effort (high / medium / low). When you enable extended thinking, Phoenix increases max_tokens to a minimum of budget_tokens + 1.
• Google (Gemini 2.5 models): set thinkingBudget, thinkingLevel (low / medium / high), or both.

Higher thinking budgets and effort levels can improve quality on difficult tasks, but they increase latency and token use. Start with the defaults. Increase them only when a task needs more reasoning.

​

Comparing Prompts

Compare multiple prompt variants in the playground. The UI labels the variants A, B, C, and D.

1. Click + Compare at the top of the first prompt to add a duplicate instance.
2. Change the template, model, or parameters of each variant. Each variant keeps its own settings.
3. Click Run to compare the outputs, or run an experiment to find the best prompt and model configuration for the task.

Compare multiple different prompt variants at once

​

Using Datasets with Prompts

Run one or more prompts on each example in a dataset.

1. Load a dataset that contains the input variables of your prompt template.
2. Click Run.

Phoenix applies each configured prompt to each dataset example. It invokes the LLM for all prompt-example combinations. Phoenix records the results as an experiment on the loaded dataset. See Playground Traces.

Each example's input is used to fill the prompt template

​

Configuring Dataset Paths

By default, the playground reads template variables from input. Thus {question} resolves against input.question. If your dataset examples keep inputs in a different nested object, set the path:

1. Load a dataset.
2. Click the settings button (gear icon) in the experiment toolbar, next to the dataset selector.
3. Set Prompt variable path to the dot-notation path that contains your template variables, for example input or payload.inputs.

Phoenix then resolves template variables against that object. For example, your prompt uses {question} and your dataset example is:

{
  "input": {
    "question": "What is the weather in San Francisco?"
  }
}

For this example, keep the default prompt variable path, input.

To refer to root-level fields directly, for example {input.question} and {output.response}, clear the prompt variable path. Variables then resolve from the root object. For this dataset example:

{
  "input": {
    "question": "What is the weather in San Francisco?"
  },
  "output": {
    "response": "I can help with that."
  }
}

use template variables such as {input.question} and {output.response} in your prompt.

​

Naming the Experiment

1. Click the settings button (gear icon) in the experiment toolbar.
2. Optional: enter a Name and a Description for the next experiment that you run.

If you leave Name empty, Phoenix uses the generated default name. To change the name or description after the run, open the experiment action menu in the dataset experiments table and select Edit.

​

Appending Conversation History

When you run experiments over datasets, you can append conversation messages from your dataset examples to the prompt. Use this to:

• A/B test models: compare how different models respond to the same conversation history.
• Test system prompts: evaluate different system prompts against the same user conversations.
• Run multi-turn conversation experiments: run experiments on existing conversation threads.

​

Setting the Appended Messages Path

1. Load a dataset that contains conversation messages in OpenAI format.
2. Click the settings button (gear icon) in the experiment toolbar, next to the dataset selector.
3. Enter the dot-notation path to the messages array in your dataset examples, for example messages or input.messages.

When you run the experiment, Phoenix appends the messages at that path to the end of your prompt template. It does this after it applies the template variables. Thus you can keep your system prompt and initial instructions in the template and replay real conversation history from the dataset.

​

Dataset Format

Your dataset examples must contain messages in the OpenAI chat format:

{
  "messages": [
    {"role": "user", "content": "What is the weather in San Francisco?"},
    {"role": "assistant", "content": "Let me check that for you."},
    {"role": "user", "content": "Thanks! Also, what about New York?"}
  ]
}

Supported message roles:

Role	Content
user	User messages
assistant	Assistant/AI responses
system	System messages
tool	Tool response messages (with tool_call_id)

For nested structures, use a dot-notation path:

{
  "input": {
    "messages": [
      {"role": "user", "content": "Hello!"}
    ]
  }
}

For this example, set the path to input.messages.

​

Example: A/B Testing System Prompts

1. Create a dataset with conversation examples (user messages and expected context).
2. In the playground, configure two prompt variants (A and B) with different system prompts.
3. Load your dataset and set the appended messages path to messages.
4. Run the experiment to compare how each system prompt handles the same conversations.

Use this procedure to evaluate prompt changes across many real conversations.

​

Playground Traces

Phoenix records all LLM invocations from the playground. Use these records for analysis, annotations, evaluations, and dataset curation.

If you run an LLM with free-form inputs (no dataset), Phoenix records the spans in a project named “playground”.

All free form playground runs are recorded under the playground project

If you run a prompt over dataset examples, Phoenix records the outputs and spans as an experiment. If you do not name the experiment, its name is playground-experiment. If the playground has a saved prompt loaded, its name is playground-experiment prompt:<prompt name>.

If you run over a dataset, the output and traces is tracked as a dataset experiment