> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlit.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluations
> Score AI outputs automatically for hallucination, bias, toxicity, and more - or add your own human feedback
Automated AI evaluation to assess and monitor the quality, safety, and performance of your LLM outputs across development and production environments. Open **Monitor → Evaluations** for Analytics, Evaluators, and Configuration. For evaluation types and custom evaluators, see [Evaluators](/latest/openlit/evaluations/evaluators).
## Find the right feature
| If you want to... | Use this OpenLIT feature |
| ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Automatically score every production trace | [Configuration](/latest/openlit/evaluations/configuration) (Auto Evaluation) |
| Score one specific trace on demand | [LLM-as-a-Judge](/latest/openlit/evaluations/llm-as-a-judge) from that trace's Evaluation tab |
| Rate a trace yourself instead of an LLM judge | [Manual Feedback](/latest/openlit/evaluations/manual-feedback) - Good / Bad / Neutral plus a comment |
| Judge responses against your own ground truth, not just the model's knowledge | [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview) + [Context](/latest/openlit/prompts-experiments/context/overview) |
| Evaluate on criteria beyond the 11 built-in types | [Custom evaluators](/latest/openlit/evaluations/evaluators#custom-evaluators) |
| Only run certain evaluators for certain models, providers, or environments | [Evaluators](/latest/openlit/evaluations/evaluators) + [Rule Engine](/latest/openlit/prompts-experiments/rule-engine/overview) |
| Test prompt or model changes before shipping | [Programmatic evaluations](/latest/sdk/quickstart-programmatic-evals) via the SDK |
| Block a deploy on a quality regression | [Programmatic evaluations](/latest/sdk/quickstart-programmatic-evals) in a CI/CD pipeline |
| See pass-rate trends across evaluators and time | [Analytics](/latest/openlit/evaluations/analytics) |
## Setup
```mermaid theme={null}
flowchart LR;
A[LLM / Agent call] -->|creates| B[Trace in OpenLIT];
B -->|Auto Eval schedule or manual Run Evaluation| C[LLM Judge];
C -->|score + classification + reasoning| D[Trace detail & Dashboard];
```
Go to **Evaluations → Settings**, choose a provider and model to act as the judge (OpenAI, Anthropic, Google, Mistral, and 7+ others), and add its API key from [Vault](/latest/openlit/developer-resources/vault/overview).
Switch to the **Evaluation Types** tab and turn on the evaluators you want. Hallucination, Bias, and Toxicity are enabled by default; Relevance, Coherence, Safety, and 5 others are opt-in.
Back in Settings, enable **Auto Evaluation** with a cron schedule so every new trace gets scored automatically - or skip this and click **Run Evaluation** from any trace's Evaluation tab to score it on demand.
Open any trace's **Evaluation** tab for its score, classification, and reasoning, or check the **Evaluations** dashboard for aggregate trends across models and time.
```mermaid theme={null}
flowchart LR;
E[Dataset] -->|SDK evaluate call| F[LLM Judge];
F -->|score| G[Pass / fail gate];
```
In OpenLIT, go to **Settings → API Keys** and create a key. Offline evaluations call your OpenLIT server, so you'll need this key plus your OpenLIT URL.
```python theme={null}
import openlit
openlit.init(
openlit_url="http://localhost:3000",
openlit_api_key="openlit-xxxxx",
)
result = openlit.eval(
prompt="What is the capital of France?",
response="The capital of France is Lyon.",
contexts=["Paris is the capital and largest city of France."],
)
assert result.passed, f"Evaluation failed: {result.failed_evals}"
```
```typescript theme={null}
import openlit, { isPassed, getFailedEvals } from "openlit"
openlit.init({
openlitUrl: "http://localhost:3000",
openlitApiKey: "openlit-xxxxx",
})
const result = await openlit.eval({
prompt: "What is the capital of France?",
response: "The capital of France is Lyon.",
contexts: ["Paris is the capital and largest city of France."],
})
console.log(isPassed(result), getFailedEvals(result))
```
This uses the exact same evaluators, custom types, and Rule Engine context matching configured in your dashboard - see the [Programmatic evaluations quickstart](/latest/sdk/quickstart-programmatic-evals) for batch runs and CI/CD gating.
***
Built-in and custom evaluation types
Auto Evaluation schedule and judge model
Use advanced LLMs to evaluate AI application quality with automated scoring
Quick start guide for implementing custom evaluations in your code