Evaluations

Score AI outputs automatically for hallucination, bias, toxicity, and more - or add your own human feedback

Automated AI evaluation to assess and monitor the quality, safety, and performance of your LLM outputs across development and production environments. Open Monitor → Evaluations for Analytics, Evaluators, and Configuration. For evaluation types and custom evaluators, see Evaluators.

Find the right feature

If you want to…	Use this OpenLIT feature
Automatically score every production trace	Configuration (Auto Evaluation)
Score one specific trace on demand	LLM-as-a-Judge from that trace’s Evaluation tab
Rate a trace yourself instead of an LLM judge	Manual Feedback - Good / Bad / Neutral plus a comment
Judge responses against your own ground truth, not just the model’s knowledge	Rule Engine + Context
Evaluate on criteria beyond the 11 built-in types	Custom evaluators
Only run certain evaluators for certain models, providers, or environments	Evaluators + Rule Engine
Test prompt or model changes before shipping	Programmatic evaluations via the SDK
Block a deploy on a quality regression	Programmatic evaluations in a CI/CD pipeline
See pass-rate trends across evaluators and time	Analytics

Setup

Online

1. Pick a judge model

   Go to Evaluations → Settings, choose a provider and model to act as the judge (OpenAI, Anthropic, Google, Mistral, and 7+ others), and add its API key from Vault.

2. Enable evaluators

   Switch to the Evaluation Types tab and turn on the evaluators you want. Hallucination, Bias, and Toxicity are enabled by default; Relevance, Coherence, Safety, and 5 others are opt-in.

3. Turn on Auto Evaluation

   Back in Settings, enable Auto Evaluation with a cron schedule so every new trace gets scored automatically - or skip this and click Run Evaluation from any trace’s Evaluation tab to score it on demand.

4. Review results

   Open any trace’s Evaluation tab for its score, classification, and reasoning, or check the Evaluations dashboard for aggregate trends across models and time.

Offline

1. Get an API key

   In OpenLIT, go to Settings → API Keys and create a key. Offline evaluations call your OpenLIT server, so you’ll need this key plus your OpenLIT URL.

2. Run an evaluation

   Python

   import openlit
   
   openlit.init(
       openlit_url="http://localhost:3000",
       openlit_api_key="openlit-xxxxx",
   )
   
   result = openlit.eval(
       prompt="What is the capital of France?",
       response="The capital of France is Lyon.",
       contexts=["Paris is the capital and largest city of France."],
   )
   
   assert result.passed, f"Evaluation failed: {result.failed_evals}"
   

   Typescript

   import openlit, { isPassed, getFailedEvals } from "openlit"
   
   openlit.init({
     openlitUrl: "http://localhost:3000",
     openlitApiKey: "openlit-xxxxx",
   })
   
   const result = await openlit.eval({
     prompt: "What is the capital of France?",
     response: "The capital of France is Lyon.",
     contexts: ["Paris is the capital and largest city of France."],
   })
   
   console.log(isPassed(result), getFailedEvals(result))
   

   This uses the exact same evaluators, custom types, and Rule Engine context matching configured in your dashboard - see the Programmatic evaluations quickstart for batch runs and CI/CD gating.

---

Evaluators

Built-in and custom evaluation types

Configuration

Auto Evaluation schedule and judge model

LLM-as-a-Judge

Use advanced LLMs to evaluate AI application quality with automated scoring

Programmatic evaluations

Quick start guide for implementing custom evaluations in your code