Programmatic Evaluations

Quickly evaluate your LLMs and AI Agent responses for Hallucination, Bias, and Toxicity

This guide shows how to run OpenLIT evaluations programmatically from your own code - the same evaluation engine, evaluators, and Rule Engine used for online/auto evaluations, called directly via the SDK.

1. Get an API key

   In OpenLIT, go to Settings → API Keys and create a key. Programmatic evaluations run against your OpenLIT server, so you’ll need this key plus your OpenLIT URL.

2. Run an evaluation

   Python

   import openlit
   
   openlit.init(
       openlit_url="http://localhost:3000",
       openlit_api_key="openlit-xxxxx",
   )
   
   result = openlit.eval(
       prompt="What is the capital of France?",
       response="The capital of France is Lyon.",
       contexts=["Paris is the capital and largest city of France."],
   )
   
   assert result.passed, f"Evaluation failed: {result.failed_evals}"
   

   Typescript

   import openlit, { isPassed, getFailedEvals } from "openlit"
   
   openlit.init({
     openlitUrl: "http://localhost:3000",
     openlitApiKey: "openlit-xxxxx",
   })
   
   const result = await openlit.eval({
     prompt: "What is the capital of France?",
     response: "The capital of France is Lyon.",
     contexts: ["Paris is the capital and largest city of France."],
   })
   
   console.log(isPassed(result), getFailedEvals(result))
   

   You can also set the URL and API key via OPENLIT_URL / OPENLIT_API_KEY environment variables instead of init().

3. Evaluate a whole dataset

   Pass a list of prompt/response pairs to run them concurrently - useful as a CI/CD quality gate:

   Python

   batch_result = openlit.eval_batch(dataset=[
       {"prompt": "What is 2+2?", "response": "2+2 equals 4."},
       {"prompt": "Who wrote Hamlet?", "response": "Hamlet was written by Charles Dickens."},
   ])
   
   assert batch_result.all_passed, f"Pass rate: {batch_result.pass_rate:.0%}"
   

   Typescript

   import openlit, { isAllPassed } from "openlit"
   
   const batchResult = await openlit.evalBatch({
     dataset: [
       { prompt: "What is 2+2?", response: "2+2 equals 4." },
       { prompt: "Who wrote Hamlet?", response: "Hamlet was written by Charles Dickens." },
     ],
   })
   
   console.log(isAllPassed(batchResult))
   

Evaluations run programmatically use the exact same evaluators, custom types, and Rule Engine context matching configured in your OpenLIT dashboard - trace attributes like service.name and deployment.environment are auto-resolved from openlit.init() for rule matching, or you can pass your own via the attributes parameter.

---

Full SDK reference

Parameters, result objects, batch evaluation, and Rule Engine attribute resolution in depth

CI/CD integration

Use evaluations as quality gates in your test suite or pipeline

LLM-as-a-judge

Automatically add evaluation scoring to production traces

Integrations

60+ AI integrations with automatic instrumentation and performance tracking

Destinations

Send telemetry to Datadog, Grafana, New Relic, and other observability stacks