> ## Documentation Index
> Fetch the complete documentation index at: https://docs.openlit.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Programmatic Evaluations

> Quickly evaluate your LLMs and AI Agent responses for Hallucination, Bias, and Toxicity

This guide shows how to run OpenLIT evaluations programmatically from your own code - the same evaluation engine, evaluators, and Rule Engine used for online/auto evaluations, called directly via the SDK.

**1. Get an API key**

In OpenLIT, go to **Settings → API Keys** and create a key. Programmatic evaluations run against your OpenLIT server, so you'll need this key plus your OpenLIT URL.

**2. Run an evaluation**

> [tab] Python
>
> ```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
> import openlit
>
> openlit.init(
>     openlit_url="http://localhost:3000",
>     openlit_api_key="openlit-xxxxx",
> )
>
> result = openlit.eval(
>     prompt="What is the capital of France?",
>     response="The capital of France is Lyon.",
>     contexts=["Paris is the capital and largest city of France."],
> )
>
> assert result.passed, f"Evaluation failed: {result.failed_evals}"
> ```

> [tab] Typescript
>
> ```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
> import openlit, { isPassed, getFailedEvals } from "openlit"
>
> openlit.init({
>   openlitUrl: "http://localhost:3000",
>   openlitApiKey: "openlit-xxxxx",
> })
>
> const result = await openlit.eval({
>   prompt: "What is the capital of France?",
>   response: "The capital of France is Lyon.",
>   contexts: ["Paris is the capital and largest city of France."],
> })
>
> console.log(isPassed(result), getFailedEvals(result))
> ```

You can also set the URL and API key via `OPENLIT_URL` / `OPENLIT_API_KEY` environment variables instead of `init()`.

**3. Evaluate a whole dataset**

Pass a list of prompt/response pairs to run them concurrently - useful as a CI/CD quality gate:

> [tab] Python
>
> ```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
> batch_result = openlit.eval_batch(dataset=[
>     {"prompt": "What is 2+2?", "response": "2+2 equals 4."},
>     {"prompt": "Who wrote Hamlet?", "response": "Hamlet was written by Charles Dickens."},
> ])
>
> assert batch_result.all_passed, f"Pass rate: {batch_result.pass_rate:.0%}"
> ```

> [tab] Typescript
>
> ```typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
> import openlit, { isAllPassed } from "openlit"
>
> const batchResult = await openlit.evalBatch({
>   dataset: [
>     { prompt: "What is 2+2?", response: "2+2 equals 4." },
>     { prompt: "Who wrote Hamlet?", response: "Hamlet was written by Charles Dickens." },
>   ],
> })
>
> console.log(isAllPassed(batchResult))
> ```

Evaluations run programmatically use the exact same evaluators, custom types, and [Rule Engine](https://docs.openlit.io/latest/openlit/prompts-experiments/rule-engine/overview.md) context matching configured in your OpenLIT dashboard - trace attributes like `service.name` and `deployment.environment` are auto-resolved from `openlit.init()` for rule matching, or you can pass your own via the `attributes` parameter.

***

> [card] [Full SDK reference](https://docs.openlit.io/latest/sdk/features/evaluations.md)
>
> Parameters, result objects, batch evaluation, and Rule Engine attribute resolution in depth

> [card] [CI/CD integration](https://docs.openlit.io/latest/sdk/features/evaluations.md#ci-cd-integration)
>
> Use evaluations as quality gates in your test suite or pipeline

> [card] [LLM-as-a-judge](https://docs.openlit.io/latest/openlit/evaluations/llm-as-a-judge.md)
>
> Automatically add evaluation scoring to production traces

> [card] [Integrations](https://docs.openlit.io/latest/sdk/integrations/overview.md)
>
> 60+ AI integrations with automatic instrumentation and performance tracking

> [card] [Destinations](https://docs.openlit.io/latest/sdk/destinations/overview.md)
>
> Send telemetry to Datadog, Grafana, New Relic, and other observability stacks


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.