Verdict public alpha · Apache 2.0

Monitor LLM-call quality inside production apps and agents.

Bring the Verdict-compatible traces you already have—or use Verdict to capture supported provider calls—then detect calibrated PASS/FAIL quality drift by workload. Fisher's exact for binary dimensions, Mann-Whitney U for continuous scores, Cliff's δ effect sizing, and Benjamini-Hochberg correction.

Install from PyPI → Watch the demo Open GitHub Docs

Public alpha · Python 3.10+

Bring your traces—or capture supported calls.

Install the latest compatible public-alpha packages from PyPI. No Git clone or source checkout is required. Verdict requires Python 3.10 or newer; Python 3.12 is recommended on macOS.

Existing trace store

Already have Verdict-compatible traces?

If your traces are already stored in Verdict's schema in SQLite or PostgreSQL, skip application instrumentation and install the evaluation and dashboard packages.

python -m pip install --pre \
  "cognifity-verdict[dashboard,postgres]" \
  "cognifity-verdict-eval[semantic]"

verdict-dashboard --storage "$VERDICT_STORAGE"

For SQLite, omit postgres and pass a SQLite URI such as sqlite:///./verdict.db. A library-level Langfuse adapter is available for custom integrations, but there is no one-command import pipeline: Verdict does not automatically ingest arbitrary observability schemas. Validate or build an adapter before relying on external traces.

Updating an existing install

python -m pip install --upgrade --pre \
  "cognifity-verdict[dashboard,postgres]" \
  "cognifity-verdict-eval[semantic]"

Verdict capture

Need trace capture?

Install the complete public-alpha package set, then instrument the supported provider SDKs your application actually uses.

python -m pip install --pre \
  "cognifity-verdict[anthropic,openai,google,dashboard]" \
  "cognifity-verdict-eval[semantic]" \
  cognifity-verdict-inspect

Choose only the anthropic, openai, or google extras you use. Add postgres for shared PostgreSQL storage. Do not install the unrelated verdict distribution from PyPI; it uses the same import namespace.

Updating an existing install

python -m pip install --upgrade --pre \
  "cognifity-verdict[anthropic,openai,google,dashboard]" \
  "cognifity-verdict-eval[semantic]" \
  cognifity-verdict-inspect

SDK and dashboard Evaluation engine Inspection CLI

After the packages

Add agent-guided instrumentation

The current skill supports the complete instrumentation workflow: it inspects a Python application, proposes supported capture points, and verifies evidence through storage and the dashboard. It does not auto-detect or import external trace systems.

The skill does not install Python dependencies by itself. Install the packages first, then use your coding agent's native skill installer. The URL follows the maintained main branch, so the website does not pin a release number.

Inspect the skill source

Use your native skill installer to install verdict-instrument-app from:

https://github.com/cognifityai/verdict/tree/main/skills/verdict-instrument-app

Then read the skill, inspect this application, and return a bounded plan.
Do not change code until I approve the plan.

Optional Verdict capture

Capture is optional. When you need it, no agent rewrite is required.

verdict.init() wraps supported Anthropic, OpenAI, and Google SDK calls. No proxy sits in front of your traffic. Metadata capture is immediate; content capture is opt-in and passes through bounded, best-effort sanitization before storage—not a compliance guarantee.

Manual @verdict.trace() spans are available for retrieval, reranking, and tool calls. Full agent-run stitching and task-success metrics are v1 roadmap items.

import verdict
from anthropic import Anthropic

verdict.init(
  service_name="checkout-agent",
  storage="sqlite:///./verdict.db",
)

# Use the SDK normally. Supported calls are captured.
# Run the judge and drift workflows with your own
# provider key when you want quality scoring.
client = Anthropic()
resp = client.messages.create(
  model="claude-haiku-4-5",
  max_tokens=512,
  messages=[{"role": "user", "content": q}],
)

The method

Start with traces, calibrate, then monitor drift.

Verdict separates trace collection from evaluation. Start with an existing compatible store or capture supported calls with Verdict. You can run metadata, structural checks, and semantic drift without a provider key; add one when you want judge-based PASS/FAIL scoring and workload-specific calibration.

1. Bring or capture traces

Use an existing Verdict-compatible SQLite or PostgreSQL store, or record provider, model, tokens, latency, estimated cost, finish reason, errors, and optional best-effort-sanitized content from supported SDK calls.

2. Run key-free checks

Inspect refusal rate, JSON validity, response length, latency, and token usage. Add the local MiniLM extra for semantic drift; the minimal hash fallback is lexical only.

3. Calibrate a judge

Sample traces, label PASS/FAIL dimensions yourself, then measure judge agreement before trusting drift alerts for that workload.

4. Compare like with like

Cluster similar prompts so quality shifts are evaluated by workload instead of hidden inside one global average.

5. Emit drift signals

Compare recent windows against a reference baseline and emit signals only when statistical and practical effect-size gates both clear.

Validation you can reproduce

Verdict ships scripts for checking the claims yourself.

The repo includes live capture checks, a synthetic regression injector, pairwise judge-alignment harnesses, and a rubric-alignment workflow for measuring agreement against your own labels.

The defensible claim is deliberately narrow: calibrated PASS/FAIL drift monitoring on real LLM traces, with per-workload judge calibration. Public benchmarks are sanity checks, not proof that a judge is calibrated for your workload.

Workflow	What it checks	Key required?
Smoke test	storage, schemas, basic pipeline	No
Live capture check	real SDK capture paths	Provider key
Rubric alignment	judge agreement vs your labels	Provider key
Regression injector	known corruptions through the eval path	Optional
Packaged dashboard	read-only SQLite or PostgreSQL inspection	No

Judge quality depends on the model, rubric, and workload. Run the alignment scripts on your own labeled traces before relying on alerts.

What it catches

Three drift modes that quietly cost enterprises money.

SILENT UPDATE

The minor version that broke your refusal rate.

Provider, model, or prompt changes can shift refusal behavior. Verdict gives you captured traces and structural signals so a reviewer can inspect the change by workload.

MIGRATION RISK

The $8k/month swap you'd otherwise A/B blindly.

Use pairwise comparison and workload grouping to inspect where a candidate model helps, hurts, or needs more labels before you switch production traffic.

COST RUNAWAY

The prompt change that doubled your token bill.

Prompt changes can alter response length and token usage. Verdict keeps quality, cost, latency, and error metadata in the same trace store for faster investigation.

Start with Verdict locally.

The Apache 2.0 SDK uses SQLite by default. The packaged read-only dashboard starts without a source checkout and can also inspect an existing PostgreSQL Verdict store.

Install from PyPI → Open GitHub Talk to enterprise team