Cognifity exists because production AI needs evidence, not vibes. Verdict — our first product — is an Apache 2.0 toolkit for capturing LLM calls, measuring response quality, and inspecting drift with reproducible workflows.
Production AI teams have a measurement gap. Logs, latency charts, token dashboards, and error rates are necessary, but they do not answer whether model behavior changed for a specific workload.
Cognifity focuses on that gap: instrumentation and evaluation tools that make LLM behavior inspectable, comparable, and reproducible. The goal is not to replace application observability. It is to add the quality layer that production AI teams keep having to build themselves.
Verdict is the first product: open-source LLM-call observability and calibrated drift monitoring for engineers running AI in real applications. It starts at the individual LLM-call layer, with agent-run outcomes and richer task-level analytics kept explicit as roadmap work.
Teams exploring LLM reliability, model migration, prompt changes, or evaluation workflows can get in touch to compare notes.
Drift detection uses Fisher's exact for binary PASS/FAIL dimensions, Mann-Whitney U for continuous scores, Cliff's δ effect-size gating, and Benjamini-Hochberg multiple-comparison correction.
Verdict is open source. You can run the SDK, evaluation package, inspection CLI, and local dashboard yourself, with your own storage and provider keys.
Core systems sit behind interfaces: provider adapters, storage adapters, judge providers, and in-memory test implementations. The goal is to keep evaluation logic separate from vendors.
The docs say what ships today and what is v1 roadmap. Judge calibration is workload-specific, so the repo gives users scripts to measure agreement on their own labels.
If a feature exists to look good in a demo and not to be used at 3am during an outage, it doesn't ship. We speak Kubernetes probes, SLOs, p99 latency, blast radius. We do not speak "AI doomerism" or "AGI is here." Our audience is platform engineers and the leaders they report to.
Content capture is opt-in. SQLite runs locally. Postgres is available when teams need shared storage. API keys stay in your environment.
Whether you're an enterprise AI buyer, a potential investor, a candidate, or another engineer who's felt these gaps — we'd like to hear from you.