Cognifity exists because production AI needs evidence, not guesswork. Verdict — our first product — is an Apache 2.0 toolkit for capturing LLM calls, measuring response quality, and inspecting drift with reproducible workflows.
Production AI teams have a measurement gap. Logs, latency charts, token dashboards, and error rates are necessary, but they do not answer whether model behavior changed for a specific workload.
Cognifity focuses on that gap: instrumentation and evaluation tools that make LLM behavior inspectable, comparable, and reproducible. The goal is not to replace application observability. It is to add the quality layer that production AI teams keep having to build themselves.
Verdict is the first product: open-source LLM-call observability and calibrated drift monitoring for engineers running AI in real applications. It starts at the individual LLM-call layer, with agent-run outcomes and richer task-level analytics kept explicit as roadmap work.
Teams exploring LLM reliability, model migration, prompt changes, or evaluation workflows can get in touch to compare notes.
Cognifity is led by Rehan Shah, an engineering leader who has spent nearly two decades building and scaling enterprise software and production AI applications.
He started Cognifity after seeing the same gap repeatedly: teams putting LLM-powered systems into production without a rigorous way to tell when quality, cost, or behavior had changed. Verdict is the first answer to that gap: open-source, statistically grounded observability for the LLM stack teams actually run.
Drift detection uses Fisher's exact for binary PASS/FAIL dimensions, Mann-Whitney U for continuous scores, Cliff's δ effect-size gating, and Benjamini-Hochberg multiple-comparison correction.
Verdict is open source. You can run the SDK, evaluation package, inspection CLI, and local dashboard yourself, with your own storage and provider keys.
Core systems sit behind interfaces: provider adapters, storage adapters, judge providers, and in-memory test implementations. The goal is to keep evaluation logic separate from vendors.
The docs say what ships today and what is v1 roadmap. Judge calibration is workload-specific, so the repo gives users scripts to measure agreement on their own labels.
If a feature exists only to look good in a demo and cannot help during a real production investigation, it does not ship. The product is built for platform engineers and the leaders they report to: clear signals, inspectable evidence, and workflows that hold up when trust is on the line.
Content capture is opt-in. SQLite runs locally. Postgres is available when teams need shared storage. API keys stay in your environment.
Whether you're an enterprise AI buyer, a potential investor, a candidate, or another engineer who's felt these gaps — we'd like to hear from you.