Automated AI Evaluation. Trusted Every Time.
Shyena is built to evaluate any AI system before it ships — live today for conversational and voice AI. It runs real, agent-driven conversations against your live bot, judges the quality of every turn with LLM-based evaluation and deterministic checks, and makes it structurally impossible for a broken conversation to report a green pass.
Already running Cognigy? This is the evaluation layer built for you.
Shyena isn't a generic testing tool with a Cognigy integration bolted on — Cognigy is our live, flagship platform. Agentic personas, real conversation execution over the same channel your customers use, and the execution-integrity gate all run against your actual Cognigy agent today.
Automated
End-to-end automated AI evaluation at scale.
Accurate
Precise, consistent and unbiased results.
Trusted
Reliable insights. Confident decisions.
The platform
Three engines. One release gate.
Evaluation — live today
Shyena tests and evaluates Cognigy-built conversational and voice AI agents — driving real chat and voice sessions against your live Cognigy bot and judging every turn.
Security — live today, via Ziran
Intelligence — CIS
CIS is Shyena's proprietary conversation intelligence system — it understands your agent's business rules and logic on its own and generates thousands of test conversations from that understanding, so coverage grows with your agent instead of your team's authoring time.
The problem
Conversational agents don't fail like normal software.
Non-determinism breaks your test signal
LLM-driven agents answer differently every run. The same scenario passes on Tuesday and fails on Wednesday, so teams stop trusting the suite and start ignoring red builds.
Manual QA can't cover open-ended dialogue
A realistic agent has thousands of viable conversation paths. Reviewing transcripts by hand covers a handful per release, and the ones you skip are the ones that reach customers.
Test tools assume scripted click-paths
Traditional automation asserts that a selector exists. It cannot judge whether the agent handled an angry renewal request correctly, stayed in policy, or actually resolved the issue.
31
metrics evaluated on every case, by default
117
metrics in the full catalog, including custom ones you define
0
false green passes — the gate structurally prevents it
Default depth for a standard case; accessibility-gated runs evaluate more.
Where Shyena fits
Not a prompt tester. Not an observability tool.
Tools like Promptfoo and DeepEval test prompts and outputs against datasets you define. Arize Phoenix observes traces after the fact. Ragas scores whatever dataset you hand it. None of them drives a live conversation through your actual customer-facing channel. Shyena is the only one that executes a full live conversation, judges it on semantics and orchestration as well as wording, and refuses to let a broken run report a pass.
They're not mutually exclusive — teams often unit-test prompts with tools like these before Shyena runs the full conversation as the release gate none of them cover.
How it works
One run. 31 metrics. A verdict you can defend.
Every regression run follows the same four stages, and each stage produces evidence the next one is allowed to trust — from a single persona definition to a case scored against 31 metrics by default, spanning LLM-judged quality, deterministic assertions, semantic state-transition validity, and orchestrator-level decision analysis.
Define agentic test personas
Describe a goal, a persona and a playbook — not brittle scripted steps. Shyena improvises like a real customer would.
Execute real conversations
A real browser or API session drives your live agent end to end, across chat and voice, with retries and backpressure built in.
Evaluate against 31 metrics
LLM-as-judge scoring, deterministic assertions, six-construct semantic assurance, and orchestrator-level decision analysis — 31 metrics evaluated by default, from your quality pillars down to whether the agent dispatched the right tool call.
Gated, trustworthy verdicts
If the execution didn't complete, the verdict is capped at FAIL regardless of score. No false green passes, ever.
Platform
Everything a release gate for conversational AI needs.
Built first for conversational and voice AI. RAG evaluation is next — the judge model already includes five RAG-specific quality dimensions.
Agentic Test Personas
Model the customers who actually call you: confused, impatient, multilingual, off-script. Each persona pursues a goal instead of replaying a transcript.
Real Conversation Execution
Runs against your live conversational AI platform through the same surface your customers use — no mocks, no simulated backends.
LLM-as-Judge Metrics
Turn-level scoring for grounding, resolution, tone, policy adherence and escalation quality, with the reasoning stored alongside each score.
Deterministic Assertions
Hard checks for the things that must never be fuzzy: refund amounts, disclosure text, redaction, handoff targets and latency budgets.
Semantic Assurance
Six-construct state-transition validity model — intent integrity, context memory, dialogue state correctness, business compliance, tool decisions, and recovery — with a causal root-cause taxonomy behind every violation.
Orchestrator Quality
Scores the agent's internal decisions, not just its replies: correct tool/route dispatch, missed invocations, and decision oscillation across a conversation — weighted and traceable to the exact turn.
Execution-Integrity Gate
Incomplete, timed-out or errored runs can never be scored into a pass. Integrity is evaluated before quality, not after.
Automated Bug Report Generation
Every FAIL gets an LLM-generated root-cause report — a 5-Whys chain, severity, and duplicate detection — rendered as Jira-ready markdown, automatically, no manual write-up required.
Custom Metrics
Extend the 31-metric default catalog with your own — subclass a documented SDK, register it, and it runs alongside the built-ins with the same exception isolation and latency tracking.
Full Audit Trail
Every prompt, judge call, assertion and retry is recorded and replayable, so a verdict can be defended in a release review or an audit.
A broken conversation should never look like a passing one.
Most tools score whatever transcript they collected. If the agent stalled at turn 17, they grade the first sixteen turns and call it green. Shyena evaluates execution integrity first.
turn 17 · session terminated before goal resolution
quality score 0.81 · integrity check FAILED
verdict capped → FAIL (execution incomplete)
Scored honestly and capped. The team sees exactly which turn broke, with the full judge reasoning attached.
16 turns collected · no assertion errors raised
average score 0.81 → threshold 0.75
verdict → PASS
A false green: nothing crashed loudly, so the run reports healthy — and the regression reaches production.
Automated
No manual effort, continuous evaluation.
Measurable
Quantitative metrics that matter.
Reliable
Consistent, repeatable and unbiased.
Explainable
Transparent results with clear evidence.
Actionable
Insights that drive real improvement.
Continuous
Always learning, always improving.
Trust
Nothing here asks you to take our word for it.
We're early — no wall of customer logos, no analyst quadrant, on purpose: we'd rather get the evaluation model right first. Here's what you can actually check yourself today, not what we claim.
We use an independent, open-source red-teaming engine
Our red-teaming service runs on Ziran — an independent open-source project we have no role in developing. Read the actual detection logic on GitHub, not a description of it. github.com/taoq-ai/ziran
The metric catalog is documented, not just claimed
31 metrics run by default, 117 in the full catalog — see exactly what each one scores in the docs.
The gate is checkable on your very first run
Bring one real scenario to a free pilot. If a run breaks mid-conversation, watch it get capped at FAIL yourself — nothing to take on faith.
Resources
Field notes on testing conversational AI
Why Conversational AI Needs a Different Testing Model
The same test persona can take a different but equally valid path every run. That breaks the assumptions conventional QA is built on.
6 min read
Quality AssuranceThe Problem With Green Checkmarks on Broken Conversations
When a test runner reports success on a conversation that never reached its goal, your metrics are lying to you.
5 min read
See it evaluate your own agent
Bring one real scenario. We'll run it against your live conversational AI agent and walk through every judged turn with you.