GH Repository · Jwuthri
Tracely-ai
Trace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.
- stars
- 1,405
- 30-day movement
- +19264/day
- Related entries
- 77
- Connections
- 4
Pythonpython/uvtracingself-hostedopentelemetrymonitoringmcpmakedockerai-agentsci-cdllm-evaluationllmopsevalsllm-observabilityagentpythonclickhousellmmcp-serverllm-opsagent-observabilityevaluationllm-as-judge
Tracely-ai is an MCP server for trace-native CI/CD of AI agents: it detects failures in production traces, clusters them, freezes them into hermetic regression cases, and replays them in CI so failures block the PR. It is a self-hosted Python toolchain (uv, Docker, Make) with tracing via OpenTelemetry, ClickHouse storage, and llm-as-judge evaluation.
Reach for it when you want real production agent failures turned into free, automated regression tests that gate code changes before merge.
Use it to
- Turn production trace failures into regression test cases
- Replay frozen failure cases in CI to block bad PRs
- Auto-detect and cluster agent failures from traces
- Observe and evaluate agent behavior with llm-as-judge
- Self-host tracing and evals on ClickHouse
For Teams shipping AI agents with CI/CD pipelines
- Role
- mcp-server
- Language
- Python
- Licence
- MIT
- Forks
- 172
- Open issues
- 7
- Last push
- 2026-09-16
- Skills shipped
- 17
topicsai-agentsci-cdevalsllm-observabilitytracingself-hosted