Open-source security and quality testing for AI agents. Fail CI on forbidden actions, canary leaks, missing approvals, and degraded behavior—then keep every run on infrastructure you control.
No login · Sample data · Fully interactive
Prefer managed hosting? Join the Cloud waitlist →
import tokensurf as ts @ts.tool def web_search(query: str): ... @ts.track # capture the agent's complete trajectory def research_agent(question: str): sources = web_search(question) return synthesize(question, sources)
Explore real dashboard workflows with populated projects, agent traces, security scores, and x402 economics. Click around before installing anything.
Turn an agent run into a testable trajectory, then fail CI when its tools, spend, security, or quality regress.
import tokensurf as ts @ts.tool def web_search(query: str) -> list[str]: return search_the_web(query, limit=3) @ts.track # capture output, tool calls, cost, and timing def research_agent(question: str) -> str: sources = web_search(question) return synthesize(question, sources) data = ts.Dataset.from_list([ {"input": "What changed in the latest Python release?"}, {"input": "Compare two approaches to agent memory."}, ]) report = ts.evaluate(task=research_agent, data=data, scorers=[ ts.ToolSequence(expected=["web_search"]), ts.StepBudget(max_steps=6), ts.NoCanaryLeak("TS_CANARY_test-only"), ]) ts.assert_eval(report, min_pass_rate=1.0) # fail CI
A prompt, model, or tool change can degrade the answer, bypass an approval, or expose data without throwing a single exception.
A model swap or prompt edit passes your unit tests but quietly degrades answer quality. Nothing errors — it just gets worse.
The agent calls a forbidden tool or performs a sensitive action before approval. The final answer can still look perfectly normal.
Protected test data reaches the final response or a tool input. Without a regression test, you discover it after deployment.
Decorate behavior, run cases, get one pass/fail report. Tool calls and approvals fit the same eval flow.
Use @ts.track, @ts.tool, and @ts.approval. Ordinary Python functions become one testable trace.
Mix security, economics, deterministic, reference, trajectory, and LLM-judge scorers. Attack prompts are ordinary dataset cases.
A pass/fail report in CI blocks unsafe or degraded behavior. Send the same run to the self-hosted UI when you want history.
Framework-agnostic — works with LangChain, LlamaIndex, or plain Python. Read the security guide →
Security and economics checks are native TokenSurf scorers—not separate tools. Mix them with quality and trajectory checks in the same CI gate.
Fail on forbidden tool calls, synthetic canary leaks, and sensitive actions without prior approval. No model call and no cloud service required.
Score accuracy, completeness, relevance, and helpfulness on a 1–10 scale (normalized to 0–1). Provider-agnostic via litellm — judge with any model.
Fast, free, and repeatable — no model call. Assert the structural facts a correct run must satisfy.
Embedding similarity against a known-good answer — catch drift when there's no single exact string to match.
Most tools score only the final answer. These grade the entire multi-step trajectory — the tool calls, their order, and how the agent recovers.
Test payment cost, count, and recipient allowlists alongside the behavior that caused the spend.
The Python library works by itself. The open-source server adds durable run history and a dashboard on your infrastructure.
Start without deploying a service.
Keep the results your team needs to inspect.
Not just a library — a self-hostable platform. The SDK runs evals; the server stores every run, shows trends over time, and is where you configure everything. Your data never leaves your infrastructure.
Run the server, the dashboard, and a Postgres database yourself. Your eval data and your agents' inputs and outputs stay inside your trust boundary — the answer to the data-residency objection.
docker compose up brings up the app and Postgres together as the easy path. Prefer your own setup? It's a standard service that runs against any Postgres you already have.
Every eval run is stored and charted over time, organized by project — one project per repo or app. See pass-rate trends, not just the last run.
Ping Slack, email, or a webhook when quality regresses — so a bad run finds you, not your users.
Set thresholds like "fail the build if pass-rate < 90%." Wire eval results straight into CI.
Configure scorers and your own judge keys once on your server; CI pulls the config centrally.
The server & dashboard are open source and self-hostable today. The Python SDK is the runner that feeds them.
Self-host free, or join the cloud waitlist. No per-request fees, no lock-in.
Self-host the whole platform. Free forever, Apache-2.0.
Hosted eval storage and dashboard, managed for your team. Pricing at general availability.
Enterprise needs — SSO, on-prem support, custom scorers?
Talk to usExplore the dashboard with sample data, then self-host the complete open-source platform when you are ready.
import tokensurf as ts @ts.track # flight recorder for the whole run def research_agent(question): sources = web_search(question) return synthesize(question, sources) report = ts.evaluate(task=research_agent, data=data, scorers=[ ts.ToolSequence(expected=["web_search"]), ts.StepBudget(max_steps=6), ts.NoCanaryLeak("TS_CANARY_test-only"), ]) ts.assert_eval(report, min_pass_rate=1.0) # fail CI
Agent security, quality testing, and self-hosting—answered.
@ts.track and its behavior with @ts.tool and @ts.approval. Run ts.evaluate() with security and quality scorers, then use ts.assert_eval() to fail CI when a check regresses.@track decorator; it never sits between your app and the model, so it can't add latency or a point of failure to production.