TokenSurf Documentation

TokenSurf is an open-source testing platform for AI agents: capture behavior, enforce security, economics, and quality expectations locally or in CI, and inspect every run in a dashboard you host.

Overview

Agent systems regress quietly. A prompt tweak, model upgrade, tool change, or payment integration ships, and behavior changes without a failing test. TokenSurf gives agents the test harness ordinary code already has:

• A capture SDK — @track, @tool, @approval, span(), and record_payment() record structured behavior without proxying model traffic.
• A scoring engine — 20 built-in scorers across six families grade quality, security, economics, references, and complete agent trajectories.
• An offline eval harness — turn a dataset of cases into pass/fail verdicts you can enforce in CI (“pytest for agents”).

The optional self-hosted platform adds FastAPI, Postgres, and a login-protected dashboard. It stores run history, scorer health, security regressions, x402 payment activity, quality gates, notifications, encrypted judge keys, and an audit log on infrastructure you control.

Quickstart

Define a Python file with a task, data, and scorers. This example is fully offline and needs no account, server, or provider key:

import tokensurf as ts

@ts.tool
def lookup(question):
    return "Paris" if "France" in question else "I don't know"

def agent(question):
    return lookup(question)

data = ts.Dataset.from_list([
    {"id": "q1", "input": "capital of France", "expected": "Paris"},
])

scorers = [
    ts.ExactMatch(),
    ts.ToolCalled(name="lookup"),
    ts.ForbiddenToolCalled("shell"),
]

report = ts.evaluate(task=agent, data=data, scorers=scorers)
ts.assert_eval(report, min_pass_rate=1.0)

Installation

TokenSurf v0.1.0 requires Python 3.11 or newer. It is pre-1.0 and not yet published to PyPI, so install it from the repository with uv. The SDK, eval harness, CLI, deterministic checks, and local reporting need no server. litellm ships with the framework for judge and embedding scorers; HTTP push support is the optional push extra and is included by the workspace’s uv sync.

# clone and install
git clone https://github.com/Cem-Bas/TokenSurf-Open.git
cd TokenSurf-Open
uv sync

# scaffold a starter project (example evals + a pytest CI gate)
uv run tokensurf init my-agent-tests

No account required for the framework itself — tokensurf eval runs entirely in your CI or locally. Full walkthrough: Quickstart.

Two packages ship together:

Capture with @track

@ts.track wraps any callable and records its execution as a Trace: timing, input, output, errors, and a tree of typed spans. It is framework-agnostic (works with any agent library or raw LLM calls) and best-effort — a capture failure never breaks your agent.

import tokensurf as ts

@ts.track                              # bare, or @ts.track(name="agent", sink=...)
def agent(question):
    with ts.span("retrieval", type="tool", input=question) as sp:
        sp.output = search(question)          # attach step output
    with ts.span("generate", type="llm"):
        return generate(question, sp.output)

Span type is one of llm, tool, agent, or custom. Nested @track calls are transparent — the outermost frame owns the trace, so spans from inner helpers all land on one trajectory. ts.current_trace() returns the active trace when you need it.

Traces can be written to a sink for offline inspection — JSONLSink(path) or SQLiteSink(path) ship in the box — but during an eval the harness captures them for you, so most code never touches a sink directly.

Tools & approvals

@ts.tool turns an ordinary function call into a typed tool span. It records arguments, return value, timing, attributes, and exceptions when a trace is active; outside a trace the function behaves normally.

@ts.tool(name="docs.search", attributes={"network": False})
def search_docs(query, limit=3):
    return search(query, limit=limit)

@ts.approval records whether an approval function authorized a protected action. Each granted approval authorizes one later call to the named tool; denied, late, and already-consumed approvals do not count.

@ts.approval(for_tool="send_email")
def confirm_send(request):
    return request["approved"]

@ts.tool
def send_email(body):
    ...

Datasets & evaluate()

A Dataset is a list of cases. Build one from Python, JSONL, or CSV:

data = ts.Dataset.from_list([{"id": "c1", "input": "...", "expected": "..."}])
data = ts.Dataset.from_jsonl("cases.jsonl")   # one JSON object per line
data = ts.Dataset.from_csv("cases.csv")       # header row = field names

Each case has an id, an input (passed to your task), an optional expected, and optional metadata. evaluate() runs your task over every case, captures the trace, and applies each scorer:

report = ts.evaluate(
    task=agent,              # the callable under test
    data=data,
    scorers=[ts.ExactMatch(), ts.NoLoops()],
)

report.pass_rate()           # fraction of passing scores (0.0-1.0)
report.pass_rate("ExactMatch")    # for one scorer
report.mean_score()          # mean numeric value, or None
report.error_count()         # scorer/task errors

Gate your CI

assert_eval turns a report into a pytest assertion — drop it in your test suite and a quality regression fails the build like any other test.

# test_agent_quality.py
import tokensurf as ts
from myapp import agent, cases

def test_agent_quality():
    report = ts.evaluate(task=agent, data=cases, scorers=[
        ts.ExactMatch(),
        ts.ForbiddenToolCalled("shell"),
    ])
    ts.assert_eval(report, min_pass_rate=0.9)

Or run an eval file directly from the command line and write JSONL results (see the CLI):

tokensurf eval run eval.py -o results.jsonl

The six scorer families

TokenSurf ships 20 built-in scorers across six families. Every scorer grades one trace and returns a ScoreResult with a normalized 0–1 value, pass/fail verdict, explanation, and optional cost, latency, model, or error metadata.

Scorer exceptions become errored results instead of aborting the suite. Binary checks return 1.0 or 0.0; judges and embeddings produce graded values. All concrete scorer classes are available from import tokensurf as ts.

Deterministic scorers

Fast, free, reproducible — no model in the loop. Great as the first line of defense.

Security testing

Security checks are deterministic regression tests over observed agent behavior. They run locally without a model call and use the same datasets, reports, CLI, gates, and dashboard as every quality evaluation.

security_scorers = [
    ts.ForbiddenToolCalled({"shell", "delete_user"}),
    ts.NoCanaryLeak("TS_CANARY_test-only"),
    ts.ApprovalRequired("send_email"),
]

report = ts.evaluate(task=agent, data=attack_cases, scorers=security_scorers)
ts.assert_eval(report, min_pass_rate=1.0)

Economics & x402 payments

TokenSurf can treat spending as tested agent behavior. Call record_payment() after your payment client completes an x402 or other settlement flow, then enforce USD budgets, settlement counts, and recipient allowlists in local tests or CI. TokenSurf observes payment results; it does not hold wallets, sign payments, or proxy payment traffic.

def buy_resource(client, url):
    response = client.get(url)  # your client handles 402 + settlement
    settlement = response.payment_response

    ts.record_payment(
        protocol="x402",
        amount=settlement.amount,       # native amount / base units
        amount_usd=0.025,               # conversion supplied by your app
        asset=settlement.asset,
        network=settlement.network,
        recipient=settlement.pay_to,
        payer=settlement.payer,
        success=settlement.success,
        transaction=settlement.transaction,
    )
    return response.json()

amount and amount_usd are intentionally separate. TokenSurf never guesses exchange rates or token decimals, so native amounts from unlike assets are never silently added as dollars.

economics_scorers = [
    ts.PaymentCostUnder(usd=0.10),
    ts.PaymentCountAtMost(max_payments=3),
    ts.PaymentRecipientsAllowed({"0xmerchant"}),
]

Push the report to TokenSurf Server and the Economics dashboard summarizes settled USD, successful and failed payments, unpriced settlements, spend by project, and recent attempts. It is an eval cost tracker, not an accounting ledger; balances, refunds, confirmation, and reconciliation remain the responsibility of the payment network and wallet.

LLM judge

LLMJudge asks a model to rate the response against your criteria on a 1–10 rubric, normalized to 0–1. It is provider-agnostic through litellm, but requires an explicit client. Without one it returns an errored result rather than attempting a hidden network call.

from tokensurf.scorers.llm import LiteLLMClient

ts.LLMJudge(
    criteria="answers the question accurately and completely",
    model="gpt-4o-mini",
    client=LiteLLMClient(),
    threshold=0.7,          # pass when normalized score >= threshold
)

Pass your own compatible client for full control, or a custom prompt with {criteria}, {input}, and {output} placeholders. Judge keys can come from normal provider environment variables or be pulled from your server at eval time (see Centralized config).

Reference-based

When you have a labeled “golden” answer, EmbeddingSimilarity scores the cosine similarity between the output and case.expected — objective grading without exact-string brittleness.

ts.EmbeddingSimilarity(
    model="text-embedding-3-small",
    threshold=0.8,
)

litellm is a core framework dependency, so no embeddings extra is required. Supply a compatible embedding client or configure the provider for the selected model. Missing references or provider failures become errored results.

Agent trajectory

The differentiator: grade how the agent worked, not just the final answer. These read the span tree captured by @track.

Custom scorers

A scorer is a class with a unique name and a keyword-only score() method that returns a normalized ScoreResult. Subclass Scorer and use it like any built-in. Registry decoration is optional for private scorers, and async score() is supported.

from tokensurf import Scorer, ScoreResult

class MentionsPolicy(Scorer):
    name = "MentionsPolicy"

    def score(self, *, trace, case=None):
        ok = "refund policy" in str(trace.output).lower()
        return ScoreResult(scorer=self.name, value=1.0 if ok else 0.0, passed=ok)

The self-hosted platform

The framework produces runs; tokensurf-server remembers them. It is a FastAPI service backed by Postgres, with a Bearer-authenticated ingest API and a login-protected admin dashboard. Docker Compose starts Postgres 16, runs migrations, and serves the app on port 8000:

git clone https://github.com/Cem-Bas/TokenSurf-Open.git
cd TokenSurf-Open
docker compose up -d

# read the one-time first-run proof from inside the app container
docker compose exec app cat tokensurf_setup_token

Open http://localhost:8000/setup, paste the setup token, and create the first admin. Until an admin exists, every dashboard page redirects to the setup wizard. The server logs the token file path, not the raw token; once the first admin exists, /setup redirects to login permanently.

Create a project and a scoped ingest key. The raw tsk_... key is printed once; only its SHA-256 hash and display prefix are stored:

docker compose exec app uv run tokensurf-server create-project "My Agent"
docker compose exec app uv run tokensurf-server create-key my-agent --label ci

Point the eval CLI at the server and the run appears in the dashboard:

tokensurf eval run eval.py \
  --server https://tokensurf.internal \
  --key tsk_your_project_key       # run appears in your dashboard

Projects & run history

Organize runs by project, usually one per repo or agent. The Projects view shows stored-run counts, latest pass rate, and trends. The Runs view adds project and quality-gate filters. Run detail shows every case, normalized score, gate verdict, captured tool trajectory, errors, cost, and latency.

Scorer dashboard

The Scorers page is an operational dashboard, not a reference page. It aggregates every stored result into scorer count, total results, overall pass rate, and failed/error totals. A health table sorts the weakest scorer pass rates first, with mean score, result count, failures, errors, and last-seen time; Recent problems links regressions back to the project and run that produced them.

Economics dashboard

The Economics page reads payment spans already stored inside eval traces—no new ledger or database migration. It shows total settled USD, successful and failed payment counts, unpriced settlements, spend by project, and the latest attempts with protocol, network, recipient, asset, and transaction metadata. Payment data stays inside your TokenSurf deployment.

Quality gates

A quality gate is a per-project threshold evaluated automatically on every run the server ingests. Define gates like “pass-rate ≥ 90%” or “ExactMatch pass-rate ≥ 95%”; a run that breaches one is flagged in the dashboard and triggers notifications.

Comparisons are gte, gt, lte, or lt. Gate evaluation is best-effort: persisted results annotate the run and drive alerts, but an evaluation or delivery failure never rolls back ingestion. Runs can be filtered by passed or failed gate status.

Notifications

When a run breaches a gate or contains errored scores, the platform fires every enabled project channel:

• Slack — an incoming-webhook message to a channel.
• Webhook — a JSON POST to any endpoint you control.
• Email — via your own SMTP server.

Slack and generic webhook URLs are encrypted at rest and never rendered back. Email uses the server’s TOKENSURF_SMTP_* environment settings plus the channel’s recipient address. Settings offers Test and Delete actions. Delivery is best-effort; each real attempt is logged with success or only the exception class name, never an error message that could expose a secret URL.

Centralized config

Rather than setting judge-provider keys in every CI job, the platform can store them once (encrypted at rest) so your eval runs pull them at run time with the project key they already use. The keys stay on your server, in your database — TokenSurf never holds or proxies them for you.

When tokensurf eval run has both --server and --key, it calls GET /api/v1/config before evaluation, maps openai, anthropic, and gemini keys to their normal environment variables, and never overwrites a variable already set locally. Pass --no-config-pull to disable this.

• Secrets are Fernet-encrypted with a key derived from TOKENSURF_SECRET_KEY; missing-key failures are loud and plaintext is never stored.
• The response sets Cache-Control: no-store and is rate-limited per project (default 30/60).
• Every successful pull records a config.pull audit event with key prefix, IP, and key count—never secret values.

Server security

The dashboard uses signed HttpOnly, SameSite=Lax session cookies and PBKDF2-HMAC-SHA256 password hashes. State-changing forms use signed double-submit CSRF tokens, including login and logout. The ingest API uses project-scoped Bearer keys stored only as SHA-256 hashes.

TLS, HSTS/CSP/frame headers, request-body limits, gateway-wide rate limits, trusted proxy headers, database backups, and egress restrictions remain deployment responsibilities. See the complete server security model before exposing TokenSurf beyond localhost.

Open source vs Cloud

The complete framework and self-hosted platform are Apache-2.0. Run them yourself for free, or join the early-access Cloud waitlist for a managed deployment.

CLI

The tokensurf command scaffolds projects, runs and reports evals, and—when tokensurf-server is installed—exposes server administration. An eval file defines module-level task, data, and scorers.

Key options for eval run:

# local run, then the same run pushed to your dashboard
tokensurf init my-agent-tests
tokensurf eval run eval.py
tokensurf eval run eval.py --server https://tokensurf.internal --key tsk_... --label main

Full reference pages

This page is the maintained public overview. The repository contains the complete, code-reviewed reference set: