Python SDK

View as Markdown

The Python SDK records what each AI call in your app costs, which feature made it, and which of your customers it was for. It sends ai-tally counts, costs and hashed ids. Your prompts and answers stay in your app.

Your calls always go ahead as normal. The SDK records them in the background, and if something inside it goes wrong, it logs a warning and carries on.

Install

pip install ai-tally

The package is called ai-tally, and you import it as aitally. It needs Python 3.10 or later and has no dependencies of its own.

Two ways to connect

Pick one:

tally run records your app with no code change. You put tally run -- in front of the command that starts your app.
aitally.init() is one line at the top of your app, for when you want to set things up in code.

Both record the same calls.

Run your app with `tally run` (preview)

Install the SDK in the same environment as your app, save your ai-tally API key as TALLY_KEY, and put tally run -- in front of the command that starts your app:

pip install ai-tally
export TALLY_KEY="paste-your-key-here"
tally run -- uvicorn main:app

Your app runs as before, with its AI calls recorded. Your code stays exactly as it is, and so does your app’s own output. Set TALLY_KEY wherever your app runs, such as your container settings, your platform’s config vars or your secret manager.

Once your app imports an AI library (for Bedrock, at its first call), each process writes a line to stderr saying what it records:

tally run[4211]: metering openai 2.8.1 (from first use) | account from header x-tenant-id, features from routes | eval capture off | frameworks seen: starlette, fastapi

A library or framework your app picks up after that line gets a line of its own.

In a uv, Poetry or Pipenv project, run it from the project’s environment, for example uv run tally run -- uvicorn main:app or poetry run tally run -- uvicorn main:app.

Where the start command goes

Put tally run -- in front of the command you already use to start your app. Change it in the one place that matches how you start your app today. Docker is optional.

From a shell, a script or a systemd unit:

tally run -- uvicorn main:app --host 0.0.0.0

CMD exec tally run -- uvicorn main:app --host 0.0.0.0

exec makes your server the container’s main process. It gets the stop signal directly and sends its last calls before it exits. The exec form, CMD ["tally", "run", "--", "uvicorn", "main:app", "--host", "0.0.0.0"], works the same way.

web: tally run -- gunicorn app:app
worker: tally run -- celery -A tasks worker

Any command that starts Python from the environment ai-tally is installed in works, such as python main.py, gunicorn, uvicorn, celery or python manage.py runserver. The Python processes your command starts, such as gunicorn, uvicorn and Celery workers, are recorded too.

To set a variable or change directory first, write it like this:

env VAR=1 tally run -- uvicorn main:app
cd app && exec tally run -- uvicorn main:app

What it records on its own

Model calls. The libraries and calls in What is recorded automatically: openai, anthropic, google-genai, and Bedrock through boto3. Each library is recorded from the moment your app first uses it, and your requests are sent exactly as your code wrote them. For the cost of an OpenAI stream, pass stream_options={"include_usage": True} in your call.
Features, from your routes. In FastAPI, Starlette, Flask and Django apps, each model call made while serving a request is named after the request’s matched route. The route /api/v1/tickets/{id}/summary becomes the feature api-tickets-id-summary: a leading version such as v1 is dropped, and each path parameter shows by its name.
Route details. If two routes would share a name, the method is added (-get, -post). Streamed responses, and work the request hands to a ThreadPoolExecutor or a new threading.Thread, count toward the request while it is open.
Background jobs. Each Celery task and RQ job is recorded on its own, named after the task: tasks.summarize becomes tasks-summarize. Its customer comes from the [jobs] setting below.
Your customer. Tell tally run where your app keeps the customer id, in the settings below, and each call is recorded for that customer. The id is hashed inside your process, as with aitally.with_account().

Where your code calls with_account() or sets a feature_tag itself, that value is used. With no settings at all, features come from your routes and calls are recorded without a customer.

Settings

Settings go in a tally.toml file in your app’s working directory:

[account]
from = "header"         # header, jwt_claim, request_state or none
name = "x-tenant-id"    # the header, claim or attribute that holds your customer id

[features]
ignore = ["health"]                                      # no feature name for /health
rename = { api-tickets-id-summary = "ticket-summary" }   # a shorter name for one route

[jobs]
account_from = "arg0"   # the task's first argument, or the name of a keyword argument

You can also put them in your pyproject.toml, under [tool.tally.account], [tool.tally.features] and so on. If both files are there, tally.toml is used. To use another file, set TALLY_RUN_CONFIG to its path.

Each setting also has an environment variable, which wins over the file:

Setting	Variable	Value
`[account] from`	`TALLY_ACCOUNT_FROM`	`header`, `jwt_claim`, `request_state` or `none` (the default)
`[account] name`	`TALLY_ACCOUNT_NAME`	The header, claim or attribute to read.
`[account] label`	`TALLY_ACCOUNT_LABEL`	Optional. Where a readable customer name is, read the same way.
`[features] from`	`TALLY_FEATURES_FROM`	`route` (the default) or `none`
`[features] ignore`	`TALLY_FEATURES_IGNORE`	Feature names to leave blank, such as a health check.
`[features] rename`	`TALLY_FEATURES_RENAME`	New names for features, as `old=new` pairs in the variable.
`[jobs] account_from`	`TALLY_JOBS_ACCOUNT_FROM`	`arg0`, or a keyword argument’s name.

In the variables, lists and pairs are comma-separated.

Where the customer id comes from:

header: a request header, such as x-tenant-id.
jwt_claim: a claim in the request’s Authorization: Bearer token, such as org_id, or org.id for a nested claim. tally run reads the claim to name the customer, and your app’s own auth checks the token as it does today.
request_state: a plain attribute your app sets before the model call: on request.state in FastAPI and Starlette, on flask.g in Flask, or on the request in Django. A dotted path such as user.org_id follows attributes.

A label is sent as readable text, so point it at a company or account name. Label sources that name a person (name, email, given_name, family_name) are left out.

Your app always starts. A setting tally run cannot use is named in a warning on stderr, and the section it belongs to is set aside: for example, with a problem in [account], calls are recorded without a customer. On Python 3.10, reading the file needs tomli installed. The environment variables work on every version.

See what leaves your app: `tally doctor`

tally doctor -- <your start command> runs your app for real under tally run, so run it where real requests and model calls are fine, such as staging or your laptop:

tally doctor -- uvicorn main:app

For the first five model calls of each process (--calls N for more), it prints every field that call would send, where the customer came from (for example from header x-tenant-id) and which route named the feature. Then it prints a summary. It sends to ai-tally only when you add --send.

The fields include counts, costs, model names, feature names, trace ids and your customer’s hashed id, plus the customer label if you set one. Your customer ids leave only as a hash. tally doctor --fields prints the full list from the version you have installed.

Turning it off

Set TALLY_DISABLED=1 to turn tally run off with the same start command and image. Your app then starts as it does without tally run. An aitally.init() call in your own code keeps recording. Unset it, or set it to 0, to turn tally run back on.

In your code: `aitally.init()`

Two lines connect your app. This is the same example the dashboard shows you when you create a key:

# pip install ai-tally
import aitally

aitally.init("YOUR_TALLY_KEY")
# From here your unmodified openai, anthropic, google-genai and Bedrock converse calls
# are auto-instrumented: model, tokens, and cost land in AI Tally with no per-call record_* code.

Set with_account(...) once per request to unlock cost per customer; start_trace(feature_tag=...) for cost per feature.

In your own code, leave the key out and let it come from TALLY_KEY, and say which customer and which feature each call is for:

import aitally
from openai import OpenAI

aitally.init(feature_tag="assistant")  # once, at startup; reads your key from TALLY_KEY
client = OpenAI()

def handle(request):
    # Every AI call inside this block counts toward this customer.
    with aitally.with_account(request.customer_id):
        # And toward this feature, instead of the default above.
        with aitally.start_trace(feature_tag="summarize"):
            return client.chat.completions.create(model="gpt-4o-mini", messages=request.messages)

That is all most apps need.

`aitally.init`

Call it once when your app starts.

aitally.init(key: str | None = None, *, endpoint: str | None = None, feature_tag: str | None = None, instrument: bool | Iterable[str] = True, instrument_stream_usage: bool = False, flush_interval_s: float = 1.0, catalog: PriceCatalog | None = None, allow_insecure_http: bool | None = None, replay_capture: "bool | Literal['local']" = False, replay_dir: str | os.PathLike[str] | None = None, replay_sample_rate: float | None = None, replay_retention_days: int | None = None, eval_judge_model: str | None = None) -> TallyClient

Connect the process to ai-tally in one line. Idempotent, non-blocking, never raises.

Option	Default	What it does
`key`	`None`	Your ai-tally API key. If left out, it is read from `TALLY_KEY`.
`endpoint`	`None`	Where to send records. If left out, it is read from `TALLY_ENDPOINT`, or else `https://ingest.ai-tally.com`.
`feature_tag`	`None`	The feature name used when a call has no other.
`instrument`	`True`	Which libraries to record. `True` records every one you have installed, a set such as `{"openai", "anthropic"}` records only those, and `False` records none. The names are `"openai"`, `"anthropic"`, `"google"` and `"bedrock"`.
`instrument_stream_usage`	`False`	For streamed OpenAI chat calls, ask OpenAI to report token counts. See Streaming with OpenAI.
`flush_interval_s`	`1.0`	How often, in seconds, records are sent.
`catalog`	`None`	Your own price list for the SDK’s local estimate. ai-tally works out the final cost itself either way.
`allow_insecure_http`	`None`	Lets the SDK send over plain `http://` to another machine. See Secure connections.

init() is safe to call more than once, and it does not wait on the network. It needs a key: with none set, it logs one warning and your app runs as usual.

What is recorded automatically

Library	Calls	Async	Streaming
`openai`	`chat.completions.create`, `responses.create`, `embeddings.create`	Yes	Yes
`anthropic`	`messages.create`, `messages.stream`	Yes	Yes
`google-genai`	`models.generate_content`, `models.generate_content_stream`	Yes (`client.aio`)	Yes
`boto3` (Bedrock)	`converse`, `converse_stream`	No	Yes

Both tally run and aitally.init() record these. The SDK works with the libraries you already have installed and adds none of its own. Clients made before or after init both count. For any other library, add one line after the call with record_llm_call. For LiteLLM, see the SDK’s PyPI page.

Other providers through the openai library. Calls to Groq, Fireworks, Together, DeepSeek, xAI, OpenRouter, Mistral, Azure or Gemini through the openai library are labeled with the real provider. A host the SDK does not recognize is labeled openai-compatible.
Direct calls to OpenAI and Anthropic. Once the SDK records openai, anthropic or google, a call your app sends to api.openai.com or api.anthropic.com with httpx is recorded too, as is a method such as chat.completions.parse or beta.messages.create. The SDK reads the model and token counts from the response, and each call is counted once. Set TALLY_HTTPX_GAP=0 to turn this off.
Bedrock streams are recorded once your code has read the stream to the end.
Forking servers. Workers forked after init(), such as gunicorn --preload, Celery prefork or multiprocessing with fork, each send their own calls. You call init() once, before the fork.
Calls through the ai-tally proxy are recorded by the proxy, so the SDK leaves them to it and each call is counted once.

Recording a library later: `aitally.patch`

Recording a library means importing it, so the default init() imports every supported library you have installed. To import nothing up front, call aitally.init(instrument=False), then aitally.patch("openai") once your code has imported that library. It takes the same names as instrument, is safe to call more than once, and returns False when there was nothing to record yet.

Streaming with OpenAI

OpenAI reports token counts on a stream when you ask for them with stream_options={"include_usage": True}. Add it to your streamed calls to see their cost.

aitally.init(instrument_stream_usage=True) adds it for you. This changes your request. OpenAI then ends each stream with one extra chunk whose choices list is empty. Some OpenAI-compatible hosts reached through the openai library reject the parameter. Turn it on when your stream handling skips empty chunks and every host you call accepts it.

A recorded stream works with for, next(), async for, anext(), with and close(), and is recorded once when it ends, is closed, or leaves its with block.

Cost per customer: `with_account`

aitally.with_account(account_id: str | None, *, label: str | None = None) -> Iterator[TraceContext]

Scope an account to a block without touching the trace.

Every call inside the block counts toward that customer. Set it once per request, for example in your web framework’s middleware. Under tally run, the [account] setting does this with no code.

The id is protected. The SDK turns your customer id into a hashed customer id that cannot be traced back, before anything leaves your app. ai-tally sees only the hash.
Clearing it. Pass None to clear it, for example in a background job that should not count toward any customer.
Labels. label is an optional readable name, so the dashboard can show “Northwind Traders” instead of a short hash. It is kept apart from your call records.

The SDK fetches your organization’s hashing key when init() runs. Calls made before it arrives are recorded without a customer.

Cost per feature: `start_trace`

aitally.start_trace(*, feature_tag: str | None = None, session_id: str | None = None, account_id: str | None = None, account_label: str | None = None) -> AbstractContextManager[TraceContext]

Begin a fresh trace (always a new trace_id). Returns the context manager.

Starts a new trace, which is one traced unit of work, like handling one request. Calls inside it carry its feature name. A new trace starts without a customer, so pass account_id if you want one.

For work that moves between processes, like Celery tasks, use this instead. It takes the same options plus trace_id.

aitally.with_trace_context(*, trace_id: str | None = None, feature_tag: str | None = None, session_id: str | None = None, account_id: str | None = None, account_label: str | None = None, inherit: bool = True) -> Iterator[TraceContext]

Set the trace context for the duration of the block, then restore prior values.

Recording other calls: `record_*`

Use these for calls the SDK does not record on its own. All options are named, and none of them raise errors. They take effect once init() has run.

aitally.record_llm_call(*, provider: str, model: str, usage: Usage, signals: TraceSignals | None = None, at: date | None = None, account_id: str | None = None, account_label: str | None = None, billing_mode: str | None = None) -> LlmCallResult

Record an LLM call end-to-end. Never raises.

aitally.record_embedding_call(*, provider: str, model: str, input_tokens: int, at: date | None = None, account_id: str | None = None, account_label: str | None = None, billing_mode: str | None = None) -> EmbeddingCallResult

Record an embedding call so the span lands in the gateway's embeddings bucket.

aitally.record_tool_call(*, provider: str, tool: str, cost_micro_usd: int | None = None, input_tokens: int | None = None, output_tokens: int | None = None, latency_ms: int | None = None, call_id: str | None = None, account_id: str | None = None, account_label: str | None = None) -> None

Record a tool call so the span lands in the gateway's tools cost-layer bucket.

aitally.record_vector_call(*, provider: str, index: str, operation: str, cost_micro_usd: int | None = None, record_count: int | None = None, latency_ms: int | None = None, account_id: str | None = None, account_label: str | None = None) -> None

Record a vector-DB call so the span lands in the gateway's vector cost-layer bucket.

aitally.pricing.Usage(input_tokens: int = 0, output_tokens: int = 0, cached_input_tokens: int = 0)

Usage(input_tokens: 'int' = 0, output_tokens: 'int' = 0, cached_input_tokens: 'int' = 0)

What each one is for:

record_llm_call: an AI model call through any other library, such as Bedrock invoke_model or the older google-generativeai package. For Gemini, use provider="google".
record_embedding_call: an embedding call, priced on its input tokens.
record_tool_call: a paid tool or API your feature calls, like a web search API.
record_vector_call: a vector database call, like a Pinecone query.

Costs are whole micro-dollars. 1_000_000 is $1.

Each one also takes account_id to set the customer for just that call, which suits a batch job that works through many customers.

Other functions

aitally.flush(timeout: float = 5.0) -> None

Drain buffered spans to the gateway synchronously (bounded). Safe no-op before init.

aitally.uninstrument() -> None

Reverse all provider patches and tear down the process-global client.

aitally.get_client() -> TallyClient | None

The process-global client, or None before init.

aitally.hash_account(account_id: str, *, key: str | None = None, endpoint: str | None = None) -> str

Return the HMAC-SHA256 hex of account_id under the tenant's active key.

aitally.flush: sends everything waiting right now. The SDK also sends what it has buffered when your process exits. Call this to send sooner, for example at the end of a short script.
aitally.uninstrument: turns the SDK off and removes its hooks, which helps in tests.
aitally.get_client: returns the running SDK client, or None before init.
aitally.hash_account: returns the hashed customer id for the proxy’s X-Tally-Account-Id-Hash header. It fetches your organization’s hashing key, so unlike everything else here, it raises an error when it cannot get that key, for example when there is no API key.

Secure connections

The SDK sends your key with every request, so it connects over https://. Plain http:// works for your own machine (localhost or 127.0.0.1), so local testing works as usual. For http:// to another machine, such as one on your network, pass allow_insecure_http=True to init() or set TALLY_ALLOW_INSECURE_HTTP=1. Under tally run, set a different address with TALLY_ENDPOINT.

The SDK sends to the exact address you give it, so give it the final one.

How it keeps your app running

Your call runs first. Records go into a queue in memory, and a background thread sends them.
Errors inside the SDK are caught. Your AI call goes ahead as normal, and errors from the AI provider reach your code unchanged.
Sends are retried. When ai-tally is slow or unreachable, each send is tried up to five times, and each record is counted once.
The queue is capped. While ai-tally is unreachable, the queue holds up to 10,000 records, then makes room by dropping the oldest and logs a warning.
A refused key gets a warning. When ai-tally refuses a wrong key or a read key, the SDK logs a warning and moves on without retrying.

Command line

The SDK comes with a tally command. It reads TALLY_KEY, or else the key tally login saved.

tally run -- uvicorn main:app     # starts your app with its AI calls recorded (preview)
tally doctor -- uvicorn main:app  # runs your app and prints what its first calls would send
tally login                       # saves your key, hidden as you type, to ~/.config/tally/config.ini
tally logout                      # removes the saved key
tally health                      # what your key can do, and model calls in the last 24 hours
tally keys                        # your organization's keys, without the secrets (needs a write or admin key)
tally test-event                  # sends one empty test event and shows what arrived
tally spend --month 2026-10       # that month's spend by feature and provider
tally --version                   # the installed SDK version

TALLY_KEY wins over the saved key, so CI needs no config file.
tally run and tally doctor read TALLY_KEY. aitally.init() reads the key you pass, or else TALLY_KEY. The key tally login saves is for the other commands.
--json. health, keys, test-event and spend take --json.
Your key stays hidden in all output.

Supported versions

The SDK supports the last two major versions of each library it records. Each release is tested on Python 3.10 to 3.13, at the oldest and the newest release of each:

Library	Supported majors	Oldest tested	Newest tested
`openai`	2.x, 3.x	2.0.0	The newest release
`anthropic`	1.x, 2.x	1.0.0	The newest release
`google-genai`	2.x, 3.x	2.0.0	The newest release
`boto3`	1.x	1.34.116, the first with Bedrock Converse	The newest release
`fastapi`	0.x	0.115.0	The newest release
`starlette`	0.x, 1.x	0.37.2, the oldest FastAPI 0.115.0 accepts	The newest release
`flask`	2.x, 3.x	2.0.0	The newest release
`django`	5.x, 6.x	5.0	The newest release
`celery`	5.x	5.0.0	The newest release
`rq`	2.x, 3.x	2.0.0	The newest release

Upgrading

Upgrade ai-tally like any other library your app depends on:

pip install -U ai-tally

Or bump it in your lockfile, or let Dependabot or Renovate open the pull request for you. To see whether a newer version is out:

pip list --outdated

The changelog lists what each release changes and the library versions it supports.

Nothing showing up? See Check it works.
See how each call is priced: Prices and blanks.