Skip to main content

Overview

The OpenHands SDK provides metrics tracking at two levels: individual LLM metrics and aggregated conversation-level costs:
  • You can access detailed metrics from each LLM instance using the llm.metrics object to track token usage, costs, and latencies per API call.
  • For a complete view, use conversation.conversation_stats to get aggregated costs across all LLMs used in a conversation, including the primary agent LLM and any auxiliary LLMs (such as those used by the context condenser).

Getting Metrics from Individual LLMs

A ready-to-run example is available here!
Track token usage, costs, and performance metrics from LLM interactions:

Accessing Individual LLM Metrics

Access metrics directly from the LLM object after running the conversation:
The llm.metrics object is an instance of the Metrics class, which provides detailed information including:
  • accumulated_cost - Total accumulated cost across all API calls
  • accumulated_token_usage - Aggregated token usage with fields like:
    • prompt_tokens - Number of input tokens processed
    • completion_tokens - Number of output tokens generated
    • cache_read_tokens - Cache hits (if supported by the model)
    • cache_write_tokens - Cache writes (if supported by the model)
    • reasoning_tokens - Reasoning tokens (for models that support extended thinking)
    • context_window - Context window size used
  • costs - List of individual cost records per API call
  • token_usages - List of detailed token usage records per API call
  • response_latencies - List of response latency metrics per API call
For more details on the available metrics and methods, refer to the source code.

Reported costs and custom model names

For Chat Completions and Responses API calls, the SDK selects one charge per response in this order:
  1. A valid provider cost header (llm_provider-x-litellm-response-cost in the assembled response’s additional headers), including a zero charge.
  2. A positive response usage.cost, unless both custom token rates are configured. This lets custom model aliases report charges without a static pricing entry.
  3. LiteLLM’s cost calculator, using explicit input_cost_per_token and output_cost_per_token when both are configured, or available model pricing.
Reported costs must be finite amounts in USD. Headers may report zero; usage fallback requires a positive charge. A zero usage value leaves existing price-map calculation in place because some gateways use zero when pricing is unavailable. Header and usage values are alternative representations of a charge; the SDK does not add them together. A valid zero header takes precedence over a positive usage value. Streaming providers must deliver the cost in usage that survives final response assembly. A confirmed zero appears as a zero-valued entry in llm.metrics.costs. If no reported cost or usable pricing is available, the SDK emits a cost-calculation warning and adds no cost entry. An accumulated total of zero alone therefore does not establish that every call was free. The SDK cannot infer separately billed routing or model-selection calls from the final model’s token counts. Those charges must be included in an agreed reported total or recorded separately by the application.

Prompt Cache Support

The SDK enables automatic prompt cache markers when caching_prompt=True and model capability detection reports support. It uses LiteLLM model metadata when available, with a built-in capability list as a fallback. That fallback includes Claude Haiku 5.5 and Opus 5.5, including Anthropic and LiteLLM-proxy model names. Set caching_prompt=False when constructing LLM to disable automatic cache markers. After a provider call, inspect cache_write_tokens and cache_read_tokens to check reported cache activity. Actual cache use depends on the provider’s requirements, including its minimum cacheable prompt length.

Ready-to-run Example (LLM metrics)

This example is available on GitHub: examples/01_standalone_sdk/13_get_llm_metrics.py
examples/01_standalone_sdk/13_get_llm_metrics.py
You can run the example code as-is.
The model name should follow the LiteLLM convention: provider/model_name (e.g., anthropic/claude-sonnet-4-5-20250929, openai/gpt-4o). The LLM_API_KEY should be the API key for your chosen provider.
ChatGPT Plus/Pro subscribers: You can use LLM.subscription_login() to authenticate with your ChatGPT account and access Codex models without consuming API credits. See the LLM Subscriptions guide for details.

Using LLM Registry for Cost Tracking

A ready-to-run example is available here!
The LLM Registry allows you to maintain a centralized registry of LLM instances, each identified by a unique usage_id. This is particularly useful for tracking costs across different LLMs used in your application.

How the LLM Registry Works

Each LLM is created with a unique usage_id (e.g., “agent”, “condenser”) that serves as its identifier in the registry. The registry maintains references to all LLM instances, allowing you to:
  1. Register LLMs: Add LLM instances to the registry with llm_registry.add(llm)
  2. Retrieve LLMs: Get LLM instances by their usage ID with llm_registry.get("usage_id")
  3. List Usage IDs: View all registered usage IDs with llm_registry.list_usage_ids()
  4. Track Costs Separately: Each LLM’s metrics are tracked independently by its usage ID
This pattern is essential when using multiple LLMs in your application, such as having a primary agent LLM and a separate LLM for context condensing.

Ready-to-run Example (LLM Registry)

This example is available on GitHub: examples/01_standalone_sdk/05_use_llm_registry.py
examples/01_standalone_sdk/05_use_llm_registry.py
You can run the example code as-is.
The model name should follow the LiteLLM convention: provider/model_name (e.g., anthropic/claude-sonnet-4-5-20250929, openai/gpt-4o). The LLM_API_KEY should be the API key for your chosen provider.
ChatGPT Plus/Pro subscribers: You can use LLM.subscription_login() to authenticate with your ChatGPT account and access Codex models without consuming API credits. See the LLM Subscriptions guide for details.

Getting Aggregated Conversation Costs

Beyond individual LLM metrics, you can access aggregated costs for an entire conversation using conversation.conversation_stats. This is particularly useful when your conversation involves multiple LLMs, such as the main agent LLM and auxiliary LLMs for tasks like context condensing.
examples/01_standalone_sdk/21_generate_extraneous_conversation_costs.py
You can run the example code as-is.
The model name should follow the LiteLLM convention: provider/model_name (e.g., anthropic/claude-sonnet-4-5-20250929, openai/gpt-4o). The LLM_API_KEY should be the API key for your chosen provider.
ChatGPT Plus/Pro subscribers: You can use LLM.subscription_login() to authenticate with your ChatGPT account and access Codex models without consuming API credits. See the LLM Subscriptions guide for details.

Understanding Conversation Stats

The conversation.conversation_stats object provides cost tracking across all LLMs used in a conversation. It is an instance of the ConversationStats class, which provides the following key features:

Key Methods and Properties

  • usage_to_metrics: A dictionary mapping usage IDs to their respective Metrics objects. This allows you to track costs separately for each LLM used in the conversation.
  • get_combined_metrics(): Returns a single Metrics object that aggregates costs across all LLMs used in the conversation. This gives you the total cost of the entire conversation.
  • get_metrics_for_usage(usage_id: str): Retrieves the Metrics object for a specific usage ID, allowing you to inspect costs for individual LLMs.

Conversation Budgets

Set max_budget_per_run when creating a conversation:
The Agent Server also accepts max_budget_per_run in POST /api/conversations. The value is a finite, positive USD amount; omitting it or sending null leaves the conversation unbounded. Despite the option’s name, the existing SDK limiter compares the limit with accumulated conversation cost across all tracked LLMs, including the condenser. Starting another run does not reset that cost. The SDK persists the limit in base_state.json, alongside conversation state; the server’s meta.json does not store it. Both standalone SDK resume and server restart restore the saved limit. A fork keeps the limit and resets cost by default; reset_metrics=false preserves cost. When reopening a local conversation, omitting max_budget_per_run or passing None keeps its saved limit. Passing a number replaces it. Set conversation.max_budget_per_run = None on the local conversation to clear and persist the limit. The limit is checked after each agent step, except while waiting for tool confirmation. An LLM call or step can overshoot the budget. If the limit is reached and the agent still has work to do, the check stops the run with execution_status="error" and a ConversationErrorEvent whose code is MaxBudgetReached. A step that finishes the task keeps finished status, including when it overshoots.

Displaying Budget Usage

Read the limit from max_budget_per_run in the create response or GET /api/conversations/{id}. Sum the accumulated_cost values under stats.usage_to_metrics to get the spend compared with that limit:
While the conversation runs, ConversationStateUpdateEvent messages with key="stats" carry updated metrics. Replace the cached stats with the event’s value and recalculate the sum. A state update with key="max_budget_per_run" carries a changed limit. REST and base_state.json reflect completed steps; an in-flight call’s cost is available only after usage is reported. Hide the percentage when the limit is absent or null. It can exceed 100%: keep the actual spend visible even if the progress bar is capped visually. The legacy per-LLM max_budget_per_task metric is not this conversation limit.

Next Steps

  • Context Condenser - Learn about context management and how it uses separate LLMs
  • LLM Routing - Optimize costs with smart routing between different models