# llm-cost-reduction Adopt this skill: https://proskills.md/skills/other/build-with-ai-hq-llm-cost-reduction Page: https://proskills.md/skills/other/build-with-ai-hq-llm-cost-reduction This document is the public skill bundle for that page. It contains SKILL.md and every helper file the catalog carries. No login is required. ## File: SKILL.md size: 20252 sha: 3fb44c29b25d14f60fa0292fd775f79d748ca44e upstream: https://github.com/build-with-ai-hq/llm-cost-reduction/blob/HEAD/SKILL.md ````text --- name: llm-cost-reduction description: Find and cut LLM/AI API spend in applications and agents without breaking quality. Measures cost per feature and per request first, then right-sizes models per task, routes simple requests to cheap models and hard ones to frontier models, trims bloated prompts and RAG context, turns on provider prompt caching with stable prefixes, adds exact and semantic response caching, moves offline work to Batch APIs, caps max_tokens and agent loops, reuses embeddings, and sets per-team and per-feature budgets and cost attribution. Use when someone says "our OpenAI/Anthropic bill spiked", "surprise LLM bill", "reduce token costs", "LLM cost optimization", "everything goes to GPT/Claude Opus", "which feature is burning tokens", "prompt caching isn't hitting", "cache LLM responses", "the agent looped and cost $X", "set a budget per team", "route cheap queries to a smaller model", "model routing by complexity", or "track AI spend by customer". Gives Python/TypeScript fixes, a grep audit, and gateway controls with Bifrost. license: MIT metadata: author: build-with-ai-hq version: "1.0.0" --- # LLM Cost Reduction Cut what an AI application spends on model calls while holding output quality steady. LLM cost is `requests x (input tokens x input price + output tokens x output price)`. Every fix below attacks one factor: fewer requests (caching, dedup, loop caps), fewer tokens (trimming, output limits), or a lower price per token (smaller models, prompt-cache discounts, batch discounts). Work in this order: **measure, then fix the biggest line item, then verify quality did not drop.** Optimizing without per-feature numbers usually trims the wrong thing. ## When to use - The provider invoice jumped and nobody can say which feature, team, or customer caused it. - Every request, including classification, extraction and short chat turns, goes to a frontier model. - The same or near-identical prompts are sent repeatedly (FAQ bots, retries, dashboards, evals). - System prompts, tool schemas, few-shot examples or RAG chunks run to thousands of tokens per call. - `max_tokens` is unset, agents run without an iteration or spend cap, or retries stack up. - Provider prompt caching is available but `cached_tokens` / `cache_read_input_tokens` is zero. - Nightly or backfill jobs call the synchronous API one item at a time. - Embeddings are recomputed for unchanged documents on every ingest. ## Quick diagnosis (run first) Run from the repo root. Each hit is a lead; read the surrounding code. ```bash g() { grep -rnE "$@" --exclude-dir={node_modules,.git,dist,build,.venv,venv,__pycache__} . ; } # 1. Every LLM call site and which models they use g "chat\.completions\.create|responses\.create|messages\.create|generateContent|generate_content|ChatOpenAI|ChatAnthropic|streamText|generateText|embeddings\.create" g -o "\"(gpt-[0-9a-z.-]+|o[1-9][a-z0-9-]*|claude-[a-z0-9.-]+|gemini-[0-9a-z.-]+)\"" | sort | uniq -c | sort -rn # 2. Output limits: calls with no max_tokens / max_output_tokens / maxTokens nearby g "max_tokens|max_completion_tokens|max_output_tokens|maxTokens|maxOutputTokens" | wc -l # 3. Agent / tool loops without an obvious cap g "while True|while \(true\)|for \(;;\)|max_iterations|maxSteps|max_turns|recursion_limit|stopWhen" # 4. Retry wrappers (retry storms multiply spend) g "max_retries|maxRetries|tenacity|backoff|p-retry|retry\(" # 5. Is usage being recorded at all? g "usage\.(prompt_tokens|completion_tokens|input_tokens|output_tokens)|usageMetadata|cached_tokens|cache_read_input_tokens" # 6. Prompt-cache killers: timestamps, UUIDs, or per-user data at the top of prompts g "(system|instructions).{0,80}(datetime\.now|Date\.now|new Date\(|uuid|time\.time)" # 7. Embeddings recomputed on every run g "embed(_documents|Documents|dings\.create)" ``` Then get numbers. If the app already logs usage, group it; if not, add the instrumentation in step 1 below and collect a day of traffic. You need, per feature: requests/day, average input tokens, average output tokens, model, and cost. Rank features by cost. The top one or two usually account for most of the bill. ## Fixes, ordered by typical impact ### 1. Attribute every call to a feature (do this first) Wrap the client once so every call records feature, model and usage. Without this you cannot prove any later fix worked. ```python import time, json, logging from openai import OpenAI client = OpenAI() log = logging.getLogger("llm_usage") def llm_call(feature: str, **kwargs): t0 = time.monotonic() resp = client.chat.completions.create(**kwargs) u = resp.usage log.info(json.dumps({ "feature": feature, "model": resp.model, "input_tokens": u.prompt_tokens, "output_tokens": u.completion_tokens, "cached_tokens": getattr(u.prompt_tokens_details, "cached_tokens", 0) if u.prompt_tokens_details else 0, "latency_ms": int((time.monotonic() - t0) * 1000), })) return resp ``` ```typescript import OpenAI from "openai"; const client = new OpenAI(); export async function llmCall(feature: string, params: OpenAI.Chat.ChatCompletionCreateParamsNonStreaming) { const t0 = Date.now(); const resp = await client.chat.completions.create(params); const u = resp.usage; console.log(JSON.stringify({ feature, model: resp.model, input_tokens: u?.prompt_tokens, output_tokens: u?.completion_tokens, cached_tokens: u?.prompt_tokens_details?.cached_tokens ?? 0, latency_ms: Date.now() - t0, })); return resp; } ``` Multiply tokens by the provider's current price sheet to get dollars, or let a gateway compute cost for you (see "Do it at the gateway" below). For per-team/per-customer attribution and budgets, read [references/budgets-and-attribution.md](references/budgets-and-attribution.md). ### 2. Right-size the model per task Most production traffic does not need the most capable model. Classify call sites: | Task shape | Typical right size | |---|---| | Classification, routing, extraction, short rewrites, yes/no checks | Small/fast model | | Summaries, standard RAG answers, routine code edits | Mid-tier model | | Multi-step reasoning, hard debugging, design, long synthesis | Frontier model | How to do it safely: 1. Build a small eval set (50-200 real inputs per feature with expected outputs or a rubric). 2. Run the current model and the cheaper candidate on it. Compare quality, not vibes. 3. Switch call sites that pass. Keep the frontier model only where the eval shows it matters. Use whatever models the project already uses or has access to; the model IDs in this skill are examples only. When one endpoint receives both easy and hard requests (a chat box, a coding assistant), a static per-call-site choice is not enough. Route **per request** by complexity. Bifrost's Complexity Router does this at the gateway with no app changes; see section "Do it at the gateway" and [references/complexity-routing.md](references/complexity-routing.md). ### 3. Cap output and loops Output tokens usually cost several times more than input tokens. - Always set an output limit sized to the task (`max_tokens` / `max_completion_tokens` / `max_output_tokens` / `maxTokens`). A classifier needs single digits; a summary needs a few hundred. - Ask for the format you need: "Answer in at most 3 bullets", JSON schema / structured outputs instead of prose plus parsing. - For reasoning models, set the reasoning effort or thinking budget explicitly; the default can dominate the bill. - Cap agent loops on three axes: iterations, total tokens, and wall time. Stop and return a partial result when any cap is hit. ```python MAX_STEPS, MAX_TOKENS_PER_RUN = 12, 200_000 used = 0 for step in range(MAX_STEPS): resp = llm_call("agent", model=MODEL, messages=msgs, tools=tools, max_tokens=1024) used += resp.usage.total_tokens if used > MAX_TOKENS_PER_RUN: break # return partial result + flag for review if not resp.choices[0].message.tool_calls: break msgs += run_tools(resp) # append tool results ``` ```typescript const MAX_STEPS = 12, MAX_TOKENS_PER_RUN = 200_000; let used = 0; for (let step = 0; step < MAX_STEPS; step++) { const resp = await llmCall("agent", { model: MODEL, messages: msgs, tools, max_tokens: 1024 }); used += resp.usage?.total_tokens ?? 0; if (used > MAX_TOKENS_PER_RUN || !resp.choices[0].message.tool_calls) break; msgs.push(...(await runTools(resp))); } ``` - Cap retries (2-3 with exponential backoff and jitter). Never retry 400-class errors; they fail identically and still cost time and sometimes tokens. - Detect repeated identical tool calls in agent loops (same tool + same arguments twice in a row) and stop. ### 4. Trim prompts and context Input tokens are paid on every call, including every turn of a conversation. - **System prompt:** remove duplicated instructions, long examples the model no longer needs, and boilerplate. Measure with a tokenizer (`tiktoken`, the provider's count-tokens endpoint) before and after. - **RAG:** retrieve fewer, better chunks. Lower `top_k`, add a reranker, deduplicate overlapping chunks, and strip markup/boilerplate before indexing. - **Conversation history:** keep the last N turns verbatim and replace older turns with a running summary produced by a small model. - **Tool definitions:** send only the tools relevant to the current step. Twenty tool schemas can cost thousands of tokens per call. - **Tool results:** truncate or summarize large tool outputs (logs, HTML, JSON dumps) before appending them to the conversation. - **Images and files:** downscale images to the resolution the task needs; send extracted text instead of whole PDFs where possible. ### 5. Use provider prompt caching Major providers discount repeated prompt prefixes heavily (cached input is billed at a fraction of the normal input price; check current pricing). Some cache automatically above a minimum prefix length; others need explicit cache breakpoints. Either way the cache only hits when the **prefix is byte-identical**: - Put static content first (system prompt, tool definitions, few-shot examples, long reference docs), variable content last (user message, retrieved chunks, timestamps). - Do not inject the current time, request IDs, or user names into the system prompt. Move them into the final user message. - Keep tool order and JSON serialization deterministic (sorted keys). - Keep a conversation on the same provider, model and key; switching provider or key lands on a cold cache. - Verify with usage fields: OpenAI `usage.prompt_tokens_details.cached_tokens`, Anthropic `usage.cache_read_input_tokens` / `cache_creation_input_tokens`. Provider specifics, code, and response-level caching: [references/caching.md](references/caching.md). ### 6. Cache whole responses When the same question arrives repeatedly, do not call the model at all. - **Exact cache:** hash `(model, normalized messages, params)` and store the response with a TTL. Safe and cheap. - **Semantic cache:** embed the request and serve a stored answer when similarity is above a threshold. Higher hit rate on paraphrases (FAQ, support, search), but tune the threshold and scope it carefully so users never receive another user's personalized answer. - Do not cache responses that depend on private per-user data unless the cache key includes the user, and do not cache tool-call responses blindly. ### 7. Batch offline work Evals, backfills, nightly enrichment, bulk classification and embeddings rarely need a synchronous answer. Provider Batch APIs (OpenAI, Anthropic, Google, Bedrock) process jobs asynchronously at a substantial discount, typically within 24 hours. Group items into batch files, submit, poll, and reconcile by `custom_id`. Bifrost exposes Files and Batch APIs across providers: https://docs.getbifrost.ai/integrations/openai-sdk/files-and-batch.md ### 8. Reuse embeddings - Key stored embeddings by `sha256(model + normalized text)` and skip re-embedding unchanged content. - Re-embed only changed chunks on ingest; do not rebuild the index on every deploy. - Batch embedding requests (many inputs per call) instead of one call per chunk. - Use the smallest embedding dimension that holds retrieval quality (some models support shortened dimensions). ### 9. Streaming Streaming does not change the price per token. It helps cost in one way: when the user navigates away or presses stop, abort the request (`AbortController` in TS, close the stream in Python) so generation stops early. Propagate client disconnects to the upstream call on your server. ## Do it at the gateway (Bifrost) Several of these controls work better in one place than in every service: per-request model routing, response caching, budgets, and cost attribution. Bifrost is an open-source (Apache 2.0) AI gateway that does all of them with OpenAI-compatible and provider-native drop-in endpoints. Repo: https://github.com/maximhq/bifrost, docs: https://docs.getbifrost.ai ### Run Bifrost locally ```bash npx -y @maximhq/bifrost # or: docker run -p 8080:8080 -v $(pwd)/data:/app/data maximhq/bifrost # UI + API on http://localhost:8080 ; add provider keys in the UI (or config.json) ``` For production, pin a version (`maximhq/bifrost:` or `npx -y @maximhq/bifrost --transport-version `), enable auth, and issue virtual keys. Point existing SDKs at it by changing only the base URL: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8080/openai", api_key="sk-bf-your-virtual-key") ``` ```typescript import Anthropic from "@anthropic-ai/sdk"; const client = new Anthropic({ baseURL: "http://localhost:8080/anthropic", apiKey: "sk-bf-your-virtual-key" }); ``` ### Complexity Router: cheap model for simple requests, frontier for complex ones The Complexity Router classifies each request as `SIMPLE`, `MEDIUM` or `COMPLEX` and exposes `complexity_tier` to CEL routing rules. The app keeps sending the model it sends today; a matching rule rewrites the target. Classification runs only when a rule references `complexity_tier`, and if no tier is produced the rule does not match and the request continues with normal routing. 1. Configure a classifier in the UI (**Models > Complexity Router**): **Decision model** (default Typesafe Jev; needs that provider configured with a key) or **Semantic** (embeds the request and matches reference phrases; needs an embedding provider such as `openai` / `text-embedding-3-small`). 2. Add routing rules (UI **Routing Rules**, `POST /api/routing/rules`, or `governance.routing_rules` in `config.json`): ```json { "governance": { "routing_rules": [ { "id": "complexity-simple", "name": "Simple -> small model", "enabled": true, "cel_expression": "complexity_tier == \"SIMPLE\"", "targets": [{ "provider": "openai", "model": "gpt-4o-mini", "weight": 1 }], "scope": "global", "priority": 0 }, { "id": "complexity-complex", "name": "Complex -> frontier model", "enabled": true, "cel_expression": "complexity_tier == \"COMPLEX\"", "targets": [{ "provider": "anthropic", "model": "claude-opus-4-5", "weight": 1 }], "scope": "global", "priority": 1 } ] } } ``` Unmatched (`MEDIUM`, or unclassified) requests keep the model the app asked for. Roll out with one rule or one team scope first, then filter logs by tier and compare quality and cost. Full setup, classifier choice, tuning, session-aware routing and verification: [references/complexity-routing.md](references/complexity-routing.md). ### Semantic and exact response caching Enable the `semantic_cache` plugin with a vector store (Redis, Weaviate, Qdrant, Pinecone; or direct-hash-only mode with no embeddings). Caching engages only for requests that carry `x-bf-cache-key` (or when `default_cache_key` is set). Hits are reported in `x-bifrost-cache-hit` response headers and the `bifrost_cache_hits_total` metric. Config and safety rules: [references/caching.md](references/caching.md). ### Get provider prompt-cache hits - **Session affinity:** send a stable `x-bf-session-id` per conversation or agent run. Bifrost keeps the session on the provider and key that served it, so each turn lands where its prompt cache lives. Claude Code, Codex CLI and OpenCode session headers are picked up automatically. Docs: https://docs.getbifrost.ai/providers/session-affinity.md - **Auto prompt caching:** set `"prompt_cache": {"auto_inject": true}` on a provider (e.g. `anthropic`) and Bifrost adds a cache breakpoint to requests from clients that send none. Off by default; caller markers always win. Docs: https://docs.getbifrost.ai/features/prompt-caching.md ### Budgets, rate limits and cost attribution - **Virtual keys** per team, feature or customer, each with `budgets` (e.g. `{"max_limit": 100.00, "reset_duration": "1M"}`), `rate_limit`, and `allowed_models`. Once a budget is used up, further requests are rejected with `402` until it resets; rate limits return `429`. Teams and customers can carry their own budgets above the key. - **`x-bf-dim-` headers** (e.g. `x-bf-dim-feature: summarize`) tag a request; the tag flows to logs, OpenTelemetry spans, and Prometheus custom labels. - **`bifrost_cost_total`** (USD) on `/metrics`, labeled by `provider`, `model`, `virtual_key_name`, `team_name`, `customer_name`, `routing_rule_name`, `complexity_tier`. - **Custom pricing** overrides catalog prices with your negotiated rates so reported cost matches the invoice: https://docs.getbifrost.ai/providers/custom-pricing.md Setup and PromQL: [references/budgets-and-attribution.md](references/budgets-and-attribution.md). At org scale, Bifrost Enterprise adds budget and rate-limit alerting to Slack, PagerDuty and webhooks (14-day free trial: https://www.getmaxim.ai/bifrost/enterprise). ## Verify 1. **Cost moved:** compare cost per feature (or `bifrost_cost_total` per label) for equal traffic windows before and after. Report cost per 1,000 requests, not totals, so traffic changes do not hide the result. 2. **Quality held:** rerun the eval set for every feature whose model, prompt, or context changed. Spot-check production outputs for routed and cached traffic. 3. **Caches hit:** provider `cached_tokens` > 0 on repeat prefixes; response-cache hit rate is non-zero where expected and zero where it must be (personalized answers). 4. **Caps work:** force an agent into a loop in a test and confirm it stops at the cap; send a request to a key with a $0.01 budget and confirm it is refused. 5. **Latency:** cheaper models are usually faster, but a classifier, embedding lookup or LLM fallback adds latency per request. Check p95. ## Common mistakes - Optimizing before measuring, then trimming a feature that was 2% of spend. - Switching a call site to a smaller model with no eval set, and shipping a quality regression. - Putting timestamps, request IDs or user data at the start of the prompt and wondering why prompt caching never hits. - Load-balancing one conversation across providers or keys, breaking prompt-cache locality. - Semantic cache with a low threshold or a shared key across users, serving wrong or private answers. - No `max_tokens`, no loop cap, and unlimited retries on an agent. - Counting only chat calls; embeddings, classifier calls, LLM-judge evals and retries also cost money. - Using list prices in dashboards when the contract has discounts or committed-use pricing. - Moving interactive traffic to batch APIs; batch is for work that can wait hours. ## References - [references/complexity-routing.md](references/complexity-routing.md): read when routing per request by difficulty (Bifrost Complexity Router setup, rules, rollout, tuning, app-side alternative). - [references/caching.md](references/caching.md): read when enabling provider prompt caching, response caching (exact/semantic), session affinity, or embedding reuse. - [references/budgets-and-attribution.md](references/budgets-and-attribution.md): read when the question is "who is spending what", or when adding budgets, rate limits, per-feature tags, cost dashboards or custom pricing. ```` ## File: README.md size: 2944 sha: 7ffb6b2329e04e41778c6b5886e5595339547dad upstream: https://github.com/build-with-ai-hq/llm-cost-reduction/blob/HEAD/README.md ````text llm-cost-reduction: Find and cut LLM spend without breaking quality. # llm-cost-reduction [![skills.sh](https://skills.sh/b/build-with-ai-hq/llm-cost-reduction)](https://skills.sh/build-with-ai-hq/llm-cost-reduction) An Agent Skill that finds and cuts LLM and AI API spend in applications and agents, measuring first and holding output quality steady, with Python and TypeScript fixes. ## Problems this solves - A surprise provider bill, and no way to tell which feature, team or customer caused it. - Every request, including trivial ones, going to a frontier model. - Repeated and near-duplicate prompts paying full price every time. - Bloated system prompts, tool schemas, conversation history and RAG context. - Unset `max_tokens`, uncapped agent loops and retry storms. - Provider prompt caching available but never hitting. - Offline and bulk jobs running on the synchronous API instead of discounted Batch APIs. - Embeddings recomputed for unchanged content. - No budgets or rate limits per team, feature or customer. ## Install ```bash npx skills add build-with-ai-hq/llm-cost-reduction ``` ## What the skill does When an agent loads this skill, it: 1. Runs a grep audit to find every model call site, the models in use, missing output limits, uncapped loops, retries, unrecorded usage and prompt-cache killers. 2. Instruments calls so cost is known per feature and per request, then ranks features by spend. 3. Applies fixes in order of impact: right-sizing models with an eval set, output and loop caps, prompt and context trimming, provider prompt caching with stable prefixes, exact and semantic response caching, Batch APIs, embedding reuse, and stream cancellation. 4. Moves the controls that work best centrally to an open-source gateway (Bifrost): the Complexity Router to send simple requests to a cheap model and complex ones to a frontier model with no app changes, semantic caching, session affinity and auto prompt caching for provider cache hits, virtual keys with budgets and rate limits, `x-bf-dim-*` cost attribution, `bifrost_cost_total` metrics, and custom pricing. 5. Verifies that cost per 1,000 requests dropped, quality held on the eval set, caches hit, and caps actually stop runaway runs. ## Contents ``` SKILL.md Core workflow: diagnosis, fixes by impact, gateway controls, verification, common mistakes references/complexity-routing.md Per-request model routing: Bifrost Complexity Router setup, routing rules, rollout, tuning; app-side alternative references/caching.md Provider prompt caching per provider, prefix stability, session affinity, auto prompt caching, response and embedding caches references/budgets-and-attribution.md Attribution dimensions, virtual keys, budgets and rate limits, x-bf-dim headers, PromQL, custom pricing, alerting ``` ## License MIT. See [LICENSE](LICENSE). ```` ## File: references/budgets-and-attribution.md size: 10072 sha: 4fa351c5ffcdbdffb931604008e050b8f23a6cd3 upstream: https://github.com/build-with-ai-hq/llm-cost-reduction/blob/HEAD/references/budgets-and-attribution.md ````text # Budgets and cost attribution You cannot cut what you cannot attribute. The goal: every model call is tagged with **who** (team, customer, user) and **what** (feature, endpoint, environment), its cost is computed with the prices you actually pay, and each spender has a hard limit that fails safely. ## 1. Choose attribution dimensions Keep label sets small and bounded (tens to hundreds of values, not unbounded IDs in metrics): | Dimension | Example values | Where it goes | |---|---|---| | `feature` | `support-chat`, `summarize`, `code-review`, `nightly-enrich` | Metrics, logs | | `team` / `owner` | `growth`, `platform` | Metrics, logs, budgets | | `customer` / `tenant` | `acme` (only if bounded or high-value) | Budgets, logs; metrics only if bounded | | `environment` | `prod`, `staging`, `eval` | Metrics, logs | | `user_id`, `request_id`, `conversation_id` | unbounded | Logs and traces only, never metric labels | Count everything that costs money: chat, embeddings, classifiers, LLM-as-judge evals, retries, background jobs. ## 2. App-side attribution (no gateway) Extend the usage wrapper from SKILL.md with tags and a price table: ```python PRICES = { # USD per 1M tokens; fill from your contract or the provider price page "small-model-id": {"in": 0.15, "cached_in": 0.075, "out": 0.60}, "frontier-model-id": {"in": 5.00, "cached_in": 0.50, "out": 25.00}, } def cost_usd(model: str, input_tokens: int, cached: int, output_tokens: int) -> float: p = PRICES[model] return ((input_tokens - cached) * p["in"] + cached * p["cached_in"] + output_tokens * p["out"]) / 1e6 ``` ```typescript const PRICES: Record = { "small-model-id": { in: 0.15, cachedIn: 0.075, out: 0.60 }, "frontier-model-id": { in: 5.0, cachedIn: 0.5, out: 25.0 }, }; export const costUsd = (m: string, inTok: number, cached: number, outTok: number) => ((inTok - cached) * PRICES[m].in + cached * PRICES[m].cachedIn + outTok * PRICES[m].out) / 1e6; ``` (The prices above are placeholders, not real rates.) Emit `cost_usd` with the tags as a structured log line and as a counter (`llm_cost_usd_total{feature,team,model}`). Enforce budgets with a counter in Redis keyed by `team:month` and refuse calls once over the limit. This works, but every service must implement it identically, price tables drift, and limits are not shared across services. A gateway centralizes all three. ## 3. Bifrost: virtual keys, budgets and rate limits Each team, feature, or customer gets a **virtual key** (`sk-bf-...`). Apps send it as the API key; Bifrost holds the real provider keys. Docs: https://docs.getbifrost.ai/features/governance/virtual-keys.md and https://docs.getbifrost.ai/features/governance/budget-and-limits.md Create a team with a monthly budget, then a key for one feature under it: ```bash curl -X POST http://localhost:8080/api/governance/teams \ -H "Content-Type: application/json" \ -d '{ "name": "growth", "budgets": [{ "max_limit": 500.00, "reset_duration": "1M" }] }' curl -X POST http://localhost:8080/api/governance/virtual-keys \ -H "Content-Type: application/json" \ -d '{ "name": "growth-support-chat", "team_id": "", "provider_configs": [ { "provider": "openai", "allowed_models": ["gpt-4o-mini"], "key_ids": ["*"], "weight": 1.0 } ], "budgets": [{ "max_limit": 150.00, "reset_duration": "1M" }], "rate_limit": { "token_max_limit": 2000000, "token_reset_duration": "1h", "request_max_limit": 600, "request_reset_duration": "1m" }, "is_active": true }' ``` How enforcement works: - Budgets exist at provider-config, virtual-key, team and customer level. Every applicable budget must have room; cost is deducted from all of them. - Once a budget is exhausted, requests are rejected with HTTP `402` until it resets. Rate limits return `429`. - Reset durations: `1m` (minute), `1h`, `1d`, `1w`, `1M` (month), `1Q` (budgets only), `1Y`. Set `calendar_aligned: true` on the owner to reset at UTC calendar boundaries (e.g. the 1st of the month) instead of rolling windows. - `allowed_models` per provider config stops a feature key from quietly calling a frontier model. - Turn on `client.enforce_auth_on_inference` so requests without a virtual key are refused. Handle `402` and `429` in the app: degrade (shorter answer, cached answer, smaller model) or show a clear message; do not retry a `402`. ```python from openai import OpenAI, APIStatusError client = OpenAI(base_url="http://localhost:8080/openai", api_key=os.environ["BIFROST_VK"]) try: resp = client.chat.completions.create(model=MODEL, messages=msgs, max_tokens=400) except APIStatusError as e: if e.status_code == 402: return fallback_answer("AI budget for this feature is used up for this period.") raise ``` ```typescript try { return await client.chat.completions.create({ model: MODEL, messages: msgs, max_tokens: 400 }); } catch (e: any) { if (e?.status === 402) return fallbackAnswer("AI budget for this feature is used up for this period."); throw e; } ``` Routing rules can also react to spend: the CEL variable `budget_used` (budget usage percentage, 0-100, for the provider/model) lets you move traffic to a cheaper model before the budget runs out: ```json { "name": "Near budget -> cheaper model", "enabled": true, "cel_expression": "budget_used > 80", "targets": [{ "provider": "openai", "model": "gpt-4o-mini", "weight": 1 }], "scope": "team", "scope_id": "", "priority": 0 } ``` (Create it with `POST /api/routing/rules` on Bifrost v2.0.0+, or the equivalent `/api/governance/*` path on earlier versions; see https://docs.getbifrost.ai/providers/routing-rules.md.) ## 4. Bifrost: per-request dimensions When one virtual key serves several features, tag each request with `x-bf-dim-` headers. Dimensions are forwarded to request logs, OpenTelemetry span attributes, Prometheus custom labels (for configured label names), and Datadog. ```python resp = client.chat.completions.create( model=MODEL, messages=msgs, max_tokens=400, extra_headers={"x-bf-dim-feature": "summarize", "x-bf-dim-environment": "prod"}, ) ``` ```typescript const resp = await client.chat.completions.create( { model: MODEL, messages: msgs, max_tokens: 400 }, { headers: { "x-bf-dim-feature": "summarize", "x-bf-dim-environment": "prod" } }, ); ``` Rules: do not put secrets or PII in dimensions (they reach every observability backend); dimensions cannot override reserved keys such as the virtual key or request ID; keep values bounded if they become metric labels. For Prometheus, the dimension name must match a label configured on the telemetry plugin (`client.prometheus_labels`); see https://docs.getbifrost.ai/providers/request-options.md and https://docs.getbifrost.ai/features/observability/prometheus.md. ## 5. Bifrost: cost metrics and queries Telemetry is on by default at `GET /metrics`. `bifrost_cost_total` (USD) and token counters carry labels including `provider`, `model`, `virtual_key_name`, `team_name`, `customer_name`, `routing_rule_name`, `complexity_tier` and `selected_key_name`. ```promql # Daily spend by team and model sum by (team_name, model) (increase(bifrost_cost_total[1d])) # Top 10 virtual keys (features) by spend this week topk(10, sum by (virtual_key_name) (increase(bifrost_cost_total[7d]))) # Output vs input tokens by model (output is usually the expensive side) sum by (model) (increase(bifrost_output_tokens_total[1d])) / sum by (model) (increase(bifrost_input_tokens_total[1d])) # Spend per complexity tier after enabling complexity routing sum by (complexity_tier) (increase(bifrost_cost_total[1d])) # Cache effectiveness sum(increase(bifrost_cache_hits_total[1d])) ``` Request-level detail (model, tokens, cost, latency, virtual key, routing decision, dimensions) is in the built-in log store: UI at `http://localhost:8080/logs`, API `GET /api/logs`. Export traces with the `otel` plugin if you already run an OpenTelemetry stack. ## 6. Bifrost: custom pricing Bifrost prices requests from a built-in catalog synced every 24 hours. If you have negotiated rates, committed-use discounts, or self-hosted models with no list price, add overrides so reported cost matches the invoice. Overrides can be global or scoped to a provider, provider key, virtual key or user; the most specific match wins. Docs: https://docs.getbifrost.ai/providers/custom-pricing.md ```bash curl -X POST http://localhost:8080/api/governance/pricing-overrides \ -H "Content-Type: application/json" \ -d '{ "name": "Negotiated gpt-4o rate", "scope_kind": "global", "match_type": "exact", "pattern": "gpt-4o", "request_types": ["chat_completion"], "patch": { "input_cost_per_token": 0.0000025, "output_cost_per_token": 0.000010 } }' ``` Fields also exist for batch, priority, flex, cached and long-context tiers (e.g. `input_cost_per_token_batches`). In `config.json`, overrides live under `governance.pricing_overrides` with `pricing_patch` as a JSON-encoded string. Providers that return their own `usage.cost` are priced from that figure unless `ignore_provider_cost` is enabled on the provider. Budgets are enforced on the computed cost, so custom pricing also makes budgets accurate. ## 7. Alerting OSS: alert from Prometheus (Alertmanager) on `increase(bifrost_cost_total[1h])` by team or virtual key, and on rising `bifrost_error_requests_total{status_code=~"402|429"}`. ## Checklist - [ ] Every call carries a feature and owner (virtual key per team/feature, plus `x-bf-dim-feature` where keys are shared). - [ ] Embeddings, evals, classifiers and retries are included in cost. - [ ] Prices match the contract (custom pricing overrides). - [ ] Every spender has a budget and a rate limit; `402`/`429` are handled gracefully. - [ ] A dashboard shows spend per feature per day and cost per 1,000 requests. - [ ] An alert fires on abnormal hourly spend before the monthly budget is gone. ```` ## File: references/caching.md size: 13933 sha: 210572b60543e667eb3596e5ae94397dd22b49a3 upstream: https://github.com/build-with-ai-hq/llm-cost-reduction/blob/HEAD/references/caching.md ````text # Caching for LLM cost reduction Three different caches, often confused: | Cache | What is reused | Saves | Where | |---|---|---|---| | Provider prompt cache | The provider's processed prompt prefix | Input-token price on the cached prefix (large discount) and latency; output still billed | Provider side; you control prefix stability | | Response cache (exact or semantic) | The whole previous answer | The entire call | Your app or the gateway | | Embedding cache | Vectors for text you already embedded | Embedding calls | Your app / vector DB | ## 1. Provider prompt caching ### How the providers differ (check current docs and pricing before relying on numbers) - **OpenAI:** automatic for prompts above a minimum length (around 1,024 tokens). Cached input tokens are billed at a discount. Reported in `usage.prompt_tokens_details.cached_tokens`. A `prompt_cache_key` request parameter can improve hit rates when many requests share a prefix. - **Anthropic:** two modes. Automatic caching: add one top-level `cache_control: {"type": "ephemeral"}` field to the request and the breakpoint is placed on the last cacheable block, moving forward as the conversation grows (simplest for multi-turn chat). Explicit breakpoints: put `cache_control: {"type": "ephemeral"}` on specific system blocks, tools, or messages (a small number of breakpoints per request; minimum cacheable length applies). Cache writes cost more than normal input, cache reads cost a small fraction; the default TTL is short (minutes) with a longer-TTL option. Reported in `usage.cache_creation_input_tokens` and `usage.cache_read_input_tokens`. - **Google Gemini:** implicit caching on recent models, plus explicit context caching (a cached-content resource with a TTL and a storage charge) for large shared context. - **AWS Bedrock:** prompt caching with cache checkpoints on supported models. A cache write that is never read costs more than no cache at all on providers that charge a write premium. Cache only prefixes that repeat within the TTL. ### Make prefixes cacheable Order every request as: tools -> system prompt -> static reference material / few-shot examples -> conversation history -> latest user message and per-request data. Checklist: - No timestamps, request IDs, user names, A/B flags or random example selection in the system prompt. Move per-request values into the last user message. - Deterministic tool list order and JSON serialization (`json.dumps(..., sort_keys=True)`, stable key order in TS objects). - Same model, same provider, same key (or org) for the whole conversation. Load balancing a session across keys or providers sends each turn to a cold cache. - Do not rewrite earlier history (e.g. re-summarizing every turn) unless you accept a cache miss on everything after the change. Summarize in larger, infrequent steps. - RAG chunks change per query; put them after the stable prefix, not inside the system prompt. ### Anthropic example ```python import anthropic client = anthropic.Anthropic() resp = client.messages.create( model=MODEL, max_tokens=800, system=[ {"type": "text", "text": LONG_STATIC_INSTRUCTIONS_AND_DOCS, "cache_control": {"type": "ephemeral"}}, ], messages=history + [{"role": "user", "content": f"{user_msg}\n\n(current time: {now_iso})"}], ) u = resp.usage print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens) ``` ```typescript import Anthropic from "@anthropic-ai/sdk"; const client = new Anthropic(); const resp = await client.messages.create({ model: MODEL, max_tokens: 800, system: [{ type: "text", text: LONG_STATIC_INSTRUCTIONS_AND_DOCS, cache_control: { type: "ephemeral" } }], messages: [...history, { role: "user", content: `${userMsg}\n\n(current time: ${nowIso})` }], }); console.log(resp.usage.cache_creation_input_tokens, resp.usage.cache_read_input_tokens); ``` ### OpenAI example Nothing to enable; make the prefix stable and check the counter. ```python resp = client.chat.completions.create( model=MODEL, max_tokens=500, messages=[{"role": "system", "content": STATIC_SYSTEM_PROMPT}, *history, {"role": "user", "content": user_msg}], ) print(resp.usage.prompt_tokens_details.cached_tokens, "of", resp.usage.prompt_tokens) ``` ### Diagnosing a zero hit rate 1. Diff the first ~2,000 tokens of two consecutive requests. Any difference before the end of the intended prefix is the cause. 2. Check the prefix is above the provider's minimum cacheable length. 3. Check requests in the same conversation hit the same provider, model and key. 4. Check the gap between requests is shorter than the cache TTL. ### Keep sessions on one provider and key with Bifrost When a gateway load-balances across keys or providers, it can scatter a conversation across caches. Bifrost session affinity binds a session to the provider and key that last served it, so later turns land on the warm cache. Send a stable session ID per conversation or agent run: ```python client = OpenAI(base_url="http://localhost:8080/openai", api_key="sk-bf-your-virtual-key") resp = client.chat.completions.create( model=MODEL, messages=msgs, max_tokens=500, extra_headers={"x-bf-session-id": conversation_id}, ) ``` ```typescript const resp = await client.chat.completions.create( { model: MODEL, messages: msgs, max_tokens: 500 }, { headers: { "x-bf-session-id": conversationId } }, ); ``` - Default binding TTL is one hour, refreshed on each served request; override per request with `x-bf-session-ttl` (e.g. `30m`). - Claude Code, Codex CLI and OpenCode session headers are adopted automatically. - Affinity applies when routing offers more than one provider or key; a request that names its provider (`openai/gpt-4o`) is never reordered across providers but still gets key-level affinity. - Disable for a request with `x-bf-session-affinity: off`. Docs: https://docs.getbifrost.ai/providers/session-affinity.md . If you also use the Complexity Router, enable its session-aware routing so the tier (and therefore the model) does not change mid-conversation. Caller-supplied `cache_control` markers are forwarded unchanged. Confirm hits with the usage counters above on responses that pass through the gateway. ### Auto-inject cache breakpoints with Bifrost Clients that send no cache markers (many agent harnesses, Codex, generic OpenAI-compatible code calling Claude) never get Anthropic-style prompt caching, so every turn pays full input price for a prefix that barely changed. Bifrost's Auto Prompt Caching marks the first cacheable block of requests that arrive without markers, per provider: ```json { "providers": { "anthropic": { "keys": [{ "name": "anthropic-key-1", "value": "env.ANTHROPIC_API_KEY", "models": ["*"], "weight": 1.0 }], "prompt_cache": { "auto_inject": true } } } } ``` - Off by default. Caller markers always win; a request that already has markers is forwarded unchanged. - Injected only for models that accept explicit markers (Claude on Anthropic, Vertex, Bedrock and OpenRouter; Amazon Nova; newer OpenAI models that take `prompt_cache_breakpoint`). Implicit-caching providers are left alone. - Optional `"ttl": "1h"` requests the longer cache lifetime (the only accepted value; omit for the provider default). Longer TTL carries a higher write price on Anthropic, so use it only when turns are minutes apart. - `cache_control_injection_points` targets specific messages instead of the first block. - Per-request override: `x-bf-prompt-cache-auto-inject: true|false`. - Also settable in the UI (Model Providers > provider > Prompt Caching tab) or via `PUT /api/providers/{provider}` (that endpoint replaces the provider-level config, so send existing network settings along). Docs: https://docs.getbifrost.ai/features/prompt-caching.md . Pair it with session affinity so the cache that was written is the one that gets read. ## 2. Response caching ### When it is safe - Answers that depend only on the prompt: FAQ, documentation Q&A, product lookups, classification of repeated inputs, deterministic transforms, eval reruns. - Not safe without a per-user key: anything using private user data, account state, or time-sensitive facts. - Not safe by default: responses containing tool calls (replaying one can run a tool with someone else's arguments). ### App-side exact cache ```python import hashlib, json, redis r = redis.Redis() def cached_call(feature: str, ttl: int = 3600, **kwargs): key = "llm:" + hashlib.sha256(json.dumps(kwargs, sort_keys=True).encode()).hexdigest() if (hit := r.get(key)) is not None: return json.loads(hit) resp = llm_call(feature, **kwargs) # instrumented wrapper from SKILL.md if not resp.choices[0].message.tool_calls: r.setex(key, ttl, resp.model_dump_json()) return json.loads(resp.model_dump_json()) ``` ```typescript import { createHash } from "node:crypto"; import { createClient } from "redis"; const redis = await createClient().connect(); export async function cachedCall(feature: string, params: any, ttlSec = 3600) { const key = "llm:" + createHash("sha256").update(JSON.stringify(params)).digest("hex"); const hit = await redis.get(key); if (hit) return JSON.parse(hit); const resp = await llmCall(feature, params); if (!resp.choices[0].message.tool_calls) await redis.set(key, JSON.stringify(resp), { EX: ttlSec }); return resp; } ``` Use `temperature: 0` for cacheable calls so a cached answer is representative. Note that `JSON.stringify` key order follows insertion order; build params consistently. ### Gateway cache with Bifrost (exact + semantic) Bifrost's `semantic_cache` plugin does a direct hash lookup first, then (optionally) a semantic vector search on a miss. Docs: https://docs.getbifrost.ai/features/semantic-caching.md Start a vector store locally: ```bash docker run -d --name redis-stack -p 6379:6379 redis/redis-stack:latest ``` `config.json`: ```json { "vector_store": { "enabled": true, "type": "redis", "config": { "addr": "localhost:6379" } }, "plugins": [ { "enabled": true, "name": "semantic_cache", "config": { "provider": "openai", "embedding_model": "text-embedding-3-small", "dimension": 1536, "ttl": "5m", "threshold": 0.8, "conversation_history_threshold": 3, "cache_by_model": true, "cache_by_provider": true } } ] } ``` Weaviate, Qdrant and Pinecone are also supported. For exact-match only (no embedding calls), set `"dimension": 1` and omit `provider` and `embedding_model`. Caching engages only on requests that carry a cache key: ```python resp = client.chat.completions.create( model=MODEL, messages=msgs, max_tokens=300, temperature=0, extra_headers={"x-bf-cache-key": "support-faq", "x-bf-cache-ttl": "1h"}, ) ``` ```typescript const resp = await client.chat.completions.create( { model: MODEL, messages: msgs, max_tokens: 300, temperature: 0 }, { headers: { "x-bf-cache-key": "support-faq", "x-bf-cache-ttl": "1h" } }, ); ``` Per-request controls: `x-bf-cache-ttl`, `x-bf-cache-threshold` (can only make matching stricter), `x-bf-cache-type: direct|semantic`, `x-bf-cache-no-store: true` (read but do not write). A `default_cache_key` in the plugin config turns caching on for requests without the header; leave it empty unless every request through that gateway is safe to cache. Safety defaults worth knowing: - Entries are scoped to the requesting virtual key (and user, where resolved) on top of the cache key, so two virtual keys never share entries unless `disable_identity_scoping` is set. - Tool-call responses are not cached unless `cache_tool_call_responses: true`. - Conversations longer than `conversation_history_threshold` messages are not cached (long chats rarely repeat exactly). Verify: responses carry `x-bifrost-cache-hit`, `x-bifrost-cache-hit-type` (`direct` / `semantic`) and, on semantic hits, `x-bifrost-cache-similarity`. Prometheus exposes `bifrost_cache_hits_total`. Clear entries for a key with `DELETE /api/cache/clear-by-key/{cacheKey}`. Threshold tuning: start at 0.8-0.9 for FAQ-style traffic; sample semantic hits with their similarity and the original vs. new question. Raise the threshold the moment you see a hit that answers a different question. ## 3. Embedding reuse ```python import hashlib def embed_cached(texts: list[str], model: str) -> list[list[float]]: keys = [hashlib.sha256(f"{model}\n{t.strip()}".encode()).hexdigest() for t in texts] found = store.get_many(keys) # your KV / vector DB missing = [i for i, k in enumerate(keys) if k not in found] if missing: resp = client.embeddings.create(model=model, input=[texts[i] for i in missing]) # batched for i, d in zip(missing, resp.data): found[keys[i]] = d.embedding store.put_many({keys[i]: found[keys[i]] for i in missing}) return [found[k] for k in keys] ``` ```typescript import { createHash } from "node:crypto"; export async function embedCached(texts: string[], model: string): Promise { const keys = texts.map(t => createHash("sha256").update(`${model}\n${t.trim()}`).digest("hex")); const found = await store.getMany(keys); // Map const missing = keys.map((k, i) => (found.has(k) ? -1 : i)).filter(i => i >= 0); if (missing.length) { const resp = await client.embeddings.create({ model, input: missing.map(i => texts[i]) }); missing.forEach((i, j) => found.set(keys[i], resp.data[j].embedding)); await store.putMany(missing.map(i => [keys[i], found.get(keys[i])!])); } return keys.map(k => found.get(k)!); } ``` Also: chunk deterministically (same chunker settings produce the same chunks and therefore the same hashes), re-embed only changed documents, and move bulk re-embedding to a Batch API. ```` ## File: references/complexity-routing.md size: 13150 sha: efde956559bb3fccfe0f272a57f3e20418d0c92c upstream: https://github.com/build-with-ai-hq/llm-cost-reduction/blob/HEAD/references/complexity-routing.md ````text # Complexity-based model routing Goal: send each request to the cheapest model that can handle it. Simple requests (lookups, reformatting, short factual answers) go to a small model; complex ones (multi-step reasoning, hard debugging, design) go to a frontier model. This matters most for endpoints that receive mixed traffic, such as chat boxes, copilots and coding assistants, where one static model choice is always either too expensive or too weak. ## Before routing: know your traffic 1. Sample 200-500 real requests from the endpoint (strip PII). 2. Label each one SIMPLE / MEDIUM / COMPLEX by hand, or have a strong model label them and spot-check. 3. Run the candidate cheap model on the SIMPLE bucket and the mid-tier model on the MEDIUM bucket. Score against the frontier model's answers or a rubric. 4. Estimate savings: `share_simple x (price_frontier - price_small) + share_medium x (price_frontier - price_mid)`, minus classifier cost. If 70% of traffic is SIMPLE and the small model holds quality on it, routing is worth doing. If almost everything is COMPLEX, spend the effort on prompt trimming and caching instead. ## Option A: Bifrost Complexity Router (recommended, no app changes) Bifrost classifies each request into `SIMPLE`, `MEDIUM` or `COMPLEX` and publishes the tier as `complexity_tier` to its CEL routing rules. The application keeps calling the gateway with the model it uses today; a matching rule overrides provider and model. Docs: https://docs.getbifrost.ai/features/complexity-router.md and https://docs.getbifrost.ai/providers/routing-rules.md Key behaviors: - Classification runs only when some routing rule references `complexity_tier`. - If the classifier is unavailable, times out, or produces no tier, the complexity rule does not match and the request continues through normal routing (the app's requested model). Failure is safe. - The Router only sets the variable; you write one rule per tier you want to route. - Classifier cost counts toward the request's cost and budgets under the model that answered. ### Step 1: Pick a classifier | | Decision model | Semantic | |---|---|---| | How it decides | A small decision model judges the work against tier definitions, signals and examples | Embeds the latest user message and takes the tier of the nearest reference phrase | | You maintain | Nothing to start; optionally tune definitions/signals/examples | Reference phrases per tier (150 shipped, up to 750) | | Requires | A provider serving the decision model (Typesafe Jev by default; also OpenAI, OpenRouter, Cloudflare, Perplexity, Ollama, Azure, Databricks, or self-hosted) | An embedding provider/model with an enabled key; optionally a shared vector store | | Added latency | One decision call per classified request (Jev typically 600-800 ms; default timeout 1.5 s) | One embedding call plus a vector lookup, usually much faster | | Best when | Traffic is varied and you do not want to curate data | Traffic is predictable and latency matters most | You can combine them: Semantic as primary with the decision model as fallback (`semantic.fallback: "decision"`) for requests no phrase matches confidently. Configure in the UI under **Models > Complexity Router**, via `PUT /api/routing/complexity-analyzer-config`, or in `config.json` under `governance.complexity_analyzer_config`. The `/api/routing/*` endpoints exist in Bifrost v2.0.0 and above; earlier versions use `/api/governance/*`. Decision-model primary (`config.json`): ```json { "governance": { "complexity_analyzer_config": { "classifier": "decision", "decision": { "provider": "typesafe", "model": "jev-latest", "previous_message_count": 1, "timeout": "1.5s" }, "session": { "enabled": true }, "keywords": { "simple_keywords": ["fix the grammar in this sentence."], "medium_keywords": ["apply a standard method to this technical question."], "complex_keywords": ["design a solution under several competing constraints."] } } } } ``` Semantic primary with OpenAI embeddings: ```json { "governance": { "complexity_analyzer_config": { "classifier": "semantic", "semantic": { "provider": "openai", "embedding_model": "text-embedding-3-small", "timeout": "1.5s", "min_similarity": 0, "message_history_count": 1, "count_toward_budgets": false, "vector_store": "embedded", "fallback": "none" }, "session": { "enabled": true }, "keywords": { "simple_keywords": ["what is a mutex?", "fix the grammar in this sentence."], "medium_keywords": ["add api-key auth: hash the keys, reject revoked ones, and never log them."], "complex_keywords": ["balance testing, prescribing rules, and staffing against rising resistant infections."] } } } } ``` Notes that trip people up: - The `keywords` lists (reference phrases) are required in a saved config even when the decision model is primary. They are unused by the decision model. - The update API replaces the whole configuration. Leaving out `decision.criteria` resets tier guidance to the shipped defaults. To change one field, `GET` the config, edit, `PUT` it back. - `GET /api/routing/complexity-analyzer-status` reports semantic warmup state (`warming`, `ready`, `failed`) and returns the shipped decision-model guidance as `decision_defaults`. - With several Bifrost nodes and the Semantic classifier, use a shared vector store (`"vector_store": "vector_store"`) so each node does not re-embed phrases on restart. ### Step 2: Add routing rules Start with a single carve-out, validate it, then add tiers. Rules live in the UI (**Routing Rules**, with a visual CEL builder), the API (`POST /api/routing/rules`), or `governance.routing_rules` in `config.json`. Rules are first-match-wins within a scope, ordered by `priority` (lower first; each priority unique per scope). Target `weight`s in one rule must sum to 1. Cheapest first move, sending SIMPLE traffic to a small model while everything else stays as is: ```bash curl -X POST http://localhost:8080/api/routing/rules \ -H "Content-Type: application/json" \ -d '{ "name": "Simple -> small model", "enabled": true, "cel_expression": "complexity_tier == \"SIMPLE\"", "targets": [{ "provider": "openai", "model": "gpt-4o-mini", "weight": 1 }], "fallbacks": ["anthropic/claude-haiku-4-5"], "scope": "global", "priority": 0 }' ``` Full ladder (`config.json`); substitute the models your project already uses: ```json { "governance": { "routing_rules": [ { "id": "complexity-simple", "name": "Simple -> fast model", "enabled": true, "cel_expression": "complexity_tier == \"SIMPLE\"", "targets": [{ "provider": "groq", "model": "llama-3.1-8b-instant", "weight": 1 }], "scope": "global", "priority": 0 }, { "id": "complexity-medium", "name": "Medium -> balanced model", "enabled": true, "cel_expression": "complexity_tier == \"MEDIUM\"", "targets": [{ "provider": "openai", "model": "gpt-4o-mini", "weight": 1 }], "scope": "global", "priority": 1 }, { "id": "complexity-complex", "name": "Complex -> frontier model", "enabled": true, "cel_expression": "complexity_tier == \"COMPLEX\"", "targets": [{ "provider": "anthropic", "model": "claude-opus-4-5", "weight": 1 }], "scope": "global", "priority": 2 } ] } } ``` Rules in `config.json` load at startup; changes need a restart. API and UI changes apply live. Combine the tier with other context to limit blast radius: ```cel complexity_tier == "SIMPLE" && headers["x-feature"] == "support-chat" request_type == "chat_completion" && complexity_tier != "COMPLEX" team_name == "ml-research" && complexity_tier == "COMPLEX" ``` Pilot on one team by setting `"scope": "team", "scope_id": ""` before going global. Scope order is virtual key, then team, then customer, then global. If requests use a virtual key, the rule's target provider/model must be allowed by that key's `provider_configs`, otherwise governance refuses the attempt with `403`. ### Step 3: Session-aware routing for agents and chats Model switches mid-conversation hurt provider prompt-cache reuse and can change answer style. With `"session": {"enabled": true}`, a session keeps the highest tier it has reached (it can go up, never down) for 24 hours of inactivity. Session ID comes from `x-bf-session-id`, or automatically from Claude Code (`x-claude-code-session-id`) and Codex headers. Session-aware routing keeps the **tier** stable, not the provider or key. Add session affinity (same `x-bf-session-id`) to also keep the provider and key, so the conversation keeps hitting a warm prompt cache: https://docs.getbifrost.ai/providers/session-affinity.md ### Step 4: Verify and tune - Logs (UI `/logs`, or API) record `complexity_tier`, `complexity_mechanism` (`semantic`, `decision`, `llm`, `session`, `skipped`) and, for Semantic, `complexity_score`: ```bash curl "http://localhost:8080/api/logs?complexity_tiers=SIMPLE&complexity_mechanisms=decision" ``` - Prometheus metrics carry `complexity_tier`, `complexity_mechanism` and `routing_rule_name` labels: ```promql sum by (complexity_tier, model) (increase(bifrost_cost_total[1d])) sum by (complexity_tier) (rate(bifrost_success_requests_total[1h])) sum by (complexity_mechanism) (rate(bifrost_success_requests_total{complexity_mechanism="skipped"}[1h])) ``` A high `skipped` share means the classifier is timing out or not ready; those requests fall through to the default model and save nothing. - Review a sample of SIMPLE-routed requests weekly. If hard requests land in SIMPLE, tune: - Decision model: edit tier `definition`, `signals`, `examples` under `decision.criteria.`. A list replaces the default list, so include the defaults you want to keep. Describe the work, not the topic ("change must stay consistent across several files", not "mentions Kubernetes"). Keep guidance short; it is sent with every classification. - Semantic: add short reference phrases taken from real prompts to the right tier; balance questions and imperatives across tiers; raise `min_similarity` and add a fallback if accidental matches appear (the routing decision log shows the matched phrase). - Account for classifier cost: decision calls are billed to the request; Semantic embedding spend is in `bifrost_routing_embedding_cost_total`. ## Option B: app-side router (no gateway) If you cannot add a gateway, route in code. Keep the router cheap and fail toward quality. ```python ROUTES = {"SIMPLE": "small-model-id", "MEDIUM": "mid-model-id", "COMPLEX": "frontier-model-id"} CLASSIFY_PROMPT = ( "Classify the user's request by the work it needs. " "SIMPLE: answerable in one step from the text or common knowledge. " "MEDIUM: needs subject knowledge or a few steps. " "COMPLEX: multi-stage reasoning, design, or hard debugging. " "Reply with one word." ) def classify(text: str) -> str: try: r = client.chat.completions.create( model=ROUTES["SIMPLE"], max_tokens=3, temperature=0, timeout=1.5, messages=[{"role": "system", "content": CLASSIFY_PROMPT}, {"role": "user", "content": text[-4000:]}]) tier = r.choices[0].message.content.strip().upper() return tier if tier in ROUTES else "COMPLEX" except Exception: return "COMPLEX" # fail toward quality ``` ```typescript const ROUTES = { SIMPLE: "small-model-id", MEDIUM: "mid-model-id", COMPLEX: "frontier-model-id" } as const; type Tier = keyof typeof ROUTES; async function classify(text: string): Promise { try { const r = await client.chat.completions.create( { model: ROUTES.SIMPLE, max_tokens: 3, temperature: 0, messages: [{ role: "system", content: CLASSIFY_PROMPT }, { role: "user", content: text.slice(-4000) }] }, { timeout: 1500 }); const tier = r.choices[0].message.content?.trim().toUpperCase() as Tier; return tier in ROUTES ? tier : "COMPLEX"; } catch { return "COMPLEX"; } } ``` Downsides compared with gateway routing: every service must embed the router, rules change only with a deploy, and there is no shared view of tier distribution and cost per tier unless you build it. ## Pitfalls - Routing by prompt length or keywords. Length does not predict difficulty; a short question can be hard. - No eval before switching SIMPLE traffic to a small model. - Letting a conversation bounce between models every turn (use session-aware routing plus session affinity). - Ignoring classifier latency on latency-sensitive endpoints; prefer Semantic or scope routing to endpoints where a few hundred ms is acceptable. - Forgetting that a request the classifier skips uses the app's requested model; if that is the frontier model, a broken classifier quietly removes all savings. Watch the `skipped` share. ```` ## File: assets/banner.png size: 59282 sha: 51edb41a377b385720b225fb686fbb3aeea82ffa upstream: https://github.com/build-with-ai-hq/llm-cost-reduction/blob/HEAD/assets/banner.png Content is not inlined in the catalog. Read the upstream file: https://github.com/build-with-ai-hq/llm-cost-reduction/blob/HEAD/assets/banner.png ## File: assets/banner.svg size: 3803 sha: 8cb304518e70949d1e248a8cdd8d80859cdab55e upstream: https://github.com/build-with-ai-hq/llm-cost-reduction/blob/HEAD/assets/banner.svg ```text build-with-ai-hq llm-cost-reduction Find and cut LLM spend without breaking quality. $ npx skills add build-with-ai-hq/llm-cost-reduction ``` ## File: assets/icon.svg size: 413 sha: 4223744642fcf974772136d4f2f387f514bf1710 upstream: https://github.com/build-with-ai-hq/llm-cost-reduction/blob/HEAD/assets/icon.svg ```text ``` ## File: LICENSE size: 1073 sha: 11268562f64958b1645c516e99518afde9d720fe upstream: https://github.com/build-with-ai-hq/llm-cost-reduction/blob/HEAD/LICENSE ```text MIT License Copyright (c) 2026 build-with-ai-hq Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ```