An AI gateway sits in front of OpenAI, Anthropic or a local model so the app has one place for a spend cap, a log of what was sent, a rate limit and a second provider when the first fails. Support tools, shop assistants, internal search and every other feature that needs a model all call that layer. The layer holds the keys.
Contents
Otherwise the key lands in a browser bundle or gets copied into every microservice, and the invoice arrives with no feature name on it. A timeout takes the screen down while another model could have answered. Customer text sits in a log nobody planned to keep. An AI gateway is the small service that turns those four problems into one.
CodeWiz builds it for teams that already call a model and want each call capped, redacted, limited and able to fail over. We do not promise what a token will cost. Prices change, and a long prompt costs more than a short one.
Why an AI gateway holds the provider keys
A browser must not see an OpenAI or Anthropic key. The page runs on the visitor’s machine, so a key in JavaScript is a key you have published. A login decides who may open the page. The page source is still readable.
Each microservice can keep its own business logic. None of them should keep a private copy of the provider key. One leaked repository or one shared staging file exposes every product that used that copy.
The application calls your endpoint. That endpoint is the AI gateway. It reads the key from a secret store, checks which feature is calling, and only then forwards the prompt. Staging and production use different keys. The repository stores the name of the secret. Callers use a server-side token, mutual TLS, or a private network.
A request walks a short path. The app sends the feature name, the messages and a timeout budget. The AI gateway checks the cap and the rate, redacts what must not be stored, calls the provider and returns the answer. On error or timeout, the next name on the list is tried. One feature’s policy can be this small:
{ "feature": "support-summary", "primary": "openai", "fallback": "anthropic", "timeout_ms": 8000, "daily_call_cap": 2000 }Cloudflare documents how a zone can record model calls when the site already runs there. We use that when the log can live with the zone. When the keys must stay on your server, we add a small service to the API you already have. OpenAI’s notes on production use cover timeouts, bounded retries and secrets that stay off the client.
| Direct provider call | Through an AI gateway | |
|---|---|---|
| Spend | Each service spends until the provider bill arrives | A cap per feature, with an alert before the hard stop |
| Logs | The provider dashboard, or no log you control | Your log, with secrets and customer data redacted |
| Failover | The feature fails when that one provider fails | A second provider, or a local model, can answer |
| Keys | Copied into every service, sometimes the browser | Held once, on the server, rotated in one place |
Spend caps and alerts on an AI gateway
A model call is metered. A loop, a retry storm or a popular new feature can spend a month of budget in an afternoon. A spend cap is a number you choose: stop, or switch to a smaller model, when the day’s calls or the tracked usage reach it.
The cap is a ceiling you operate, separate from a price list that can change. What you can bound is the number of calls, plus the usage figures the provider returns, so someone hears about the trend while a working day is left.
Write down what “stop” means. Some features refuse the call and show a plain message. Others drop to a smaller model, or to the local one, and return a shorter answer. An alert near seventy percent of the daily number leaves time to raise the cap or pause the feature.
The alert belongs with the rest of your monitoring. The layer emits a counter and a threshold for someone who can act. We treat it like any other production signal, as in our monitoring and logging work: few alerts, a named owner, and retention that will not fill the disk.
What to log and what to redact
A useful log answers a short list. Which feature called, which model answered, whether it succeeded, and how many milliseconds it took. Keep enough context to debug a bad answer the next morning. Card numbers, passwords and API keys stay out of the file.
Redaction runs before anything is written. Authorization headers and provider keys are dropped. Email addresses, phone numbers and card-like digit strings are replaced. Keep the prompt as a hash plus a short prefix, for a few days, then delete it.
The AI gateway records the decision: time, feature, model, provider status, latency, cache hit or miss, and an id that matches the application log. Token counts go on that line when the provider returns them. Retention is a number of days, then delete. One dispute can keep one case.
Timeouts and failover
A slow provider is often worse than a down one. The user watches a spinner while your worker holds the connection. Match the timeout to the feature: two seconds for a short classification, longer for a summary, with the screen saying it is still working. When the timer fires, stop.
Failover is an ordered list: primary, then a second hosted model, then a local model if the task allows it. The AI gateway opens the next call only after the first has failed or timed out. The user gets one answer or one clear error.
Move on after a timeout, a rate-limit response or a server error. An invalid request should stop, because the same body will fail at the next provider. Record which branch answered. One retry on a network blip is enough. Three retries with no timeout stretch the outage and the bill.
Side effects stay outside the model call. A second attempt is safe when the model only writes a summary. When the result will send an email or take a payment, finish the model call, then do the action once. A limit per feature and per user, checked first, stops one client from using the quota everyone else needs.
Caching identical requests
The same prompt, model and settings should not be paid for twice in one minute. A cache key from the feature, the model, the system instructions and a hash of the messages can return the previous answer while it is fresh. Fixed translations, closed-set labels and FAQ answers benefit. A live chat, or a prompt with the current time, skips the cache.
The AI gateway returns a stored answer only when the request is marked cacheable and the text is safe to reuse. A long-lived cache of customer wording is another copy of data you meant to redact. Keep the window short, and leave one person’s private text out of the shared cache.
Limits per feature
One shared key with no ceiling lets a test script and the production assistant spend from the same pocket. Split the policy: support summary, product description, internal search. Each has its own cap, rate and model, so a bug in descriptions cannot empty the budget support needs in the morning.
The feature name travels with the request. The AI gateway rejects a name it does not know, which stops a copied sample from calling a model with no limit. Adding a feature is a policy change, reviewed like any other config. Staging gets its own small ceiling, so a load test cannot spend the production cap.
A local model for sensitive text
Some prompts should stay on your network: a contract clause, a staff note, a record you would not paste into a public chat. A local model beside the app is the fallback, and for some features the only route. It will not match a large hosted model on every task. It keeps the bytes on a machine you control.
The app sends the feature name and the text. The AI gateway applies the rule: this feature may call OpenAI, this one may fall back to Anthropic, and this one stays local even when the hosted APIs are healthy. Write down which features, fields and documents are on that list, and remove the direct call once the feature has moved.
CodeWiz fits local models into ordinary products, with the same request shape and a line between text that may leave and text that must stay. How that meets the rest of the product is in our AI and machine learning integration article. The rule lives here, and the application code stays plain.
Mistakes that empty the account
An AI gateway still fails when the policy is careless. The two we refuse first are a log that keeps every customer prompt forever, and one shared key with no cap. The rest appear as soon as the feature has real traffic.
- A provider key in frontend code or in a repository, because the box was “only staging”.
- Retries with no timeout, so a slow provider is called three times and billed three times.
- Failover that repeats a side effect: two emails, two charges or two tickets.
- A cache of raw customer text that outlives the log.
- No alert on the cap, so the first sign is a feature that has already stopped.
- A private key in every microservice, so rotation is a project instead of a config change.
How we work
- Discovery. Which features call a model, where the keys live, and which text must never be logged or sent out.
- Plan. Cap, rate, timeout, failover order, and whether any feature needs a local model.
- Build. The endpoint, secret storage, redaction, cache rules and counters your monitoring already understands.
- Test. A timeout, a provider error, a reached cap, and a prompt that contains an email or a key.
- Launch. One feature moves onto the AI gateway first. The direct call is removed only after that path is quiet.
- Support. Alerts, retention and key rotation stay on a short list you can run without us in the room.
Why CodeWiz
- APIs are the daily work. This layer is another API, with rate limiting, secrets and logs, beside the rest of the system.
- Cloudflare, or a server you control. The edge when the zone is already there. A service next to the app when the keys must stay on your machine.
- Local models are a real option. Hosted APIs for public features. A model on your hardware for text that should stay inside.
- Founded in 2023 by Dragan Plavšić in Banja Luka, working remotely with clients in the region and abroad.
- Tailored, and open-source first. The policy fits your features, you leave with the config, and the price stays fair.
A single script that calls one model a few times a day can stay as it is. An AI gateway earns its place when more than one feature spends, or when a key has started to spread.
Frequently asked questions
Do we need this if we only use one model?
Often yes, for the key and the cap alone. Failover can wait until a second provider is worth it. A script on one server, with a key that never leaves and a bill you already watch, can stay direct. We say which you are after we see where the key lives.
Can the key stay in the browser behind a login?
No. A login decides who may open the page. The browser still receives what the page needs to run. The browser calls your server, and the server calls the model. The secret stays on a machine you administer.
Will failover double the bill?
It adds cost only for calls that move to the second provider, and only when that second call runs. The backup is tried after the first has stopped. A per-feature cap and a short cache move the invoice more than an occasional backup.
What should the log contain?
Feature, model, latency, success or failure, a cache flag, and a redacted trace deleted on a schedule. Token counts help when the provider sends them. Raw customer prompts stay out of the long-term store. If support must read one, keep a short window, then drop it.
Can a local model replace a hosted one?
For some features, yes. Where people compare answers with a large hosted model, that model usually stays primary. An AI gateway lets you mix them: sensitive text stays local, the rest uses the provider you chose, and a failure can fall back without a new deploy.
Put the model calls in one place
Tell us which features call a model, where the keys are, and what must stay out of the log. We sketch the cap, the redaction rules and the failover list before any code is written. Contact CodeWiz, or see the rest of our development services.