How to connect production LLM monitoring

An adversarial scan and production monitoring observe different inputs

A scan actively sends a selected collection of test prompts to a model or endpoint. It is useful before a release and after a fix. Production monitoring evaluates traces your application submits from actual operation.

Neither gives the other automatically. A catalogue report does not prove what your users encountered yesterday. A monitoring dashboard with no ingested traces does not demonstrate that production traffic is safe. Monitoring is also distinct from a blocking guardrail: an evaluation or alert does not by itself stop an action.

Decide what a trace may contain

Identify which prompts, responses and metadata are necessary for your review. Remove secrets and unnecessary personal data before collection. Confirm the access permissions, retention expectations and legal basis for the information your organisation sends.

Keep provider credentials separate from the STAIPH API key used to submit traces. Provider keys fund model scans; a STAIPH key authenticates application API calls. Generate and keep application keys on a trusted backend, rather than embedding them in a public browser bundle.

Connect one system before connecting all traffic

Use the API quick start in your signed-in workspace so the endpoint and authentication examples match the current service. A blank dashboard is an integration state to investigate, not a successful security assessment.

1. Activate an eligible Pro trial or subscription. Access follows verified billing events, not the checkout return URL.
2. Create a system in the monitoring workspace and generate its STAIPH API credentials.
3. Send one synthetic, authorised trace from a server-side staging environment using the workspace API guide.
4. Check ingestion, evaluation status and the matching system. Distinguish rejected input, pending work, evaluated output and provider errors.
5. Configure and test the alert destination under your control, then send a small production sample and review its usefulness.

Know the allowance and what happens at the limit

STAIPH Pro includes 5,000 successful monitoring evaluations per monthly period across up to 3 systems, at €99 excluding tax per month. Provider model API calls are billed separately. Catalogue scans and monitoring evaluations use separate allowances.

New monitoring evaluations stop at the period allowance instead of generating automatic paid overages. Review the usage meter, ingestion errors and failed evaluations. A period with no new evaluations may reflect a quota, inactive subscription or integration failure; it should not be interpreted as an absence of security events.

Turn a finding into an engineering decision

A flagged response needs someone to review the evidence, decide whether the behaviour violates the application’s policy and choose a correction. Keep false positives visible rather than turning every flag into an incident.

When a production case exposes a weak boundary, add a redacted version to your regression suite. Test the correction against that case and ordinary usage. Use repeat scans after model, system-prompt, retrieval or tool changes. A monitoring service supports this process; it does not replace application authorisation or incident response.