What an actionable AI security test report should contain

The target and the tested scope come first

A report should identify the provider, model or endpoint, the test time and the configuration available to the evaluator. Treat missing metadata as a limit, not something to fill in with an assumption.

Distinguish a provider model test from a test of an application endpoint. Do not infer that a report exercised RAG permissions, a system prompt or external tools unless the test actually included them. A catalogue mapped to OWASP categories is not an OWASP certification.

Separate planned, processed and assessed attempts

Planned attacks describe the requested run. Processed attempts describe how far execution reached. Assessed responses are the outputs for which a verdict could be calculated. Authentication errors, throttling and timeouts can reduce useful coverage.

For example, a fictional run could process 30 attempts while only 24 produce assessable responses. If 6 of those 24 are flagged, the observed attack success rate is 25%, not 20%. The 6 unassessable attempts need investigation. If zero responses are assessable, the score and rate should be unavailable.

Illustrative coverage
Planned: 30
Processed: 30
Assessed: 24
Flagged: 6
Unassessable: 6
Observed attack success rate: 6 / 24 = 25%

Read the score as a measurement, not a guarantee

STAIPH’s base security score comes from the severity-weighted rate of flagged responses. The ordinary attack success rate uses the number of flagged responses divided by assessed responses. Because the score is weighted, it is not necessarily 100 minus the unweighted rate.

Base verdicts use automated refusal detection. Both false positives and false negatives are possible. Additional ARES review, when available, is reported separately with its actual coverage and disagreements. It does not turn an automated report into a human penetration test or a probability of financial loss.

Each finding needs evidence and a next action

A useful finding connects an input to an observed response and an explanation of the boundary that may have failed. Show the attack family, severity, response excerpt and recommended correction. Preserve enough detail for a reviewer to reproduce the result on an authorised test target.

Recommendations should be specific to the observation: remove secrets from system instructions, treat retrieved text as untrusted input, enforce tool authorisation outside the model, or check output handling. After a change, rerun the failing case and legitimate usage. A recommendation is a starting point for engineering review, not proof of remediation.

1. Verify that the cited response supports the finding.
2. Check whether the target reproduced the affected application control.
3. Choose the owner and correction, then record a regression test.
4. Re-evaluate after deployment and preserve the evidence from both runs.

How STAIPH’s preview, one-off report and Pro differ

The free preview runs 30 catalogue attacks and includes a private report and PDF. Guest access lasts 24 hours after the first opening, within 30 days of the run. Provider model calls are billed separately.

The €49 one-off offer includes one full 571-attack analysis, PDF and 12 months of private report access. Start with the preview, pay from its report, then enter your key again after confirmation. The full analysis replaces the preview at the same link. It does not activate a subscription or monitoring.

Pro costs €99 excluding tax per month and adds repeated catalogue scans, history and monitoring of production traces after integration. An eligible new subscriber can use the configured 3-day trial. Keep private report links and provider keys out of public issue trackers and social posts.

An example report to examine

Fictional 30-attack example. These responses illustrate the presentation and are not customer results.