Skip to content
Qlinnt
All guides

Practical security

How to evaluate a security scanner beyond its alert count

By Qlinnt · Updated

Evaluate the whole scanner using labeled positive and negative cases, not alert volume or persuasive explanations. Measure false positives, missed cases, inconclusive results and cost under a recorded configuration. An offensive-agent leaderboard is not a scanner-accuracy benchmark.

Qlinnt conceptual illustration: How to evaluate a security scanner beyond its alert count
Original Qlinnt illustration. A conceptual explanation, not an attack screenshot or evidence of exploitation.

Understand the problem before changing your site

A scanner that reports more issues may be finding more problems, duplicating symptoms or producing more false positives. Without benign controls and ground truth, those explanations are difficult to separate. A benchmark also changes when its browser, prompts, tools, timeouts or model configuration changes.

Tasks matter. Solving a code challenge, attacking an API and recognizing a public configuration are different capabilities. Do not transfer a ranking from one task to another without testing the workflow you actually use. Public challenge exposure can also influence results.

What to review and how to reduce risk

  • Define the exact claim of each check and create a fixture where that condition is present.
  • Add benign near-matches: an HTML fallback instead of a configuration file, an empty public response, or multiple CSP policies that block the apparent permission.
  • Label expected outcomes before running the scanner. Include not-applicable and inconclusive cases.
  • Record versions, browser engine, time budget and external dependencies. Repeat meaningful comparisons under the same conditions.
  • Report correct findings, false positives, missed cases, coverage and operating cost. Review evidence quality separately from prose quality.

How to verify it with safe tests

For a small deterministic rule, start with six positive and six benign cases. Keep the fixtures isolated from production data. A successful run should identify the intended conditions without labeling the controls as confirmed vulnerabilities.

For browser behavior, compare the rule against harmless markers in the supported browser engines and record which engines actually ran. If a browser is unavailable, say so. A skipped test is not a pass, and twelve curated cases do not establish a population-wide accuracy percentage.

What Qlinnt can check and where its scope stops

Qlinnt's new CSP and inline-SVG rules use deterministic fixtures. Existing integration tests exercise scan orchestration and coverage with controlled responses. This supports bounded behavior claims, not a guarantee that the scanner finds every vulnerability.

Telemetry from visitors measures observed product usage, not detector precision. Use regression fixtures for correctness, Supabase and Sentry for operational outcomes, and PostHog for the visitor journey. Keep these questions separate when assessing improvements.

Does a larger alert count mean a better scanner?

No. Evaluate correct findings, false positives, missed conditions and verified coverage.

Can a leaderboard certify my scanner?

No. Test the scanner's own tasks, tools, versions and evidence requirements.

Sources and further reading

Review your website's public surface

Scan a website you own or are authorized to assess. Verify your email to read the free diagnosis. The scan does not replace code tests or a specialist assessment.

Scan my website for free