Methodology

The short version

Every comparison on Breachfolio is grounded in hands-on testing in a controlled lab. Every score has a version number. When the test plan changes, prior comparisons are re-run and the diff is published.

The lab

Comparisons are run on dedicated hardware that is not in production use. Network captures are isolated. Targets are systems we own or intentionally-vulnerable VMs designed for testing. No external system is ever scanned, probed, or interacted with as part of testing.

Test plans

Each comparison is governed by a written test plan that defines:

• The scenario the tools are being evaluated for.
• The exact commands or workflows being executed.
• The success criteria.
• The version of every tool involved.

Test plans are versioned. When we change a test plan, the version increments and the affected comparisons are re-tested.

Scoring

Scores are qualitative more often than numerical. When a number appears, the methodology behind it is linked from the same page. Numbers without a published method are excluded.

Reproducibility

Where it is safe and legal to do so, raw artefacts (network captures, command output, configuration files) are linked at the bottom of each comparison. The intent is that a competent reader can re-run the test themselves and disagree with our conclusion.

When we are wrong

If a vendor or reader points out a factual error, we correct the article, note the correction at the top, and keep the original wording visible. We do not silently rewrite history.

Use of AI tooling

Parts of Breachfolio — drafting, editing, code, and some illustrations — are produced with AI assistance. That does not change what gets published: every technical claim still has to survive the lab work described above before it goes live, and a human checks it against the actual test artefacts first. AI tooling speeds up the writing. It does not replace the testing, and it never publishes anything on its own.