Choosing IT security testing for your software

Choosing the right test: each one answers a different question

IT security testing is a family of checks, and each one answers a different question. A scanner asks whether you run software with known flaws. A penetration tester asks whether someone could actually get in. The five main options are vulnerability scanning, code analysis (SAST and DAST), penetration testing, security audits and red team exercises. Buying the wrong one wastes the budget.

Most teams over-buy one type. They run a scanner weekly and assume that covers them, or they commission a single annual penetration test and assume the other 364 days take care of themselves.

The useful way to compare options is by the decision each one supports. Are you shipping a release, answering a customer questionnaire, passing an audit, or finding out how exposed you are today?

NIST's SP 800-115 technical guide to information security testing makes the same split: review techniques, target identification and analysis, and target vulnerability validation. The rest of this article maps those onto what you can actually buy.

Scanners find known flaws, testers find chains

Scanners are fast and repeatable but only see what is already catalogued. Penetration testers are slow and expensive per hour but find the combinations that no catalogue lists. You need both, at different frequencies.

Vulnerability scanning covers breadth

A scanner checks your servers, containers and dependencies against public databases such as the National Vulnerability Database. It will tell you that a library you run has a published CVE.

Run it weekly at minimum. The weakness is volume: a first scan of a mature system often returns hundreds of findings, most of them irrelevant to how you deploy.

CISA's Known Exploited Vulnerabilities catalog is a sensible filter. Fix anything on that list first, because attackers are already using it.

Penetration testing covers depth

A penetration tester behaves like an attacker with a fixed time budget. They might combine a low-severity information leak with a weak password reset flow to take over an admin account. No scanner reports that chain.

The cost is that results describe one moment, on one scope. A clean report in March says little about code shipped in June.

Code analysis catches bugs while they are cheap

Testing the code itself is the earliest and cheapest point to find a flaw. The three main techniques differ in when they run and what they can see, and most teams use at least two.

SAST reads the source

Static application security testing scans source code without running it. It catches patterns such as SQL built from user input or hard-coded keys. It runs in the pull request, so the developer who wrote the line sees the finding the same day.

Its failure mode is false positives. If 40% of findings are noise, developers stop reading them, so tune the rules early.

DAST and IAST test the running app

Dynamic testing attacks a deployed build from outside, the way a user would, and finds problems such as missing authentication on an endpoint. Interactive testing places an agent inside the running app to see both the request and the code path it hit.

Both need a working environment, so they run later than SAST. Our QA testing practice wires these checks into the same pipeline as functional tests, so a security failure blocks a release the same way a broken checkout does.

Pick the test type by the decision you face

Match your next decision to the test in the list below: each entry gives what the test finds, how often to run it and its main limit.

• Vulnerability scan. Finds known flaws in software and configuration. Run it weekly or on every deploy. Main limit: it cannot see logic flaws or chained attacks.
• SAST. Finds insecure patterns in source code. Run it on every pull request. Main limit: false positives, and no view of runtime configuration.
• DAST / IAST. Finds flaws in a running application. Run it on every release candidate. Main limit: it needs a staging environment.
• Penetration test. Finds exploitable chains and business logic flaws. Run it yearly and after major changes. Main limit: it is point-in-time and depends on tester skill.
• Security audit. Finds gaps against a policy or standard. Run it yearly or per certification. Main limit: it reviews controls and may not exploit anything.
• Red team exercise. Finds detection and response weaknesses. Mature teams run it every year or two. Main limit: it needs a working security team to test.

Audits and red teams test your organisation

A security audit checks your controls against a written standard. A red team exercise tests whether your people notice an attack. Both look beyond the code and are worth buying only once the basics are in place.

An audit asks whether you have access reviews, logging and a patching policy, and whether you follow them. It produces a gap list. It rarely tries to break in, so an audit pass should not be read as a penetration test pass.

A red team runs a goal-driven attack over weeks, often including phishing and physical access, and measures how long your team takes to notice. If you have no one watching logs, a red team will simply succeed and tell you something you already knew.

For a company without a security team, an independent penetration test gives more value per pound spent than a red team.

Compliance sets a floor for what you must test

Standards decide the minimum testing you need and who may perform it. They are a floor. Meet them early, because customers and insurers ask for the evidence before they ask about your architecture.

The PCI Security Standards Council requires penetration testing at least annually and after significant changes for systems that handle card data. SOC 2 and ISO 27001 auditors expect evidence of regular vulnerability management.

NIST's Secure Software Development Framework, SP 800-218 is the reference US buyers increasingly cite in supplier questionnaires. It asks for code review, testing and vulnerability tracking as ordinary development practice.

Keep the evidence in one place: dated scan outputs, penetration test reports with their retests, and the tickets showing each finding closed. A buyer's security questionnaire is far easier to answer from a folder like that than from memory, and it is cheaper to assemble it before a deal is on the table.

Ask a supplier these questions before signing

A good supplier answers specific questions about scope, people and retesting. A weak one answers with certifications and tool logos. Ask the questions below, and walk away from vague replies.

Who does the work and what do they produce?

Ask for the names and credentials of the testers on your engagement, not the firm's. Ask for a sample report with client details removed. It should include reproduction steps and a fix for each finding.

What is in scope and what happens after?

Agree the target list, test window and rules in writing. Confirm that a retest of fixed findings is included, or priced separately, before you start.

Ask how findings reach your developers. A PDF delivered to a manager who forwards it a month later wastes the test. Findings belong in your issue tracker with owners and due dates.

Can they fix what they find?

Some suppliers only report. Others work alongside your engineers, which shortens the gap between finding and fixing. Either model can work, but check that whoever tests is independent of whoever wrote the code if an auditor will rely on the result. Ask who retests the fixes, and make sure they are not marking their own work.

Start with scans and code checks, then add one penetration test

For most companies with a web or mobile product, the sensible order is clear. Automate the cheap checks first, then pay a human to test what automation cannot see.

1. Turn on dependency scanning and SAST in your pipeline this month.
2. Run a weekly vulnerability scan and triage against the CISA catalog.
3. Commission one scoped penetration test of your main application, with a retest.
4. Add an audit only when a customer or certification requires it.

Our DevOps consulting team sets up the pipeline pieces, and our web application development engineers fix what comes back.

If findings keep stalling between the report and a shipped fix, the broken workflow assessment is a quick way to see where your delivery and testing process leaks. Or send us the scope through the contact form and we will say which test fits before anyone quotes a day.