Many articles discuss the differences between penetration tests and vulnerability assessments (just Google it), but I still run into questions from clients regarding the distinction between the two, along with questions about red team engagements and capture-the-flag exercises. To add to the confusion, vendors will often create their own definitions, conflate terminology, or make the material difficult to understand by using obscure technical jargon that leaves clients bewildered.
So, here’s my two cents on the subject that will, hopefully, clarify the differences for you. I’ll start off with some of the definitions we use at Bridges Consulting, followed by a few examples of how these concepts work. So, let’s begin…
A vulnerability scan uses a variety of commercial or open-source (i.e., free) tools, called vulnerability scanners, to “scan” a target system for known vulnerabilities
that could potentially impact the target’s web application or network environment.
A vulnerability assessment is a formatted report that outlines the results of the vulnerability scan to the client by vulnerability type and severity. Vulnerability assessments frequently use the Common Vulnerability Scoring System (CVSS), a widely-accepted industry standard, to calculate the severity of vulnerabilities found in a scan.
A penetration test, or “pentest,” is the process that seeks to validate if the vulnerabilities found during a vulnerability scan present a real threat to your web application, network, or physical environment. Pentesters employ an empirical methodology (i.e., the process of verifying from first-hand observation) to evaluate the results from a vulnerability scan through the deliberate exploitation of identified vulnerabilities and subsequent penetration of the vulnerable systems that possess them.
A red team engagement is a simulated attack conducted by a group of pentesters who assumes an adversarial role against a specified target. Red team engagements are designed to test the efficacy of an organization’s security program, incident response (if any), and mitigation controls through any and all possible methods (e.g., network hacking, social engineering) within the established parameters of the engagement (which must be accepted by both client and red team participants).
A capture-the-flag (CTF) exercise requires one of more teams to pursue and acquire a decoy (hence, capture the flag) embedded in the physical or network test environment to demonstrate the successful breach of a target. Multi-teamed CTF exercises proceed in a cooperative (i.e., working together) or competitive/zero-sum (i.e., working against each other) environment and promote innovation in network and social-hacking techniques.
WHERE DOES THE CONFUSION COME FROM?
Throughout the course of my consulting activities, I meet clients who tell me they conduct their own vulnerability assessments, perform their own penetration tests, and/or hire a third-party vendor to perform an independent penetration test on their systems to demonstrate due diligence[*] per HIPAA guidance. When I ask them to define what penetration testing means to them, they inevitably describe the process for conducting a vulnerability assessment.
When I ask them to describe the penetration testing service they outsource to third-party vendors, they tell me that the contracted pentesters start by deploying vulnerability scanners on their network and finish by listing discovered vulnerabilities, in order of severity, on a report as confirmed vulnerabilities. This scenario illustrates how difficult it is to draw a distinction between a vulnerability assessment and a penetration test, but before I outline the differences, I’ll start by addressing organizations that perform their own penetration tests and why it’s important to contract a third-party vendor to do them.
THE NEED FOR INDEPENDENT PENTESTING
First, a penetration test performed by members of your own organization will always result in a biased outcome, no matter what steps you take to make the pentest objective in scope. The reasons for this are simple: if you work for your organization, you understand aspects of the network environment that, hopefully, most outsiders don’t, so the pentester either consciously or subconsciously relies on that knowledge to inform his/her process.
Second, if the pentester supports the organization’s IT department, then he/she is susceptible to real or perceived pressure from management to present a report of relevant findings that will not make them look bad. In other words, internal pentests often satisfy the minimum requirement set by management to comply with their security policy objectives. An independent pentest, on the other hand, ensures that the third-party remains objective and unbiased. Now, on to key differences between vulnerability scans and penetration tests…
The confusion begins with clients and vendors conflating the use of vulnerability scanners with penetration testing. Organizations that perform their own vulnerability assessments deploy third-party vulnerability scanners on their network on a periodic or recurrent basis (in what can be termed as continuous monitoring, or CM, for short).
Therefore, the difference between an internal vulnerability assessment and an external one (based on the previously mentioned example) is that some organizations purchase third-party vulnerability scanners to perform CM tasks while others subscribe to a third-party vendor’s CM service to monitor their networks for vulnerabilities and call it a vulnerability assessment. Fair enough, but when that same organization hires a third-party vendor to perform the same tasks they already do for themselves and call it penetration testing, then the organization does itself a disservice by paying for something they purchased already; in other words, duplication of effort.
Third-party vendors offering penetration testing services don’t help with the confusion either. Many will market their services as penetration tests when, in fact, they are selling vulnerability assessments. Even worse, some of these same vendors claim to use their own proprietary scanners when they actually subscribe to a subscription-based scanner themselves, just like their clients do.
The problems begin when an organization performs their own version of a “vulnerability assessment” using the same scanners, unbeknownst to them, as the third-party vendor. This creates for an assessment that doesn’t help an organization’s security because the results are the same for both client and vendor. This is why Bridges Consulting draws a distinction between a vulnerability assessment and a penetration test when we market our services to potential clients. The prospective client needs to understand what they are purchasing and know that the service they choose is different from what they do already (if they do their own vulnerability assessments).
Furthermore, a true vulnerability assessment should be performed by a company that understands the client’s expectations and knows how to interpret the results of a vulnerability scan within the context of their client’s environment and HIPAA guidelines, not just copy/paste the results from the vulnerability scanner and call it a day. By offering a Risk & Vulnerability Assessment (RVA), Bridges Consulting integrates risk analysis with a vulnerability assessment to offer a probabilistic overview of our clients’ security posture, then performs penetration tests to determine whether the vulnerabilities we identified in an RVA behave as predicted under real-world conditions in the target environment.
SO, WHAT’S THE DIFFERENCE BETWEEN VULNERABILITY ASSESSMENTS, PENETRATION TESTS, RED TEAM ENGAGEMENTS, AND CAPTURE-THE-FLAG EXERCISES?
- A vulnerability assessment remains a primarily automated process whereby an organization or a third-party vendor/consultant runs a vulnerability scanner against a target environment to test against known vulnerabilities.
- The results of a vulnerability assessment typically represent a list of vulnerabilities, which require a penetration test to prove if the potential vulnerabilities are exploitable in a controlled and secure target environment.
- A red team exercise simulates a real-world attack scenario, where a team or group of teams attempt to exploit any vulnerability in the target environment and focuses on testing the organization’s security, incident response, and resilience in the face of an attack.
- A CTF exercise merely adds a decoy to the target environment as validation of a successful attack.
Additionally, a CTF exercise in the commercial space usually involves two or more teams competing against each other or working in tandem to compromise a rigged, virtualized network target environment. CTF exercises in the government, by contrast, typically involve only one team attacking a real network or physical asset that’s not virtualized. In recent years, CTF exercises in government agencies are commissioned to replicate insider threats like Edward Snowden and reflect someone stealing sensitive data from a specific folder or directory.
- Vulnerability assessments reveal industry-recognized vulnerabilities found on a target environment, but the vulnerabilities remain theoretically exploitable unless they are tested under real-world conditions.
- Penetration tests provide an empirical evaluation of theoretical vulnerabilities found during a scan and validate if they are exploitable in a real-world environment.
- Red team engagements test an organization’s resilience and incident response against a simulated attack scenario.
- Capture-the-flag exercises promote innovative techniques by creating a [competitive or collaborative] framework
for team[s] to acquire a decoy in the target environment and test an organization’s efforts to defend it.
- An independent penetration test performed by a third-party vendor is always a great way to demonstrate due diligence and
compliment your own organization’s penetration testing initiatives (if your organization performs internal penetration tests already).
- Bridges Consulting offers a full range of vulnerability assessments, including our RVA methodology, along with penetration testing, security audits, and a hybrid red team/capture-the-flag model of evaluating
when they have taken every possible step to reasonably prevent the risk of