What Makes a Good Vulnerability Management Program?

This is a question I have been pondering and talking about lately.  Full disclosure:  I'm not the best person to answer this as I have been primarily and deliberately on the Threat side of security for a while now.  My experience has been a lot of vulnerability programs are really just network or applications scans with a compliance piece bolted on.   Having said that, here's a vulnerability program according to me with some ideas thrown in from recent conversations with others. A vulnerability program needs the following steps:

  • Understand your environment
  • Identify vulnerabilities
  • Validate
  • Prioritize
  • Remediate (or accept or mitigate)
  • Report on all of the above.

If you're not careful though, it can easily turn into a compliance program rather than a security program.   Let me explain.

Vulnerability Management for compliance

A lot of organizations start by picking a tool to scan the network looking for known bad stuff.  These tools generally have a reporting engine and might even have a way to flag false positives and track remediation.  Most everybody realizes there are false positives and filter out a few glaring exceptions.  Once that is done, they agree that the list of things left are bad and must be fixed.  It becomes a routine, the metrics get better and may even approach zero.  Man, that looks great in the monthly operations review.  It looks like a very mature program and it is.  The question is:  Are they more secure?  Absolutely.   Every organization should get to this place.

Once your network hits a certain size, you have to be more selective about where to scan.  We have data centers that literally take tens of hours to finish a full network scan.   There are all kinds of shortcuts you can take like doing a ping sweep first or focusing on a particular service (HTTP, SMTP, etc).   You'll start to miss things, by the design.  You may have critical services that run on non-standard ports or boxes with host-based firewalls that don't answer to pings.  Now what?  This is the point where a lot of organizations start to flesh out the reporting, showing progress, and advertising their program's maturity.

The scenario above will take care of what I'm starting to call "commodity vulnerabilities".  They are pretty well known weaknesses that no network should have.

Now, enter this whole APT world.  I'm pretty sure most professional attackers know what commercial IDS and vulnerability scanners look for.  These attackers likely have major commercial tools in their lab.  This is their baseline.  It's their starting point.

Vulnerability Management for security

Let's start look what the next steps could look like using the list of steps above.  This is where pen-testing or "red teams" and different tools come into play.

Understand your environment

Go out to the business and have conversations about what data is important.  What data can have the biggest impact on your business if it gets into the wrong hands.  People generally know.  Sometimes they don't.  Start the conversation, build trust, and people will open up.

Start your search for vulnerabilities as close to the important data as possible and work outwards from there.   Remember also to protect administrators, their credentials, and the hosts they use just like the data repositories.

Identify vulnerabilities

This step can start with the commercial vulnerability scanner from the compliance scenario above.  It cannot stop there though.  You scan use slightly more offensive commercial tools like Metasploit to get your feet wet.  Then probe systems manually, look at header data coming back from servers, try to modify URLs of web applications, etc.  Write scripts to help you.   Automate as much as you can and no more.  One trick here is keep it simple enough that you can have metrics.

Validate the vulnerabilities

Once you have vulnerabilities identified, you have to validate them.  An analyst with a solid technical understanding of the vulnerabilities is required here.

Is it possible to cause a denial of service?  Can you actually get data?  In other words, make sure you can state the issue in plain language terms and that your assessment is accurate.  This sounds obvious, but I've seen commercial scanners say a particular instance of Apache is out-of-date only to go to the server and see the current version of Apache installed.  Tools lie sometimes.  If you do not validate the vulnerabilities, eventually, your team's credibility will suffer.  That's bad news for a vulnerability program.  It's a hard enough sell when people trust you.

Prioritize the vulnerabilities

This really is a key step and a lot of people mess it up.  Again, an analyst with technical chops is required.

Obviously, a denial of service is less important than the ability to execute arbitrary code, right.  Well, not if the denial of service is on your main customer support site.   There are these kinds of things that are difficult to put into a tool.


Now, that you've got real vulnerabilities and know which ones you want to tackle first.  What are you going to do with them?  You have about three options:

  • Accept
  • Mitigate
  • Remediate

Accepting the vulnerability is just that.  In pretty much any compliance framework, the management of an organization has the flexibility to just ignore something as long as they document it.  This officially puts an issue out to pasture.  This is where you're best bet is to put on your consulting hat.  Do not get emotionally attached to your advice if you see the senior leadership is going to accept the risk.  It's their business, not yours.  Learn FIDO.  Forget It and Drive On.  There are other vulnerabilities to fix.

Mitigating the vulnerability is putting something in place to attempt to prevent it being exploited even though the actual weakness still exists.  For example, you could have a vulnerable web server, leave it unpatched, and just put a host-based firewall that only allows traffic to that web server from known, trusted hosts.   If someone is able to exploit one of those known, trusted hosts, you're out of luck.  But, it's better than nothing.

Remediating the vulnerability is getting that weakness out of your network (as far as you know).  The cardinal rule of programming I learned over twenty years ago is:  There  is always one more bug.  This is the most secure route.  To get here, you need to learn how to explain the vulnerability in a way that resonates with the business owners and support staff of the systems.

Report on progress

This is the part where you toot your own horn.  More importantly, this is what builds your team's credibility over time.  The most effective programs combine vulnerability reporting with threat data from their network.  That can show that you're getting owned because you have not fixed problems or it can show the attacks being thwarted because you plugged the holes.  Either way, it shows you have a handle on your space.

The reporting will also help strategically with lessons learned, drive future technology deployments, and help define security standards such as baselines.   Tactically, the output of your vulnerability management program will aid in making intrusion detection more focused.  If you know where your weaknesses are, it would be a good idea to tighten up detection in those areas, if possible.

Be as specific as you can about the following:

  • What versions of a particular application are attacks targeting?  Contrast this to the application versions on your network.
  • Did you see data leaving your network owing to one of these vulnerabilities?  If so, show samples of this data if at all possible.  This makes it personal.
  • What parts of the business are affected by successful attacks?  This can help target user education programs.
  • What percentage of your company has a certain vulnerability?  Rarely is 100% of your company vulnerable.  Show where some parts are patched.  This gives you a foot in the door to patching everything else.
  • What progress is being made on remediation?  Everyone loves good news and hopefully this is good news.
  • How long does remediation take?  The Ops folks love to look at MTTR over time.

Set targets for your number based metrics and remember that Zero is a valid target.  Do you really want to advertise a goal that is more than zero successful attacks against your company's assets?  At the same time, 100% compliance on patching may not be achievable.  Do not let the perfect be the enemy of the good.

Well, that's my version of a vulnerability program for now.  I'd love to hear thoughts, suggestions, corrections, etc.