Filtering by Tag: siem

Is Big Data Analytics for Security really just SIEM 2.0?

A lot of companies are now touting big data analytics that will find badness that your other security tools do not find.   Almost every pitch I see has something very much like the SIEM Funnel from 2005.  The assertion tends to say these solutions would have bubbled those two or three compromised Target servers up to the top.  It will be interesting to see what this segment of the security market looks like in two years. For those of us that bought into the SIEM market, we found out that a SIEM is very needy in terms of man-hours.  There are several reasons for this:

  • Getting all the data feeds going (no small feat in most companies)
  • Building network and asset models
  • Creating rules, dashboards, reports, etc
  • Tuning out false positives
  • Troubleshooting the inherent performance problems of processing hundreds of millions of events per day

In short, you spend a lot of time up front getting the system going and on-going effort to keep it running.

These new products want all your SIEM data plus many other sources like DNS queries, and netflow.  Think about that volume of data for a second.

Here are the typical promises for today's big data security platforms:

  • No rules to create
  • Zero (or very few) false positives
  • Scalability - these systems will not be over-run with data
  • Little to no analyst time required to find badness
  • Full kill chain visibility for compromises

I suspect the majority of these systems will fall short of the promises laid out above.  However, I do believe we will have some great tools a couple of years from now that will make us wonder if we really do need that SIEM.  Oh, and by the way, this is not a "SIEM is dead" post by any means.  There will almost always be a place for SIEM in the SOC environments.

ArcSight Rule Testing Tip

One thing I learned today is patience when testing an ArcSight rule on old events.  This has always sort of been voodoo to me.  Maybe everyone knows this already, but I thought I'd share. When testing a new rule using the "Test" button (see image to left) in the Rule definition window there is a behavior difference.   It essentially uses an Active Channel and inserts your test rule into that stream of events.

So, it looks like any other active channel with one exception.  The channel will speed through messages (examples below) such as  showing you "Percentage Complete", telling you it is Retrieving events, and may even show a message saying "No data matches this query".    This does not necessarily mean your rule does not work.

Just let the Active Channel sit there for a few minutes and you may get results.  Apparently, when ESM says it is done, it is not done.

This has tripped me up many times and I finally figured out it was lying to me today when my ADHD kicked in.  I came back to the channel later and had results.

So, I learned something new today after seven years of using ESM.   Sort of embarrasing, but hopefully it can help you test your rules better before putting them in production.

Until next time... Wyman

Custom Email Notifications with ArcSight ESM

Email notifications from your SIEM can be very useful especially if you have a small team.  The default in ArcSight ESM is to dump every event field into an email.   While better than nothing, the format is hard to read and forces you to search for the information you need to work the event.  Enter custom notification templates. Disclaimer:  This works as of ESM 4.5.  I have not tested it on ESM 5.0.  Would love to hear your experiences!

There are a few of moving parts and I'll go through each below.

  • Rule Actions- You need at least two rule actions.  See screen shot below for an example.
    • Send Notification - Set AckRequired as you wish.   The NotificationMessage is what will ultimately be the subject line of the email.  Resource is where you pick what group of users the email will go to.  This will have to be a group.
    • Set Event Field Actions - This is the secret sauce.  You can technically pick any field name.  I typically use one of the flexString fields to avoid a conflict of that field being used in some other way.  For agentSeverity, it really is up to you and does not affect the email.

  • Notification conf file - This file is in $ARCSIGHT_HOME/config/notification/ and named Email.vm and has logic to help it decide which template to use when sending the notification.  In a clean install of ESM, it only has one option, the default template I mentioned at the top of the post.  It makes decisions based on a particular field.  In this example, it is the flexString1 field shown in the screen shot above and code snippet below.  The decision works from top to bottom using Velocity, so make sure any custom entries are above the default entry.  Add an entry like the one below to the file for each custom template you have.  The #parse field will be the name of the Notification Template File described in the next section.
#if($introspector.getDisplayValue($event, "flexString1") == "malware")
#parse ("Custom-Email-Malware.vm")
#end
  • Notification Template File - This is the file that will literally be a template for the body of the email.  Remember the subject line of the email is set in the Rule Action.    For this example, the template file name is "Custom-Email-Malware.vm".  The file format again uses Velocity and is pretty straight-forward once you have seen one.  See example below.
Description: $introspector.getDisplayValue($event, "name")
Event Time:  $introspector.getDisplayValue($event,"endTime")
--------------------------------------------------------------

User Name:  $introspector.getDisplayValue($event,"sourceUserName") 
IP Address: $introspector.getDisplayValue($event,"sourceAddress") 
Host Name:  $introspector.getDisplayValue($event,"sourceHostName")
Location:   $introspector.getDisplayValue($event,"sourceZoneName")

Target Port: $introspector.getDisplayValue($event,"targetPort")
Event Count: $introspector.getDisplayValue($event,"baseEventCount")

Extra Information (where applicable)
---------------------------------------------------------------
$introspector.getDisplayValue($event,"message")

Description of Event
---------------------------------------------------------------
This computer appears to be infected with malware 

Why this is Important
---------------------------------------------------------------
Malware can take complete control of a computer remotely.

Next Steps
--------------------------------------------------------------
Start the malware infected procedure on this computer.

Note all the field names in both the config file and template file start with a lowercase letter, have no spaces, and each word is capitalized except the first one.  This can bite you if you are not careful.

Good luck!  If you have questions, comments, or suggestions, please leave comments below and I'd be glad to help.

ArcSight Rules - Suspicious, Malicious, or Delicious?

This post was inspired by a reader that asked about creating a channel in ArcSight ESM that would prioritize events based on previous behavior of a computer.  I'll be as generic as possible as there are a lot of use cases for this type of logic. Resources Needed:

  • Active Channel
  • Active List
  • Rule
  • Notification

There used to be an example of this in the default content of ESM.  Hopefully, it is still there.

Example Use Case (from reader):

Observe external computer scanning and set priority to 1.  Observe a priority 1 computer attempt an exploit set it priority 2.  Observe a successful exploit or C2 traffic from target ring a siren and blink a red light.

Basic solution:

  1. Create an Active List (AL) called "Suspicious External Hosts" that keeps track of at least Source IP and Source Zone URI.  Set TTL on AL to something reasonable for your organization.  Could be one hour; could be 30 days.
  2. Create a rule with your definition of "scanning" with an Action of "Add to Active List".   Put the source IP in the Suspicious External Hosts" AL.  Make part of the rule condition NOT in AL to prevent repeat firing of the rule.
  3. Create an Active List called "Malicious External Hosts" just like Suspicious External hosts in Step 1.
  4. Create an Active List called "Targeted Internal Hosts" just like Suspicious External hosts in Step 1.
  5. Create a Rule looking for attempted exploit or attempted unauthorized access.
  6. Include a condition that the Source IP of this event be in the Suspicious AL.   Add an Action of "Add to Active List".  Put the source IP in the Malicious External Hosts" AL.  Put a second Action of "Add to Active List".   Add the Destination IP into the "Targeted Internal Hosts" AL.
  7. Create a rule looking for C2 traffic with a condition that the source IP be in the "Targeted Internal Hosts" AL.  Add an Action of Notification which will send you an email when the rule fires.
  8. Same as Step 7 except looking for successful exploit instead of C2 traffic for the condition.
  9. Create one or more active channels that use the Name field and looks for the names of the Rules created above.  Your analysts can either watch this channel to be proactive or wait for the notifications to come in.  Really depends on their workload.

Closing Tips

I would personally add another Active List and AL condition for Steps 7 and 8 to do what I call "throttling" to prevent the rule from firing excessively.   I'll write about throttling rules later if anyone is interested.  Essentially, it uses the SIEM's intelligence to prevent the "Boy Cried Wolf" syndrome and the reason for so many ignored email alerts in IT.

Make sure every field you use in the Condition tab is added to the Aggregation tab in the Rule.  Otherwise, the Rule will never fire.  Wish I had a dollar for every time I made that mistake.

Custom notification templates and the use of Flex String fields could come in very useful to make the notifications more actionable.  This is also a topic for another time.  Let me know if you're interested.

This is all off the top of my head.  If I missed something, let me know and I'll be happy to update the post.

ArcSight Rules - ESM is not meant to be broken

One of the most important content types you will create in ArcSight ESM will be rules.  In this post, I will try to offer some guidance based on the past seven years I've been exposed to ESM. The heart of the correlation engine is driven by rules.  The decisions your rules make will ultimately drive the decisions and workflow in your detection and response program.  You can get a false sense of security or wind up trying to drink water from a fire hose if you're not careful.

I have overwhelmed myself a few times working with rules.  At other times, I've waited and wondered why a rule was not firing only to find out I forgot a key condition or aggregation field.  Hopefully, you will further along after reading this post.

With rules there are four key areas to pay attention to:

1. Conditions * Order your condition filter from most specific to least specific condition.  This will allow ESM to quickly eliminate events that do not match. * Use = instead of !=.  This also helps ESM eliminate events quicker. * When possible use exact case.  This will reduce the resources the rule needs. * Using indexed fields reduces load on the database server. 2. Aggregation * Keep timeframe to less than five minutes.  Partial matches have be kept in memory. * For timeframes > than five minutes use Active List to eliminate partial matches. * Any field used in the rule condition must be added on the Aggregation tab or the rule will never fire.  This has bitten me many times. 3. Actions * Use notification templates to help make email alerts more readable.  The easiest way to do this is using a combination of the "Set event field" action plus some magic on the manager in $ARCSIGHT_HOME/config.  Details will come in a later blog post. * Actions also are where you would add/remove things from Active Lists to help with throttling or prioritization. 4. Throttling * This is a term I use.  It is using an Active List in the Condition statement of a rule to keep the rule from firing all the time.  Essentially, the first time your rule fires, it can add values to an Active List.  The condition would include a check of that Active List.  If the value exists, then the condition of the rule would not be met.  This eliminates what I call "The boy cried Wolf" syndrome.  We've all had monitoring systems that email mail bomb us when there is a problem in the network.  This use of Throttling unleashes the power of ArcSight ESM to make your monitoring much more intelligent.

ArcSight ESM rules are a very powerful and flexible resource if you use them well. Things like this are what turn a SIEM into a way to augment your existing headcount.  It frees your people up to work on higher quality problems instead of wading through tons of false positives.  I hope this overview has been helpful and will go into detail on some of these in later posts.

How do you use rules in ArcSight (or another SIEM) to make your team more effective?

SIEM is moving on up

ArcSight got acquired by HP last year.  Now McAfee has Nitro Security and IBM gobbled up Q1 Labs.  Who said SIEMs were not allowed in the cool kids club? SIEM is short for Security Information and Event Management.  Basically, they are high-volume correlation engines where you can dump logs from across your network and have the system make security decisions for you.

If you are doing network security work in the enterprise and do not have a SIEM, your CIO or CISO will be asking you if you need one very soon.  With a product lines in hand, you can be sure these large vendors will be looking to expand their footprint inside your organization.  And my response to choose your flavor and hop on the train!  It will be a great ride with the right people and processes in place.

Hopefully, enterprises will realize that this market is not like other security markets.  So many CISOs want a tool they can purchase and without additional headcount make the enterprise more secure.  Just not going to happen.  Why?

SIEM technology is all about people and process.  You get out what you put in and not a nickel more.  If an Outbound Network Sweep happens in the enterprise and no one sees the alert; it didn't happen.

It is very important to hire analysts along with a shiny, new SIEM.  Event correlation and Incident Response are at the heart of these systems.  Insight and Warning should come out of a SIEM and not just Information.  This is part of what some in the industry call Security Intelligence.

Make sure to read some of the work by Mike Cloppert, Richard Bejtlich and other leaders in the Incident Response and Computer Network Defense area.  While they do not tout themselves as SIEM experts, these two guys will really give you a headstart on any SIEM implementation.

HP Protect 2011

Just got back from the annual ArcSight user conference.  They were bought by HP and the conference was called HP Protect 2011.  The conference was excellent as usual.  You get to hear from developers, support staff, consultants, and other customers.  My focus this year was attending sessions on performance tuning and using threat intelligence to enhance correlation.

I gave a presentation at the conference this year on using ArcSight ESM to catch malware that was not caught by anti-virus software.  My slide deck will be available on this site soon.  Thank you to everyone that attended the session.  It was a packed room.  I hope it was worth your time.  Any feedback is appreciated!