Relevant Technologies: Security Incident Response Planning and Management

< Back to Article List

Security Incident Response Planning and Management
By: Laura Taylor
January 28, 2002

Planning for Recovery
Security incident response is the resulting processes and actions an organization takes in responding to a security intrusion. Any company or organization that does business using the Internet, or private wide-area communications networks, should have a security incident response program setup before a security incident occurs.

Each and every security incident response program that is setup will contain unique elements that exist and make sense only for their organization. It is this uniqueness of the incident response program that is most important. If it were not for the unique elements of an incident response program, organizations could simply use a scripted scenario documented from a security textbook.

Certainly there are best-practice guidelines that are worth following, many of which can be found in leading Incident Response books such as Incident Response: A Strategic Guide to Handling System and Network Security Breaches by Schultz and Shumway, or O'Reilly's Incident Response by van Wyk and Forno. However, the unique elements of an organization's incident response process should contain the specifics and details of their intranet -- details that only someone inside your organization would have knowledge of.

An incident response program details the actions that you take when a security incident occurs. Actions imply that you do something, and in order to understand what you should do in any sort of risk mitigating situation, you need procedures. You also need to know who specifically will perform which procedures, what reports will be generated, and who will review the findings in these reports. Clearly defining roles and responsibilities are part of any worthwhile incident response program.

"Once a system has been compromised you can't trust any of the information contained on it"

To ensure that the appropriate actions are taken by the appropriate people, lines of accountability, guidelines for staff, and contact lists need to be established. An overall incident response manager needs to be appointed. Depending on the size of the organization, the incident response manager could be the director of information technology, the chief information officer, the director of information security, or a security project manager.

Putting the Plan into Motion
When an incident has been detected, it is the first responsibility of the incident response manager to make sure that the incident response team is assembled and notified in a timely fashion. Therefore, prior to the occurrence of any security incident, in the planning phase, detailed contact lists must be put together, and kept up to date. Security incidents are never convenient, and there is a good chance that key personnel will need to be notified off-hours. In the contact lists, you should retain home phone numbers, cell phone numbers, pagers, and all email addresses of everyone listed.

After the team has been notified of the incident, they need to be pointed to the pre-established guidelines, and instructed on which processes and procedures need to be applied to what systems, networks, and applications. Typically it is the organizations systems administrators that will do most of the hands-on work, and then report and log their work on incident tracking forms.

Before recovery from the incident begins, the organization should decide whether the goal of the recovery is to proceed and protect, or to pursue and prosecute. Often times these two objectives conflict, and if that is the case, one or the other needs to be given priority. If the goal is to pursue and prosecute, it is very important not to tamper with the evidence. Not tampering with the evidence means that log files will not be changed in any way, access and creation dates on files cannot be changed, data cannot be overwritten, or transferred over unencrypted intranet links to other systems.

If you have customer systems that need to be repaired within a certain timeframe, and have contractual obligations to do just that, you might not have time to contain the evidence in a way that would give a prosecuting attorney a case worth pursuing. Certainly taking actions like reformatting and reinstalling a disk will most assuredly destroy important evidence. Tough decisions might need to be made on whether catching the perpetrator is more important than restoring customer systems.

"Tough decisions might need to be made on whether catching
the perpetrator is more important than restoring systems"

The uniqueness of your organization's incident response plan will include information about your intranet accounts and data, including where master password lists are kept, and what procedures should be used to access them, purge them, or recreate them. The version of all your operating systems and patch level needs to be known for each and every system on your network - the same holds true for all your applications. In planning your incident response process, you need to have pre-recorded a minimal set of criteria regarding all your systems. Once a system has been compromised, you can't trust any of the information contained on it, because you don't know what the perpetrator has or has not changed. Version numbers might have been changed, patch levels, and of course various Trojan programs could have been installed. For any system that has been breached, the only option is to reinstall the entire system.

If the breached system is a development system, simply recompiling and re-linking the code is not good enough since you don't know whether the compiler or the libraries it uses have been sabotaged. You need to completely rebuild the system, reinstalling the operating system, compilers, and any supporting libraries and applications.

Dissecting a system to find Trojan Horses, and trying to understand what system files have been changed typically takes far longer than simply reinstalling the entire system from scratch. Therefore, a standard recommendation in the world of information security has evolved to reinstall, or build a new system, to replace any compromised systems. A compromised system should not be patched, since you will never know if you patched a recovered the entire system.

Tracking the Incident
For each system that is compromised, an incident tracking form should be filled out with as much information as possible. It is recommended that an incident form be created which outlines which details you'll be looking for. When the incident happens, the systems administrator simply fills out the form, before performing the recovery procedures.

The types of information that you will want to include on your incident tracking form are things like:

Host Name and IP Address
Operating System Name and Version Patch Level Found
Resident Applications (Versions and Patch Levels)
Log files reviewed (filenames and locations)
Evidence cited and Intruder Activity (files that have been changed, edited or removed, trojans)
Security Policy Breached (document what sections of your security policy has been breached e.g. Section 3.2 Password Confidentiality)
Stakeholders (who uses, owns, or is depending on this system e.g. departments, customers)

The Recovery Process
Once the incident tracking form has been completed, recovery of the system can begin. The systems administrator should then move forward with the pre-documented recovery procedures. These procedures, which should have been drafted prior to the incident, should include information unique to your intranet and organization such as:

What media to use to install the system (and where to find the media)
Detailed installation procedures (or a pointer to the vendor installation guide)
License information for the operating system
License information for the applications
Special files that need editing (DNS, default router etc.)
Restoration of user data, accounts, and profiles
Restoration of symbolic links, mount points, or shares
Registry or kernel modifications
Restoration of high-availability or load-balancing functionality

Women's Baltimore Ravens '47 Black Billie Adjustable Hat,Green Bay Packers 4-Time Super Bowl Champions Commemorative Gnome,Mens Washington Redskins Nike Red Quarter Zip Hot Jacket Patriots Jerseys On Sale.Men's San Francisco 49ers NaVorro Bowman Nike Scarlet Player Pride Name & Number T-Shirt,Women¡¯s Detroit Lions Light Blue Game Changer Tube Dress Patriots Jerseys Wholesale.Women¡¯s Chicago Bears Cuce Shoes Black Enthusiast II Rain Boots,Men's Philadelphia Eagles Brian Dawkins Mitchell & Ness Midnight Green 1996 Throwback Authentic Jersey,Men's New Orleans Saints New Era Black Solid Cuffed Knit Hat.Women's New Orleans Saints Wedge Flip Flops,Men's New York Giants Starter Royal Pro Jacket New England Patriots Jerseys Cheap.Men's Cleveland Browns Jim Brown Majestic Brown Hall of Fame Eligible Receiver II Name & Number T-Shirt,New England Patriots Tervis 24oz. Bubble Up Wrap Tumbler with Lid,McArthur Chicago Bears Hooded Baby Towel.Men's Dallas Cowboys New Era Black Crux Line Neo 39THIRTY Flex Hat,Girls Infant New England Patriots Gray 3-Pack Field Goal Creeper Set,San Francisco 49ers Logo Coir Door Mat.Mens San Diego Chargers '47 Brand Gold Cleanup Basic Logo Adjustable Hat,Men's New Orleans Saints Majestic Gray Big Sizes Critical Victory Sleeveless T-Shirt Patriots Jerseys Cheap.Denver Broncos Team Logo Cufflinks,Pittsburgh Steelers Toddler T-Shirt & Short Set - White/Black,Men's New England Patriots New Era Navy 2015 On-Field Sport Knit Hat with Pom

Of critical important is not just restoring your systems, but in ascertaining the intruder's point of entry, and securing that entry point. Did the intruder enter your network through a buffer overflow attack? If that is the case, installing a stack security product on the appropriate systems can prevent this from happening in the future.

"If you don't have a firewall now is clearly the time to procure one and install it"

Were any setuid files exploited? If that is the case, the permissions on these files need to be corrected, or some alternative file should be put in their place.

Were passwords compromised through a dictionary attack? You need to check and make sure your users are using secure passwords that used mixed-case characters and do not spell out words. Run a password cracker on your password lists and report weak passwords to management.

If a firewall was installed and up and running, look at the log files for connection activity, and also look to make sure that that rules are configured properly. If too many holes were opened on the firewall to accommodate bellicose users, it's undoubtedly a good time to take back the firewall, tighten it up, and instruct the users on the merits of using a secure remote access VPN. If you don't have a firewall on your network, now is clearly the time to procure one and install it. A firewall pays for itself if it prevents even one incident from happening.

There are endless possibilities for points of entry. Go through a list of the various possible attack scenarios and see if one seems to fit the picture of your network and systems.

"Almost all the large technology companies have been victims of security incidents"

After the incident has been cleaned up, it is probably a good time to start thinking about procuring and installing an intrusion detection or prevention system. The good news is that there are quite a few sophisticated network and host intrusion detection products on the market today. Figure out which products make sense for your organization, and make it a priority to install them, monitor them, and understand their reporting capabilities.

Last, if your organization does not have the sources to recover from the incident, look for a consulting organization that specializes in incident recovery. When you interview these organizations, ask them what their action plan will look like. It is very possible they will not be able to give you references, even if they do a quality job, because many organizations do not want it revealed that they have suffered a security incident.

The best thing you can do is learn from the recovery process so that you can minimize the possibility of future incidents. If it makes you feel any better, it might be worth noting that almost all the large technology companies have been victims of security incidents at one time or another (though many have been adept at keeping such reports out of the Wall Street Journal).