Catch 22: Outsourced Incident Response (Plan)

Last Updated on March 7, 2013

Unfortunately, here at Pivot Point we’re all too familiar with Incident Response “Catch 22” situations. Until you have experienced a situation you often can’t (or, more frequently, don’t believe there is a true need to) develop a “comprehensive” Incident Response Plan.
This scenario is on my mind this morning because we spent the weekend helping an eCommerce client who does not have a sufficient Incident Response Plan recover from a very concerning incident. 

• On Friday I got a call from a very good client saying: “We think we were hacked. All of the passwords on a critical, client-facing application were reset.” Here is what happened:
• The “breach” was identified by customer service, which was “flooded” with calls from customers unable to access the application.
• Logs were insufficient to fully determine what happened and who perpetrated it. The IIS and MS SQL logs were reasonably complete — but the underlying operating system logs were not turned on. The firewalls were only logging rejects, so they were not as helpful as they should be.  Common “time” was not being used, which means that logs for each device were offset by a few minutes, which makes investigation a lot less fun and notably reduces response time.
• The IIS logging allowed us to determine that the application was directly taking user input and passing it to the back-end database in a Select statement. By definition this is an SQL Injection flaw. From the logs we could not directly determine if this could have resulted in the issue, so we bounced the concern and page/SQL reference to their development team. Ironically, our client had performed a penetration test against this application this past summer. A recent (insufficiently tested) functionality change caused the problem.
• One of our consultants went on-site with appropriate tools to confirm the analysis done by the client security team and to be available to test any fixes.
• A credentialed vulnerability assessment of the server was conducted to ensure that OS level access was not gained. This included user account review.
• There was no formal definition of an insourced or outsourced Incident Response team. Fortunately, we were able to jump in without delay. If having an external party is critical to your response capability – then it is critical that it is contracted with an SLA. Had it been 3AM instead of 10AM, we may not have been prepared to assist them.  Several hours later the client confirmed that the SQL Injection attack vulnerability we inferred from the IIS Logs was indeed a vulnerability via a Burp Suite scan. So a critical revenue generating application remained “down” — at great financial and reputational cost. 
• By Saturday midday, a fix was deployed for “known -bad” pages in the application and confirmed by the internal security team. Prior to re-deployment we conducted spot checks on the pages in question. On confirmation, the application was re-enabled. 
• The development team was not confident that the “spot” fixes would not have any unintended effects – so we initiated a  full vulnerability assessment of the application.
• A full Incident Response Report was developed and a meeting was scheduled to do a debriefing and update the existing Incident Response Plan.

Needless to say the total “cost” of this incident was significant: lost revenue, reputational impact, legal support, third-party incident response costs, etc. In this client’s case there were multiple lessons learned – most notably around improvements to their pre-deployment security testing practices. 
Five good practices that are likely to reduce the likelihood you spend your next weekend responding to an incident are:

1. Ensure that critical applications/systems undergo appropriate “security certification” prior to being deployed.
2. Ensure that logging is sufficient to detect and respond to security incidents. Be specific about the log events, amount of time, and log sources (OS/firewall/database/web server/app server) required to respond.
3. Put a simple log monitoring tool in place, ideally with alerting on a few events that would be indicators of being under attack and/or that a breach has occurred.
4. Make sure that anyone required to support incident response is aware of their role, contact information is readily distributed, and if necessary, an SLA on their response is in place.
5. If legal action may be considered, make sure evidentiary protocols are established ahead of time and followed.