[ previous ] [ Contents ] [ 1 ] [ 2 ] [ 3 ] [ 4 ] [ 5 ] [ 6 ] [ 7 ] [ 8 ] [ 9 ] [ 10 ] [ 11 ] [ 12 ] [ A ] [ B ] [ C ] [ D ] [ E ] [ F ] [ G ] [ H ] [ next ]

Securing Debian Manual
Chapter 11 - After the compromise (incident response)

11.1 General behavior

If you are physically present when an attack is happening, your first response should be to remove the machine from the network by unplugging the network card (if this will not adversely affect any business transactions). Disabling the network at layer 1 is the only true way to keep the attacker out of the compromised box (Phillip Hofmeister's wise advice).

However, some tools installed by rootkits, trojans and, even, a rogue user connected through a back door, might be capable of detecting this event and react to it. Seeing a rm -rf / executed when you unplug the network from the system is not really much fun. If you are unwilling to take the risk, and you are sure that the system is compromised, you should unplug the power cable (all of them if more than one) and cross your fingers. This may be extreme but, in fact, will avoid any logic-bomb that the intruder might have programmed. In this case, the compromised system should not be re-booted. Either the hard disks should be moved to another system for analysis, or you should use other media (a CD-ROM) to boot the system and analyze it. You should not use Debian's rescue disks to boot the system, but you can use the shell provided by the installation disks (remember, Alt+F2 will take you to it) to analyze [76] the system.

The most recommended method for recovering a compromised system is to use a live-filesystem on CD-ROM with all the tools (and kernel modules) you might need to access the compromised system. You can use the mkinitrd-cd package to build such a CD-ROM[77]. You might find the Caine (Computer Aided Investigative Environment) CD-ROM useful here too, since it's also a live CD-ROM under active development with forensic tools useful in these situations. There is not (yet) a Debian-based tool such as this, nor an easy way to build the CD-ROM using your own selection of Debian packages and mkinitrd-cd (so you'll have to read the documentation provided with it to make your own CD-ROMs).

If you really want to fix the compromise quickly, you should remove the compromised host from your network and re-install the operating system from scratch. Of course, this may not be effective because you will not learn how the intruder got root in the first place. For that case, you must check everything: firewall, file integrity, log host, log files and so on. For more information on what to do following a break-in, see CERT's Steps for Recovering from a UNIX or NT System Compromise or SANS's Incident Handling whitepapers.

Some common questions on how to handle a compromised Debian GNU/Linux system are also available in My system is vulnerable! (Are you sure?), Section 12.2.

11.2 Backing up the system

Remember that if you are sure the system has been compromised you cannot trust the installed software or any information that it gives back to you. Applications might have been trojanized, kernel modules might be installed, etc.

The best thing to do is a complete file system backup copy (using dd) after booting from a safe medium. Debian GNU/Linux CD-ROMs can be handy for this since they provide a shell in console 2 when the installation is started (jump to it using Alt+2 and pressing Enter). From this shell, backup the information to another host if possible (maybe a network file server through NFS/FTP). Then any analysis of the compromise or re-installation can be performed while the affected system is offline.

If you are sure that the only compromise is a Trojan kernel module, you can try to run the kernel image from the Debian CD-ROM in rescue mode. Make sure to startup in single user mode, so no other Trojan processes run after the kernel.

11.3 Contact your local CERT

The CERT (Computer and Emergency Response Team) is an organization that can help you recover from a system compromise. There are CERTs worldwide [78] and you should contact your local CERT in the event of a security incident which has lead to a system compromise. The people at your local CERT can help you recover from it.

Providing your local CERT (or the CERT coordination center) with information on the compromise even if you do not seek assistance can also help others since the aggregate information of reported incidents is used in order to determine if a given vulnerability is in wide spread use, if there is a new worm aloft, which new attack tools are being used. This information is used in order to provide the Internet community with information on the current security incidents activity, and to publish incident notes and even advisories. For more detailed information read on how (and why) to report an incident read CERT's Incident Reporting Guidelines.

You can also use less formal mechanisms if you need help for recovering from a compromise or want to discuss incident information. This includes the incidents mailing list and the Intrusions mailing list.

11.4 Forensic analysis

If you wish to gather more information, the tct (The Coroner's Toolkit from Dan Farmer and Wietse Venema) package contains utilities which perform a post mortem analysis of a system. tct allows the user to collect information about deleted files, running processes and more. See the included documentation for more information. These same utilities and some others can be found in Sleuthkit and Autopsy by Brian Carrier, which provides a web front-end for forensic analysis of disk images. In Debian you can find both sleuthkit (the tools) and autopsy (the graphical front-end).

Remember that forensics analysis should be done always on the backup copy of the data, never on the data itself, in case the data is altered during analysis and the evidence is lost.

You will find more information on forensic analysis in Dan Farmer's and Wietse Venema's Forensic Discovery book (available online), as well as in their Computer Forensics Column and their Computer Forensic Analysis Class handouts. Brian Carrier's newsletter The Sleuth Kit Informer is also a very good resource on forensic analysis tips. Finally, the Honeynet Challenges are an excellent way to hone your forensic analysis skills as they include real attacks against honeypot systems and provide challenges that vary from forensic analysis of disks to firewall logs and packet captures. For information about available forensics packages in Debian visit Debian Forensics

FIXME: This paragraph will hopefully provide more information about forensics in a Debian system in the coming future.

FIXME: Talk on how to do a debsums on a stable system with the MD5sums on CD and with the recovered file system restored on a separate partition.

FIXME: Add pointers to forensic analysis papers (like the Honeynet's reverse challenge or David Dittrich's papers).

11.4.1 Analysis of malware

Some other tools that can be used for forensic analysis provided in the Debian distribution are:

Any of these packages can be used to analyze rogue binaries (such as back doors), in order to determine how they work and what they do to the system. Some other common tools include ldd (in libc6), strings and objdump (both in binutils).

If you try to do forensic analysis with back doors or suspected binaries retrieved from compromised systems, you should do so in a secure environment (for example in a bochs or xen image or a chroot'ed environment using a user with low privileges[79]). Otherwise your own system can be back doored/r00ted too!

If you are interested in malware analysis then you should read the Malware Analysis Basics chapter of Dan Farmer's and Wietse Venema's forensics book.

[ previous ] [ Contents ] [ 1 ] [ 2 ] [ 3 ] [ 4 ] [ 5 ] [ 6 ] [ 7 ] [ 8 ] [ 9 ] [ 10 ] [ 11 ] [ 12 ] [ A ] [ B ] [ C ] [ D ] [ E ] [ F ] [ G ] [ H ] [ next ]

Securing Debian Manual

Version: 3.16, built on Sun, 08 Apr 2012 02:48:09 +0000

Javier Fernández-Sanguino Peña jfs@debian.org
Authors, Section 1.1