|
| Membership Services |
By Marcus Ranum
The following remarks were excerpted from a presentation on Intrusion Detection and Network Forensics by Marcus Ranum of Network Flight Recorder (www.nfr.net), delivered at CSI's NetSec '99: Technical Dimensions of Network Security. Ranum's comments on Intrusion Detection will appear in the Fall '99 issue of the Computer Security Journal.
Computer or network forensics is a tough problem. There's no really good art for doing such work right now--and it's possible that there never will be. It's almost always going to depend on the quality of the information that you maintain.
I've done numerous incident responses in my career. In every one, I wished I had more information. Now that I'm recording every packet going into and out of my network I still wish I had more information.
The tools you will need for doing forensics include tcpdump, Argus, NFR, tcpwrapper, sniffers, nnstat, tripwire, backups and a line printer. Yes, if you're doing an incident response, make sure that you have some kind of a printer!
Lawyers believe that paper is harder to tamper with than bits. If you take a bunch of bits that are easy to tamper with and you print them on paper, they're considered "tamper proof." Of course, it doesn't make any sense, but if you walk in with a CD ROM and say, "Here's 300 megabytes of evidence that indicates that this guy was hacking," it's going to carry less weight than wheeling in a trolley of dead trees.
A practical approach
One approach to incident response, that works nicely, is to split up into two teams.
Team A will learn what the attacker is doing, and feed the information to Team B. Team B will generate a "shutout plan," based on the attackers' techniques, to lock them out and keep them out. It is important that you determine in advance when Team A will give up and Team B will execute the shutout.
You've got Team A collecting evidence and that's all they're doing. Team B is acting on it. But what Team B is not doing is acting on it immediately, what they're really doing is preparing a shutout plan. Team A is learning the hacker's method of operations. Team B is preparing the scripts and processes necessary to cleanup afterwards.
One of the biggest mistakes that you can make is to catch somebody and lock him out of your system immediately. "Oops, I don't know how he got in." If you have an intruder on your network, you may have to tolerate it for some period of time until you can figure out the extent of the penetration.
Things to look for and how to look
Examine your log files. If your log files are all size zero, you've learned something interesting. (If you're doing a backup under fire, and you've got spare storage space, do a complete image dump of your disks if you're on an operating system that permits it. Because what most hackers do is just delete the log files. And it's easy to pull the data back off the hard disk by just searching through all the free blocks on the hard disk.)
Look for sniffers. You want to make sure nobody's set up monitoring programs on your network. It's actually kind of hard to find sniffers. What you want to do is look at the process tables of your running machines.
Look for remote control programs. You want to look for things like netbus, Back Orifice, etc.
Look for possible hacker file sharing or communications programs. Look for things like eggdrop, IRC, etc.
Look for privileged programs. The find command on UNIX machines will find any privileged executable. Just walk through the list to see if there's new stuff that doesn't make sense or old stuff that's been changed.
Is there a good way to do it without doing it manually? No. That's one of the problems--it's almost always going to be a manual process. If I developed a tool that did these things automatically, then the bad guy would say, "Oh well I'm going to hide it in the one place he's not looking."
Look for file system tampering. Use tripwire or backups.
I recommend that when sys admins install a box, they build a copy of tripwire. What you have is a database with a set of cryptographic check sums that represent the way that the vendor ships it. Next, complete the installation, make all your local changes, configurations, and patches, then add your favorite stuff and run tripwire again. Then you'll also have a database of how you like your system to look before your users get their hands on it.
If something goes wrong, or if your sys admin leaves, you can now take those two databases and compare them and you can find out what changes were made in the process of setting the system up.
That's good, just from the file systems management standpoint, because you know what patches you need to reapply and what system files you need to update.
But you've also got a tripwire database for what a stock version of let's say Solaris 2.6 looks like. If a friend's site been hacked, you can whip out your tripwire database and compare a basic version from Sun against what they've got. It'll make you look like a hero.
Someone's sitting there saying, "Oh my god, I been hacked and I don't know what files have been changed." You can pull the CD ROM out of your briefcase, stick it in the machine, and provide a complete list of every file on the system that's been touched, by comparing them to what the looked like when they came from Sun.
Examine cron and at jobs.
Look for unauthorized services.
Look for password file changes or new users.
Check system and network configurations. Pay close attention to filtering rules. If you've got a central server that all your filtering routers are pulling from, run tripwire on that configuration directory. You'll know when your network managers change your security policy. You'll also know if the hackers have changed your security policy.
Filtering rules are really important to look at.
Look for unusual files.
Look at all your hosts, especially servers.
Consider having a separate box, sitting someplace on your network, to pull copies of all these system configuration files over to. You can compare them against yesterday's copies. If something is changed, it could tell you something. It doesn't cost you anything, it takes no time, it could run at 3 o'clock in the morning, and give you a list of all the new users that were added, all the changes to all your router configs, etc. Just take whatever state you think is interesting, compare it against last night's state and have the differences brought to your attention.
Backtracking
If you're going to try to backtrack an attacker, it's going to be really difficult. Hackers are increasingly sophisticated about hiding their tracks. You probably won't be able to catch the ones that are good. And the ones you can catch aren't worth catching.
There simply aren't any good tools out there for backtracking. I can't even conceive of how you would build a tool for backtracking. If the hacker's coming at you through three different cutouts, the only way to backtrack would be to have those same tools running at each particular point that the hacker can come through.
It's very difficult to backtrack a phone call.
In the U.S., it is very difficult due to deregulation, cluelessness and apathy. In other countries where the telcos are a regulated monopoly or a branch of the state, the backtracking capability is usually owned and operated by the local equivalent of the FBI or CIA. They don't want to reveal their abilities to backtrack phone calls, so they'll simply tell you it's impossible.
I've been in circumstances where we've had a hard fix on which dial-in line the hacker came in from the second that he came in and the phone company guys said, "Well, we're starting our trace, we're starting our trace."
The hacker would log out 20 minutes later.
"Did you get him?" "No, we just missed him." "Okay, can you tell me why you can bill on that 800 number in one second increments, but can't backtrack it in 30 minutes?"
Hidden directories
If you're running a public FTP server and not tripwiring it (i.e., not getting a list of all files as they're added or deleted), you are criminally negligent. Period.
That's not a legal opinion, but it's my opinion.
You've got to watch out for directories called "...," directories called " ," files with spaces in the names, files with escape codes in the names, etc.
Be careful of these kinds of things, especially if you've got a write-able FTP area for upload. People will upload nasty stuff and you'll be held liable.
A friend of mine lost his job because somebody uploaded child pornography to his Web site. How do you explain that it's not yours? If you don't have the logs, you can't prove it. Even if you do have the logs, somebody's going to say, "How could you let something so stupid happen?"
Under attack?
Decide what you want to do:
Figure out how much you're willing to expend in terms of blood, sweat and tears. I recommend you just watch what they're doing, do the team A/team B thing for awhile. Once you know what they're doing, shut them out,
improve your security and get on with your life.
But if you're going to try to prosecute them, be prepared to spend anywhere between a quarter of a million dollars to half a million dollars in legal wrangling and evidence gathering. It's just a nightmare.
If you are being hacked, depending on where you are, and what your legal counsel says, rather than bring law enforcement in for a criminal prosecution, consider filing a civil suit for damages in excess of a million dollars. Just take the kid's parents' house. That'll fix 'em. I did that to one hacker. It really nailed 'em.
What happens if you get a hacker convicted and put him in jail? They're going to be out again in three months. They're going to get a job as a senior consultant for some firm doing audits. If you take their parent's house, they may never hack again.
If you're under attack, do a system backup immediately. If all you do is unplug the network wire, you've got a problem. You've got to worry about system disks getting zapped. There's a nasty trick going around. Some hackers put in a dead man switch. Here's how it works...while they're inside your machine, they've got a process pinging out to the network to try to reach a specified address. If that process can't reach the specified address, it assumes that the network wire has been unplugged. It may be that something is going wrong so it scrams the logs and blows the disk away. If you want to be super-duper paranoid, login, sink the disks, and turn the machine off immediately. Bring it up in single user mode, do an image dump of the disk to tape, verify that the backup is correct, then bring the system up in multi-user mode again.
The single thing most evil thing I've ever heard about a hacker doing happened at friend of mine's site.
Somebody broke in, apparently didn't do anything, and went away. (This was in the pre-tripwire days.)
My friend figured nothing was wrong. Six months later, somebody came back in, and reformatted my friend's hard disk. He went to restore from backup and discovered that in the first attack, somebody had replaced the backup utility with one that wrote all zeroes.
That's one of the reasons like tripwire is really useful. If somebody's on your machine for even a
couple of minutes, you've got to have that tripwire database to go back to and see what they did.
Other things to do
If you're watching an intruder, use something like Cloak to make yourself invisible. If they notice that you're watching, they'll probably fry your machine.
If they disabled shell command history, re-enable it.
Frequently, hackers will disable shell command history. On UNIX boxes, it saves the last 100 or 200 or 500 commands that you type. Turn that up to the maximum and you'll get a list of everything that they've done.
Building booby-trapped telnet/rlogin clients lets you monitor everything the attacker does. If you can tell what clients they're using, build a version of the client that records everything that they do. That's the best way of implementing a directed tap on a hacker.
I was watching a guy do this way back in 1995. He telnetted form one machine to another machine to a third machine, then logged into a BBS, created himself a user ID and gave his full name, home address, and telephone number. That's the example of the kind of fish that aren't worth catching. He found himself on the receiving side of a civil suit from a Fortune 500 company.
Social engineer the attacker. A buddy of mine was a security dude at a big corporation, and somebody had hacked into their site. My friend got on IRC and started bragging about how he was hacking that particular site. The other guy who actually was hacking the site, said "Yeah, me too."
My friend tried to social engineer him into getting together and comparing notes, but the guy wised up. He did, however, find out how the guy was getting in.
If hackers are leaving warez in your FTP area, log who retrieves them, then contact the administrators at the sites downloading the software. Just say, "We've got somebody who's engaging in software piracy from your machine, please search your hard disk for a file that is exactly 30 megabytes." You can roll up warez guys really easy.
Be careful about using hacker techniques against intruders. If it winds up in court, you'll probably be in a lot of trouble. "My innocent fourteen-year-old client was brutally set upon by this nasty security consultant who used all these ferocious techniques against him." That's a problem.
Be very careful about the laws for gathering evidence. They are confusing. Logs may or may not be admissible. Perpetrators may or may not be prosecutable.
Marcus Ranum of Network Flight Recorder (www.nfr.net) can be reached via e-mail at mjr@nfr.net.