Managing Network Security

Testing Your Security by Breaking In - NOT

by Fred Cohen

Series Introduction

Networks dominate today's computing landscape and commercial technical protection is lagging behind attack technology. As a result, protection program success depends more on prudent management decisions than on the selection of technical safeguards. Managing Network Security takes a management view of protection and seeks to reconcile the need for security with the limitations of technology.


The theory is that I should be able to test the effectiveness of my protection by learning how to break into my systems and seeing if I can do it. It's a great theory, and it seems sensible enough on the surface. After all, if I don't do some sort of testing of my defenses, how do I know they work at all?

Now I agree completely with the last statement. If you don't do some sort of testing, you will not be able to tell whether your systems do what they are supposed to do. The question is whether trying to break into your systems is the best way to do that testing. And here lies the rub.

In today's article, I will be testing the notion that you can effectively test your defenses by trying to break into them. I am in a more formal mood than usual, owing at least in part to the fact that I have been working for the last 48 hours or so (over a weekend of course) reconfiguring a network to get some of the defenses really right - and testing these configurations...

Let's try to break in - just to be sure.

Trying to break in never makes you sure that someone else won't be able to break in. The reasoning seems to go: "I am as good as they are, so if I can't do it, they can't do it." This could not be further from the truth - for ANYONE.

It represents a great leap of ego to believe that there is any individual or small group of individuals who are capable of figuring out and trying every attack that some other individual or groups of individuals will come up with. And there is at least one very good reason to believe that such an approach will never succeed. The reason is that the number of attacks that can be tried are far larger than the number you will be able to try. So the best you can do is cover some subset of the possible attacks with your practice attacks.

How big a portion of the attacks can you cover? It turns out that there are a potentially infinite number of attacks. So if you try 1 or 100 or 100000 or 10^100 attacks, you still haven't made a dent in all of the possible attacks that can be tried against your system. And that is at the heart of the problem. You can try till you turn blue in the face and all you will show is that you didn't succeed yet. You won't show that an attacker won't succeed on the very next try. For those of you who have studied philosophy of science, you may recognize Karl Popper's work on confirmations and refutations in this. Essentially, Popper proved that you cannot prove a negative about an infinite set by pure experiment because (to misquote) the proposition that all pigeons are white cannot be proven correct until you look at all pigeons, but even a single black pigeon will prove the proposition wrong.

If we find anything, we can fix it.

True enough, but if you find attacks, this also means that you need to fix the process you used to devise defenses, because it is not just a case of missing something, it is a case of a failed protection process - unless...

The exception to this is an attack that you believed would work and decided not to defend against. Now I have vulnerabilities in my systems that I choose not to defend against for one reason or another, and so do you, and so does everyone everywhere who runs a system of any sort. It is the nature of things. But...

If I knew of the attack and decided not to defend against it, what good is it to do a test? I already know that the system is vulnerable, and the test will only show that I was right or show that I am not as good at attacking systems as I need to be to exploit the vulnerabilities that I know exist. So it is a waste of time to even try.

Let's hire someone who is skilled at it

Fair enough. The theory is that a skilled attacker will help identify more attacks that might work against us. It is a good theory, and it works in practice. When you invite people to attack you for money, they do it. Of course you are more or less always inviting people to attack you for money if you have any significant business at stake. And they succeed at times against most operations.

But the problem is that most 'skilled attackers' are not 'skilled defenders' and they don't understand the tradeoffs involved in your operation. If they succeed, all they have done is focus you in on one attack mechanism. You will be forced to spend money on it at the expense of something else, and it may be that nobody else would have come across this vulnerability and it was thus a double waste of money... One waste to hire the attacker and another waste to defend against the attack that they used and others might not use.

It would seem far better to hire someone who is skilled at defending systems against attack than to hire someone who knows how to attack systems. Now I admit that defending systems well involves understanding how to attack them, but that's not to say that knowing how to break into systems involves knowing how to defend them from breakins. For this I look to the physical world...

The Physical Analogy

If you wanted to protect your house from breaking and entering, would you hire a burglar to tell you how? This is an interesting question, and one that has been answered with both yes and no by different folks at different times. The better question in my view is whether you should hire a professional security firm or a professional burglary firm to help you understand your defense options.

The burglar will tell you that it's easy to throw a brick through the window, to pick your door lock, to enter through a second story window, and that they like operating in the dark. They may also 'case the joint' along the way, but that's another issue - the issue of trust. They may tell you that the thing they most fear is a big dog, but maybe the next burgular has an easy way around big dogs. If you pay them, maybe they will throw a brick through your window or pick a lock or go in through the second story - does that help you? Now that you know these things, do you know how to protect your home? I don't think so. Does this mean you should get a big dog, bullet proof windows, a lock that is very hard to pick, and always keep your upstairs windows locked? You could do that, but let's look at some other options...

The professional security company will tell you all of the things the burglar told you. They probably will not throw a brick through your window. Even if they have a locksmith, they will be offended at the notion of having to pick a lock as proof that a bad person can do it. They may show you how to get to the second floor, by climbing up the trellis on your porch - assuming it doesn't collapse on top of them. But then they tell you about other options. For example, they will tell you about things like response times with different services and different sorts of alarm systems. Most burglars don't know much about these things because they avoid houses with these protections or enter them when the alarms are off. But even for the real professional burglar who specializes in high-end robberies, is it really a help for you to have them plan out a way to break in and do so? Is this more of a risk than it's worth?

Oh yeah... That network I was working on...

As I mentioned earlier, I have spent the last 48 hours reconfiguring a network to improve on some of the defenses. The network was actually doing just fine and was operating relatively securely for a period of several years the way it was. I didn't make any security patches at all. There weren't any insecure services to patch. All I did was some reconfiguration, the addition of another layer of protection, performance enhancements, and some tightening and tuning. I am not yet fully satisfied with it, but I'm getting ever closer. It had been tested by numerous real-world attackers on an ongoing basis, and none had gotten past the first layer of defenses. So why did I go and alter the protections?

The answer is simple. I do a lot of tests on my network. None of them are oriented toward breaking in, per se, they are oriented toward determining whether or not it does exactly what I want it to do. For example, I scan to see if machines I don't want detected are detectable from remote sites. I check to see whether unauthorized services are available. I check to see that the authorized services operate as they are expected to. I check to verify that audit information is as it should be and I review the audits of various efforts on a regular basis. I also check the systems from sites that have less restricted access than any real users, because I want to be certain that if a protection mechanism breaks down there will be a redundant mechanism that takes over and reports the breakdown long before the overall protection scheme is bypassed.

All of these are tests, but none of them are tests that involve breaking in. These tests go well beyond what 'hackers' do or could do against this network, because I don't want to be caught short by only doing what I think someone else might do. I want to cover the full range of possibilities, not just the attack of the day, and not just the things that the limited skills of one person with minimal training and expertise can come up with. The things that I spent a lot of time on this weekend were things that almost certainly won't prevent an attacker from breaking into my sites, but then that's why the many attackers that try to break into my sites fail. And when I look at the tradeoffs, it costs a lot less for me to spend the weekend making sure than it does to spend a month making up for it if someone gets in.


I believe strongly in testing protection, but this is not the same as 'breaking in'. It is a much more thorough process in which we identify the different methods of break-in by large classes and try to test the defenses we have designed to assure that they operate as intended. This is very different than testing by attacking in that it can be designed to provide very good coverage of the defenses we have used and recognizes the limits of those defenses without introducing unnecessary risk or cost.

Having said this, you might go to my web site and look at some of the "Strategic Services" we provide under "Security Assessments". They include:

All of these might be called examples of "Testing Your Security by Breaking In". So where do I get off telling you that this is a foolish thing to do when I sell it myself?

The difference between testing and breaking in is not always so clear. In the case of good tests, they don't really break in at all. Customers ask for these services for two reasons: (1) to test defenses that are in place to verify their effectiveness, and (2) to demonstrate the need for defenses that are not in place in areas where vulnerabilities are known or suspected.

It may seem like a distinction without a difference, but the difference is very real. Like the alarm company that identifies weaknesses and solutions, we identify weaknesses and solutions in your information protection program. Unlike the burglar hired to show they can break in, testers are not burglars and do not break in. The difference really comes in two ways:

I believe that these controls are necessary and appropriate to doing a reasonable test of protection. If the test doesn't cover a wide class of issues, it is not a very useful test. If the proper controls are not in place to make it safe and effective, it is not a very safe test. The goal is safe and effective testing, and that is a far cry from "testing a system by breaking into it".

About The Author:

Fred Cohen is exploring the minimum raise as a Principal Member of Technical Staff at Sandia National Laboratories, helping clients meet their information protection needs as the Managing Director of Fred Cohen and Associates in Livermore California, and educating defenders over-the-Internet on all aspects of information protection as a practitioner in residence in the University of New Haven's Forensic Sciences Program. He can be reached by sending email to fred at or visiting