Enterprise Linux Magazine published these three columns about a DNS attack against my original DNS servers in late 2000, but I’m not able to find them on the Wayback Machine archive at archive.org. They chronicle what I learned while fighting the attack, and the underwhelming response from law enforcement. I’m reproducing my copy of them here because this incident launched my IT Security career and helped inspire at least two novels. And even though the story is old, the lessons are timeless.
- Monitoring my Name Servers (Jan., 2001)
- International Terrorism in Minnesota (Feb., 2001)
- Recovery From International Terrorism (March, 2001)
Monitoring my Name Servers (Jan., 2001)
I’ve had a nagging problem for several weeks. For reasons I still don’t understand, my little DNS Server sometimes decides to stop translating names. It’s hassle when that happens and if I were running a bigger operation, I’d put some serious effort into troubleshooting it.
The symptoms of this problem are really annoying. I get lots of email and when my DNS server stops translating names, email gradually stops coming in. Over time, as the cached copies of my domain name expire on servers around the Internet, email starts to bounce back to the original senders after several hours or days of retries. Sometimes, people send me emails to tell me about the problem and, of course, I never see those until after I temporarily fix the problem.
One time, my sister called and asked what was wrong with my email and if I’m supposed to be such a computer hotshot, why couldn’t I keep my email server online? I was about to tell her she must have typed my address wrong because I was getting email from other people at the time, but I held my tongue, tried some ping tests, and found out she was right. After I restarted named, two new emails came in within seconds. After a while, a few more emails came in from people telling me I had email problems.
My DNS server is running Red Hat Linux V6.1, using the DNS bundled with this version of Red Hat. Near as I can tell, its official name is named 8.2.2-P5.
I also noticed a couple other curious occurrences. First, right about the time my DNS server first went offline, I saw this message on the console:
220.127.116.11 sent an invalid ICMP error to a broadcast
Hmmm – I wonder what that means? I also saw variations of this message:
named Cleaned cache of 1 RRsets
All this made me wonder, is a bad guy someplace trying to mess with my DNS Server? Almost immediately after that, an alarm went off in my head because it occurred to me, how would I even know? After all, this computer just sits there and translates names. How do I find out what’s going on inside the dad-blamed thing? If somebody from across the Internet tried to launch an attack, how would I find out about it? How do I know what kind of requests come in and how they’re handled? How do I measure how busy this thing is? And what do those messages mean?
Surely, Linux has some means to audit incoming service requests for later analysis, right? After all, we did this 15 years ago with OpenVMS. We can measure all kinds of stuff and display it graphically with Windows NT and 2000. Linux wants to be an enterprise player; surely it has some capabilities in this area, doesn’t it?
So I turned to some friends at Mission Critical Linux (MCLX) for advice. They offer support and deployment services for Linux systems and they have some of the best engineers on the planet working on clustering and other capabilities. Several of my ex-DEC (now part of Compaq) friends work there and if anyone knows how to solve my problem, these guys should.
They offered a few very helpful hints and I’m grateful for their informal mentoring on this problem and future advice on other problems. First, those console messages are not related to the problem. The invalid ICMP message means a router or host someplace sent back a bogus answer to a broadcast request. The RRset message is routine and shows up when named periodically purges entries with expired TTLs from its cache.
I found out that named provides extensive logging capability, most of which I’m still struggling to understand. It’s all driven by a file called /etc/named.conf. The man pages describe a whole series of commands inside this file that drive the named process. Among them is the logging command, complete with a several page description of its syntax.
I also found out that named can log its activity to a file when started with an appropriate debug option. It is also possible to signal a running named process, using the kill command, like this:
kill –USR1 18419
Where 18419 is the process ID of the named process, from the command, ps –ax. When I signaled named in this manner, it started logging to a file named, /var/named/named.run. Here’s some sample output:
datagram from [18.104.22.168].2329, fd 22, len 37
req: nlookup(infrasupportetc.com) id 18783 type=1 class=1
req: found 'infrasupportetc.com' as 'infrasupportetc.com' (cname=0)
ns_req: answer -> [22.214.171.124].2329 fd=22 id=18783 size=93 rc=0
I think this all means that the host at IP Address 126.96.36.199 was curious about my infrasupportetc.com domain. My DNS server evidently sent back an answer that made the requesting host happy. I wish these messages were time and date stamped.
I also learned a new word, grok. Here is an extract from an email conversation involving an (apparently) ignorant skinny bald guy from Minnesota learning a new word and an exceedingly patient MCLX engineer:
MCLX Engineer: . . . Sniffit would also be an alternative,. . . I don’t know of any package that groks this for you.
Greg: Where could I get my hands on the sniffit package you mention and some instructions on how to use it? And what does it mean to grok something? I wasn’t able to find a man entry for it.
MCLX Engineer (most likely laughing hysterically): Grok means to understand/parse. What I meant was that I don’t know of any program that will look at the logs and interpret them for you.
Evidently, I still have a long way to go before I become a real member of the Linux community. I also haven’t yet found an easy to use utility that tells me what’s going on inside my DNS server, but I’m still looking. If anyone has some good ideas, I’d love to hear from you.
International Terrorism in Minnesota (Feb., 2001)
I’ve written extensively in this column about a small Linux DNS server I run. Imagine my surprise a few weeks ago when I found my system launching a denial of service attack against the Government of Brazil. That set a chain of events in motion every bit as traumatic for me as the recent Presidential election was for everyone else.
It all started when I tried to access my email. For some reason, the response time was unbelievably slow. About that time, my wife complained she couldn’t get to the Martha Stewart Web site, or anywhere else on the Internet, and what did I do to the computers this time?
I started investigating and found my house LAN was indeed running very slow. I looked at my hubs and found port 4 on one hub going nuts. This was the port leading to my DNS server. The ps –ax command showed me the following process:
ping -s 65000 -f nn.nn.nn.nn (I won’t share the target IP address.)
My DNS server was sending 65,000 byte packets as fast as it possibly could to a system across the Internet. When I killed the process, performance went back to normal.
A feeling of dread came over me and my adrenaline started pumping. Then I got mad as I realized some jerk broke into my DNS server and set up this attack. Fortunately for the Internet, I don’t have enough bandwidth for anyone significant to seriously care about. Unfortunately for me, this jerk found out where I am and how to break in to my network. I felt violated, angry, and afraid all at the same time, especially when I thought about all the data I have squirreled away in various directories on computers all over my network. I wanted to find this jerk and strangle him or her, but I didn’t have the tools to even know where to begin.
So I called my friends at Mission Critical Linux for help. I explained the situation and we all agreed that somebody had compromised my system. I learned a lot about network break-ins that day. I learned that BIND 8.2.2-P5, the version of DNS bundled with Red Hat Linux 6.1, is aging and now has some known security vulnerabilities, and that Red Hat keeps a list of bug fixes and updates on its web site. I should have periodically checked for these updates.
I learned to shut down services such as sendmail, telnet, and ftp because they serve no useful purpose on this machine. Sendmail uses its own process while the inetd process controls ftp, telnet, and others. These commands ensure they won’t start at boot time:
–level 345 sendmail off
/sbin/chkconfig –level 345 inet off.
That’s when I remembered that telnet had been behaving strangely. When I tried to connect via telnet, it wouldn’t echo anything and lately would just tell me the process was ending.
The support person laughed and told me I’d been suckered by the oldest trick in the book. Somebody probably replaced the real telnet with a fake version designed to steal passwords for later transmission to the bad guys. The system had definitely been compromised.
The technical recommendation: Wipe the hard drive and rebuild the system from scratch. The next recommendation: Call the FBI immediately because the IP address my system attacked belongs to the Brazilian National Government, and I could face legal trouble if I didn’t report it.
As soon as we hung up, I called the Minneapolis FBI office and asked for somebody who deals with computer crime. The receptionist sent me to a lady. The conversation went like this:
Greg: Hi – I need to report a computer crime. Somebody broke into my DNS server and launched a denial of service attack against the government of Brazil.
FBI Lady: Wait a minute. Did you say D-E-S server?
Greg: No, a DNS server.
FBI Lady: Oh – D – N – S, OK. What did they do to your computer?
Greg: Somebody tried to use my computer to attack a computer that evidently belongs to the Brazilian Government.
FBI Lady: OK, . . ., who did it? Do you have their address?
Greg: No. See, a DNS server translates names to addresses on the Internet. One of my computers is a DNS server and somebody out there on the Internet tried to use my computer to attack this other computer in Brazil.
FBI Lady: OK, but we need to know who did it. We need a name or address or some way to find this person.
Greg: Well, I was kind of hoping you guys could help me figure that out.
FBI Lady: There’s not much we can do if we don’t know who broke into your computer. Don’t you have any idea how to find this person?
Greg: I wish. See, the Internet is a whole bunch of computers all around the world and they’re all connected to each other. Somebody on one of those computers found my computer and made it do this attack. Since all these computers are connected to the Internet, we don’t know if the attacker is next door or across the world someplace. But maybe they left some clues inside my computer to help track them down.
FBI Lady: OK, let me get your phone number and somebody will call you back.
Greg: (after giving my phone number) Any idea when I’ll hear from somebody?
FBI Lady: No. They’re all pretty busy, ya know.
I made that call on Tuesday, Nov. 11, 2000 at roughly 1PM central time. I called again at 4:30 PM the same day. As of this writing on December 15, 2000, I still haven’t heard back from the FBI. I don’t mean to complain, but I was hoping the FBI would be sharper than that. I’ll share how I rebuilt my DNS server and a list of helpful books in a future column.
Recovery From International Terrorism (March, 2001)
I recently shared a story about a bad guy who compromised my DNS server, evidently by exploiting some security holes in BIND 8.2.2-P5, the version of DNS bundled with Red Hat Linux V6.1. After troubleshooting some severe LAN performance problems, I found my DNS server sending 65,000 byte PING packets as fast as it could to a system inside the government of Brazil. Some jerk had hijacked my DNS server and used it to launch a denial of service attack against the Government of Brazil.
Based on strong recommendations from my friends at Mission Critical Linux, I called the Minneapolis FBI office to report the problem. The FBI call was less than productive and, two months later, I’m still waiting for a call back.
Since I can’t afford to keep my equipment out of production waiting for the wheels of justice to grind, I preserved all the relevant BIND settings, wiped the hard drive, and rebuilt the system from scratch.
This process is probably easy and routine for people who know what they’re doing. It was a challenge and yet another learning experience for me. My first problem – I didn’t have a clue how to preserve all the DNS domains and hosts I set up over the last several months. I spent a long time figuring out how to set up DNS using Linuxconf. Now, I had to quickly learn what Linuxconf did for me under the hood so I could preserve it.
Another problem – a bad guy now knows who I am and where I live on the Internet. He’s out there someplace, and he knows I’m vulnerable. So this time, my little server, and my whole network, needs to be secure – really secure.
The strategy: Download the latest stable version of Linux and relevant BIND updates, install on a spare PC, and put this interim system to work as the temporary DNS server while I rebuild the original system from scratch.
To make this work, I needed to know specifically what BIND does and how it works. What files does it use and where does it put them? I knew that zone files describe domains in clear text and I knew from random trial and error they live in the directory, /var/named, but these files don’t tell the whole story. For example, DNS provides a feature called a forwarders list, where suitably a configured DNS server queries other servers in the forwarders list to find names it knows nothing about. But the zone files have no provision for forwarders or any other such characteristic. So how does BIND know who are the servers in its forwarders list? For that matter, how does BIND know which zones it should even care about? There had to something else to describe all this.
It was time for an education. Based on suggestions from my friends at Mission Critical Linux and lots of hours browsing through bookstores, I found a few helpful books, including:
- DNS and BIND, by Paul Albitz and Cricket Liu, O’Reilly Publishing. This is a great book that explains BIND in detail. I wish I’d found this book months ago.
- Linux Administration, A Beginner’s Guide, by Steve Shah, Osborne. This book has several helpful tips and pointers to more detailed information.
- Hacking Exposed, by Joel Scambray, Stuart McClure, and George Kurtz, Osborne. This book is 2 inches thick and filled with exploitation strategies bad guys use. The more I skimmed this book, the more paranoid I became.
I learned that O’Reilly has a reputation for excellence. O’Reilly books are generally trusted to contain accurate and concise information, minus all the annoying chatter that fills up pages of many computer books. The O’Reilly DNS book lived up to its reputation. It’s a great cure for insomnia and also a great reference.
I found out much of the action for BIND lives in the file, /etc/bind.conf. Many other Linux subsystems also keep .CONF files in this directory. These files contain all the configuration parameters for various services. bind.conf contains all the configuration information for BIND, including a list of all zones on this server, location for the zone files, logging levels, and a host of other information – including forwarders. Like all the configuration files, I can edit it with any popular text editor. I also learned that BIND is capable of much more than Linuxconf gives me.
That was it. All I had to do was copy /etc/bind.conf and my zone files in /var/named to the same directories on the interim DNS server, turn off the primary DNS server, and fire up BIND on the interim server. To test, I disconnected both my primary and interim DNS servers and, from another system on the Internet, tried to ping mail.infrasupportetc.com. As expected, the name did not resolve. Next, I reconnected the interim DNS server, leaving the old DNS server offline, and successfully translated every name I tried.
It worked! After all the worry, it really was that easy. And the upgraded Linux and patches were absolutely free.
Next, I rebuilt the old BIND server and copied the files back to the appropriate directories. I’m now running the latest available BIND update, 8.2.2-P7, and the system is working flawlessly. For additional security, I turned off Telnet, FTP, and all other Internet services, like this: /sbin/chkconfig -–level 345 inet off. With no access to Telnet or other Internet services, and all other such services turned off, the only Internet access to my BIND server is via DNS requests. This should keep my network reasonably secure.
One feature is missing. OpenVMS and NT/2000 can temporarily disable an account after a specified number of unsuccessful login attempts. This is a great defense against brute force dictionary attacks. If anyone knows of such a capability with Linux, I’d love to hear from you. Finally, thanks to everyone who emailed me about the Linux find command. I’ll discuss what I learned and some other helpful hints in a future column.