Analysis of the Board Attack

So, all the readers of the 100 Hour Board should know by now (assuming you check your email account) that the Board was attacked and compromised recently.

I was waiting for all the disclosure emails to go out before posting. So, now we can discuss what happened.

On Thursday November 13, 2008 I was asked by an editor to make a code change on part of the site. Well, wasn't I surprised that when I tried to login to the server the account password had been changed. So this was when the fun began. I used an alternative access method to get onto the machine with root privileges and regain control of the normal user account. Then, of course, it was time to start digging to find out what "they" had been up to, where they came from, and how they got in.

I quickly discovered from the log files that the attackers were clever enough to gain access to the machine, however apparently incredibly stupid at the same time. So, there's a good chance it was mostly a script-kiddie attack. They remembered to clear the bash_history for the normal user, but (having accessed the root account using "sudo su"; I've revoked sudo privileges now, since I'm the only one that should be executing commands as root, and I have the root password to use that account when needed) forgot to clear root's bash_history from their last session. So I got to see some of their stupid in action, line by line. For example they list out a directory and then try to change directories into a regular file. Or run "cat" on a compressed file (so you get a bunch of garbage sent to the screen). They were, however, properly paranoid, checking for other logins several times during what appeared to be a short session.

After locking down the user accounts to prevent further shell access I went digging through more logs to find out who they were. So, while they were smart enough to erase their bash_history (most of it) they were also dumb enough to leave behind all of their auth_logs (the ones that say when people log in and from where) as well as all the of the apache logs from their attacks. So it was trivial to discover that the attackers were hitting the machine from locations in Ukraine and Russia (of course these were just their last bounces before coming to our machine and they could be coming through any number of points before that).

Based on the Apache logs they found the Board by doing a Google search looking for any ".edu" addresses that contain "id" in the URL anywhere and are running PHP. So, they appeared to be trolling for sites that might be vulnerable to SQL Injection attacks. My first thought was that they exploited a vulnerability in the Board software and executed an SQL Injection; but further analysis of the events suggest that was not the case. The particular part of the Board they attempted to attack was actually immune. Which, of course, is to suggest that other areas were less immune. I say "were" because I've been spending many waking hours of my life the last couple of weeks doing a line-by-line code review patching up security holes. It's been a tedious and tiring process, but overall the Board is much safer now.

So, what were they doing with the machine they had gained access to? Hosting spam. They weren't sending spam from our machine, which is nice, fewer headaches for us to deal with (being blacklisted, etc). But they had been hosting a Cialis/Viagra spam website in a directory and they embedded a hidden link on our site to promote search engine traffic for that page. They also installed several web interface back-doors to the system, at least 5 if I recall correctly. They were not very smart about hiding those, however, since a PHP file in an "Images" directory sticks out pretty obviously. So I cleaned out all of that garbage as well. Initial reviews of the database suggest that no malicious content has been injected there either.

So, all in all we weren't hit too badly. And had they not been arrogant enough to change the account password there's a good chance I wouldn't know about it still.



The fun, from my end, begins again when they find out that they no longer have access to the machine. Keeping a close eye on the machine they discovered the next day that their shell access was no longer valid. Then in the Apache logs they quickly discovered that all of their web-interface back-doors were missing as well. (This actually helps me because all the ones they checked were ones I had removed, which suggests I got them all.) Then they found that their spam site was no longer running, and that their hidden link on our site was gone. Now the crucial step was about to happen, they would check their next point of access to see if we had completely locked down the machine.

Much to my surprise they quickly and easily accessed the database via our phpMyAdmin interface. This was to my surprise because I had also changed the MySQL account passwords to prevent this very thing. I was afraid they had used some exploit in the phpMyAdmin code, but doing some very fast digging and checking I was sad/happy to discover that I was at fault. In my haste to get our new machine running I had apparently forgotten to remove one of the default "nobody" accounts in MySQL. I now have found that Ubuntu MySQL installations come with a script "mysql_secure_installation" which steps you through a script to make MySQL secure by removing anonymous accounts, setting the root password, deleting default databases, etc. I highly recommend it, it's apparently not well known since none of the Ubuntu MySQL Installation Tutorials I've read through have ever mentioned it.

So, now MySQL is locked down tight, and to prevent further issues of any kind access to phpMyAdmin has been restricted to on-campus only. It appears that they either found out about this change on November 20, or not at all yet. That depends on if they have a machine available to launch attacks from inside an insurance company in Canada which tried to access phpMyAdmin.

It's now been a couple days since I was writing this post, so I've lost my train of thought. So, I think I'll just end it here. We've cleaned out the machine and everything seems to be pretty good. There doesn't appear to by any XSS attacks running or malicious content hiding in the database, but if you see anything suspicious please send us an email so we can check into it.