After a week of delays and two whole days wasted on tech support calls, I was finally online. I was ready to write a damning invective criticizing Apple and an encomium praising the Computer Shoppe. I pointed my browser to my blog, when what to my wondering eyes should appear but a notice:
Your account has been suspended. Contact billing/support.
What? How could this be? Did we forget to pay the bill? I called my webhost immediately and learned that our account had been suspended because of a “massive amount” of activity on b.rox.com.
It appears that b.rox was under attack by spambots.
Without getting excessively technical, I’ll try to explain this in terms my grandma Mildred (may she rest in peace) could have understood, insofar as I even understand it myself.
A spambot is not a real person. It’s a program that surfs the web and does bad things. Imagine a robot sitting at a cybercafe sending junk e-mails hawking pharmaceuticals and casinos and you’re not far off the mark. Spambots have been hitting my blog for a long time, trying to insert their advertisements as comments to my posts. I have installed different filters that are pretty effective at keeping these advertisements, this “spam,” from appearing on my site.
However, the problem now was that the sheer volume of spambot attempts was overwhelming the server. I had a chance to examine the logs later, and there were over 3,000 attempts to access a single file on the system in just one hour.
Understand that we pay about $12 a month for space on a server which is shared with hundreds of other users and their sites. The activity on my site was bringing the whole server down, rendering all these others sites inaccessible. I’m not sure, but I suspect that my webhosting company employees an automatic method to suspend accounts that experience such overwhelming activity. Can’t blame ’em for that.
Suspending our account saved the other folks sharing our machine, but it disabled all our sites. We’ve got a couple dozen websites on our account, including rox.com, blogs such as mf.rox, and various little sites like my tribute to Grandma Mildred and Life Following the Dead. The Mid-City Neighborhood Organization is hosted on our account. All these sites were taken offline when our account was suspended, even though the problem was only related to b.rox.
Fortunately, upon my request, the system administrator quickly restored all the other sites. We agreed to keep b.rox offline until the problem could be resolved.
Now here’s where things get a little, um, complicated. The sysadmin said he would give me access to the files “and you can fix your script.” The “script” in question is WordPress, a software package which runs this blog. WordPress is pretty damn good, and it has become pretty popular. (My webhost even offers it was an “autoinstall.” Unfortunately, though they provide it as a convenience, they don’t really offer any support for it.) That popularity has made it a target for spambots. The sysadmin clearly indicated that the onus was on me to correct the problem. Per his recommendation, I consulted with the WordPress community using the WordPress Support Forum. You can read the discussion. One person suggested a plug-in called Bad Behavior. I checked it out, thought it might do the trick, and installed it. I also upgraded to the latest version of WordPress. Then I notified my webhost and asked the to reactivate b.rox.
For a few hours on Sunday, b.rox was back online. But it was again flooded with thousands of connections from spambots, and our account was again suspended, meaning rox.com and tile.rox.com and mcno.org and boozocracy.org and all the rest were down again. This time the system administrators were a little less forgiving. They agreed to bring all the other sites back, but not b.rox. They warned me if the problem occurred again our account would be permanently suspended until we upgraded from a shared account to a VPS or virtual private server. What’s the difference, you mgiht ask? About $38 a month.
I was skeptical about taking this step. In fact, the customer service aspect of this whole debacle left me feeling that it was time to move to another webhost. Moving is a pain, I didn’t want to move, but felt we had no choice. Granted, this wouldn’t resolve the technical problem of the massive spambot attack, but I’d have to deal with that later.
(In retrospect, it seems that the spambot problem was not a WordPress issue per se. The spambots were targeting my site because I used WordPress, but the filters I’d installed prevented them from polluting my site with their advertisements. The problem was the sheer number of attempts to access a certain file, not anything in the file itself. I would gladly have disabled comments on this blog if it would have stopped the problem, but it isn’t that simple. Indeed, the tech support folks verified that even if we moved the file, changed its name, or simply deleted it, the server would still be overloaded by the requests. The result is similar to a distributed Denial of Service attack, or so it seems to me. I’m not a security guru.)
Here our saga takes a turn for the better. Seeking advice on a new webhost, I posted a query at the WebHostingTalk forums. You can read the thread if you’re curious. The discussion was enlightening, but the important thing is that it caught the notice of Dan Ushman. He is one of the founders of our webhosting company. And, as it turns out, he is a frequent reader of this blog.
Dan contacted me and offered to comp us a VPS for one year at the price we’re paying now. Sweet. And so, boom, here we are on our new VPS. The migration didn’t take long. Does the site feel any faster?
In all, b.rox was offline for about four days. During that time I realized how important writing here has become to me. I’m quite attached to this blog. It was a lifeline to the rest of my world during Katrina. It’s helped me through some difficult times. Several people contacted me to ask what was wrong when they couldn’t access it, which means that people actually do read this and care about it, even if the number is small. That makes me happy, no matter what George Will thinks.
Dan, if you’re reading this, thank you. Very much. I still have concerns regarding the support and service issues which I think are clearly outlined here, but your involvement has gone a long way to keep me as a customer.
As spam attacks continue to rise, this will continue to be a problem. I hope I’m not being presumptuous in offering the following three recommendations.
- A problem with one subdomain shouldn’t cause the suspension of the whole account. If, as I suspect, this is automated, there might be a technical fix for this, which I hope would result in disabling the problem site rather than the whole account.
- When such a problem arises, the customer should be notified. I never was, not even the second time when a sysadmin promised I’d be contacted if any issues came up.
- Then there is the spambot attack itself. I don’t know what the fix is for thiss, but I know the problem wasn’t my fault. In a situation like this, the customer is an innocent victim of malicious outsiders. As I noted earlier, it’s like a DoS attack. The customer shouldn’t be further punished by being forced to upgrade. Instead, the support staff should work with the customer to find a solution. And, let’s face it, many customers using an easy package like WordPress won’t have the wherewithal to find their own fix. They’ll need plenty of help.
Meanwhile, the spambot attack continues, with over 17,000 hits today so far. I’m trying another fix. We’ll see how this goes.
Wow, this whole thing has been exhausting. Even writing about it has been exhausting. I need another vacation. Just in time for Christmas!
Discover more from b.rox
Subscribe to get the latest posts sent to your email.
As the person who actually pays the bills for our account, I’d like to offer another suggestion, since you’ve opened that dialogue: Get a better billing system. One that tells me beforehand that an invoice is due, so that I know they’re about to charge my credit card (as they did earlier this week), would be a good first step. Also, in the past, we’ve had problems with changing credit card info in their billing system. I haven’t tried to change my card in their system lately, so I don’t know if they’ve fixed this problem.
The good part, of course, is that our host is a small company, and I’ve usually had little trouble getting someone in person on the line, and usually it’s someone who gives a shit about my questions and concerns. So that’s good.
The Spambot thing sucks.
There really is nothing much you can do to stop them trying. The ISP or host need to be agressive about blocking the originating IP addresses and reporting traffic to the guys on the other end. It won’t do much good but at least they can try.
I wonder why they picked you.
I’m not sure blocking IP addresses would be worth the trouble. The spammers seem to have an endless supply. They use a few for a few hours and then rotate on.
The approach we use at one client I’ve done work for in the past is a variety of scripts that monitor connections from various IP addresses in various ways, and a dated IP blacklist database. Basically, every time you hit our server with something that looks bad, you get logged and a certain x number of points get added to your account. Once your points go above y, you’re banned until they go back below y. One point per hour is removed. Each ban adds a multiplier to your IP address so that you stay banned longer each time. We’ve got a few IP addresses that have a multiplier of 20, and can be banned for weeks at a time.
One of the tests is similarity of requests. A particular wordpress blog was getting identical spams at about the same rate that yours was. Since it was just a constant stream of POST requests to the same script with the same IP, they got banned pretty quickly.
The bad parts: It’s a conglomeration of scripts and pieces of software like PSAD and fw-snort, it’s a pain in the butt to admin, it can piss off some people who are hitting refresh over and over again for whatever reason, it took a long time to get perfect, and it’s nowhere near releasable to the general public. It will let *some* traffic in, unless you score the particular violation high enough … i.e. anyone that portscans us gets smacked pretty hard, but comment spam takes quite a few times to get a good long-term ban going on.
The good parts: It doesn’t block people who are on dynamic-IP connections unless they have a spambot running on their own system. (Well, for very long at least. We’ve had to whitelist AOL’s dialup IP range, for instance…) It’s pretty much self-sustaining now that it’s stable. It’s customizeable if we need to start blocking different categories of violations. It does the job perfectly.
Basically, what you’ve found out is that your web host isn’t running any sort of intrusion detection or anti-DDoS packages at the firewall level, because that kind of package would’ve detected this kind of traffic.
I’m switching to your host next year. iPowerWeb is starting to get under my nerves. Will talk to you about the specs later. Glad to see you back!!
Welcome back B…
Your blog has kept me plugged into Mid-City more than I can tell you…even though I drive to “Charity” a few days a week, it’s therapeutic to see the rebuilding from you and Xy’s POV…
Happy Winter Solstice.
Glad you’re back.
I love your blog and was worried about you. I blog at a tiny blog site (click the link if you like). It’s run by a fellow in Canada who is a trade unionist and wanted to encourage other labor people to blog. The server is located in his back bedroom. Our site has been shut down for up to a month at a time by similar attacks. He’s been throwing money at the problem, paying for more computing power, more bandwidth, fancier software. I’m very loyal to him, so I keep trying to post there, but I’ve been encouraging him to take up model railroading instead. I notice that Mike Whybark’s blog has been down since the big storms in the North West. I wondered if this wasn’t some kind of selective attack against former Bloomingtonians.
Your ISP(webhosting provider) should have been able to help you as well. You shouldn’t be punished for being attacked by a third party. They could have examined the source of the attacks and blocked them further upstream from you. You’re right in that you blocking the IP addresses will do no good but your webhost or their ISP may have been able to provide more help. It’s good that they did eventually take care of you, though.
I run wp on my own box as well and don’t see that kind of attack load at all. My guess is, they detected you had a vulnerable version at some point.
If you have some techie, geeky, ISP kinda questions, drop me a line. I make my living that way and have worked for ISPs in the past including cable ones (not Cox).
I use WordPress and I get tons of comment spam. It’s quarantined, but still drives me nuts. I’m a total non-geek. Is it possible to blacklist an entire country? Or continent? Is it possible to actually keep certain IPs from even downloading your blog?
I can recommend http://www.hostway.com for hosting, I’ve used them for many years. While I do not have experience with this volume of attacks they have always fixed issues extremely promptly.
Damn fine to see you back, you and Xy need to stop by for a drink over the holidays…
It’s good the big shot decided to comp you the upgrade, good for us! I think I can speak on behalf of all of the other .rox’ers to state that we all have received bot attacks on some degree, but not anything close to what you have gotten. I think that is because b.rox is probably more visited than the ROX site itself.
I know my hosting company (infosaic.com) has some interesting spam bot blocking technology, but that’s about the extent of my knowledge on that subject.
I’m glad your online at home now, and rox is back up!
I’ve ignored my blogroll for a tiem because of the holidays, final exams, etc. the oen time I did try to check ad failed, I just assumed that you were posting. making an upgrade or some such. I’m just delighted that there’s a happy ending. Faugh on spambots.
[…] on me with no warning, without even a notification. The last time was in December of 2006 during another communication meltdown. It should be noted that the “amount of email” which my webhost cited as a concern […]