&. |

A software developer’s musings on software development

Something I've learned about spam

Warning: I wrote this blog in 2008. That is a long time ago, especially on the internet. My opinions may have changed since then. Technological progress may have made this information completely obsolete. Proceed with caution.

It’s been a while since I implemented a spammer’s honeypot on this blog. It has been extremely effective, so much so that I disabled the captcha. All I do is put a hidden form before any blog posts are displayed. Humans never see it, but spambots all see it, and apparently they are configured to submit spam to the first form on the page. In fact the only spam that has gotten through in the last year has been spam that submitted to all forms on the page, not just the first one. (I think this just happened once though.)

Fast-forward to a few days ago, I noticed that the excerpt of a page that Google shows displays the hidden comment submission form. This doesn’t particularly matter, but I’d prefer it not be there. So I added a check on useragent, and if it appears to be a search engine bot the honeypot is not displayed. Well apparently spammers use a two-step process. First they scan for blogs with forms while pretending to be googlebot. Then they submit to those forms pretending to be a normal user’s browser (usually IE 5.5).

I know this because I got about fifty spam comments in the last two days. If they were scanning the page with user agent reported as IE 5.5, they would have still seen the honeypot. But the comments were submitted with user agent of IE 5.5. Anyway, I’ve gone back to printing the honeypot for everyone, but only for the homepage. Any permalink pages will not have the honeypot. I’m pretty sure spammers don’t bother to go to the permalink pages, and search bots should only be indexing the permalinks. Hopefully, both problems are solved. If not, I’ll have to go back to a more fragile solution (something requiring Javascript, something requiring cookies, or even reinstating captchas).

Or maybe the spammers were just trying to wish me a happy twenty-seventh birthday by flooding my site with links to porn.