jclark.org/weblog

Goodbye, Yahoo

For someone who has been in the business as long as Yahoo, they sure have a stupid bot. The Yahoo (neé Inktomi) Slurp bot often requests completely strange URIs constructed from various things on my pages. Take this access log entry, for example (split onto multiple lines for clarity):

68.142.249.168 - - [27/Jan/2005:19:29:58 -0800] 
  "GET /weblog/Miscellany/About/xxx@xxxxxxxx/CoolStuff/Potpourri/Spanish/
        Hardware/Toys/Miscellany/WebDev/Browsers/Programming/Rants/Apple/
        Music/XML/XSLT/XML/XSLT/Spanish/WebDev/Blogging/Hardware/Apple/
        Programming/Microsoftish/ HTTP/1.0"
  200 5798 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; 
        http://help.yahoo.com/help/us/ysearch/slurp)"  

Yes, the URI contained an email address in that rediculous chain of categories (which wasn’t mine, so I elided it). In the bot’s defense, Blosxom will take the URI in stride, and return a page with no entries. There are plugins to modify this, which I may look into. Also, my comment system has a bug that can result in bad links when a bad URI (or no URI) is supplied. However, I don’t think that URI is naturally occuring on my blog. Even if it is, I don’t see any other bots asking for things like this.

Speaking of other bots, so far this month Yahoo Slurp has sucked down over 4 times the bandwidth that Googlebot has used. During this period, Google has delivered over 2600 hits, while Yahoo has delivered 107 (for those keeping score at home, MSN has delivered 23 hits).

So, 400% of the overhead, 4% of the return. Fortunately, I’ve found a way to tweak these figures, by adding a new entry to my robots.txt:

    User-agent: Slurp
    Disallow: /

Toodle-oo, Yahoo.

No Follow

Unless you’ve been living under a rock, you’ve probably seen the big Google Announcement, entitled “Preventing comment spam”. By adding a rel='nofollow'attribute to a link, you instruct Google (and Yahoo, and MSN Search) not to consider the link in things like PageRank calculations. Blog software producers have jumped on the bandwagon, promising to use this attribute in all links within comments, referer lists, etc.- anywhere a website visitor can create a link. The idea is that this removes the incentive for comment spam: nofollow = No PageRank.

Will it work? Ben Hammersley provides a lucid explanation of the economics of spam (it’s free!) and concludes that nofollow may increase the amount of spam. I tend to agree that spammers will pay it little mind; I think the real value (and Google’s real purpose) is that it may help improve search engine results. Remember when finding things via Google was easy, and you never saw links to commerce sites?

Robert Scoble notes this has another use – it allows a blogger to link directly to someone without giving Google juice. How often have you seen someone complain in a blog post about a spammer, without linking directly, for just this reason? Phil Ringnalda points out this can be used to selectively control the PageRank you bestow. Phil has added special style rules to his user style sheet (userContent.css in Firefox) to display all nofollow links in flashing lime green, which ensures he knows when a page he is viewing is fiddling with PageRank. I liked the idea so much I copied his style rule into my user stylesheet, so far it’s been enlightening to see who is using it, and for what links.

For now, I’m taking no action on this site – I already treat any comment containing a raw html tag as spam.

Fighting Referer Spam with deferer

I’ve been getting swamped by referer spam lately. Most of it is for domains that appear to have had hosting suspended. I had already seen a correlation between this referer spam and comment spam; what I didn’t know (but should have guessed) is that lots of folks are seing this. Tim Bray wrote about the problem today, pointing to more info from John Sinteur and Ann Elisabeth. Apparently all of these referer URIs resolve to a single webhost, with an IP of 161.58.59.8.

Ann’s post is one of many on her blog about the subject, she is actively pursuing this and trying to get Verio to pull the miscreant’s hosting. I suggest reading everything on her homepage for lots of good info.

John’s post on the WordPress support blog includes PHP code that sends a 301 Moved Permanently redirection header to any request with a referer URI that resolves to the IP above. Where does he redirect them? Back to the referer URI, of course.

Now, that’s an idea I like. I liked it so much I wrote a perl version as a Blosxom plugin. It’s called deferer, and is available here. Right now, it’s a one-trick pony, but (when time permits) I intend to expand it to a more comprehensive referer spam solution. I don’t know how effective redirecting these requests is- the spider scripts sending the requests may not follow redirects. Even so, deferer reduces server load (it ends the blosxom invocation early) and saves bandwidth.

Update: Deferer has been updated to version 0+2i, to fix a bug that caused a 500 Server Error if the referer hostname could not be resolved.

Site Linking Policy

All of the content on jclark.org is Creative Commons licenced – you may use it as you like as long as you give attribution and share your changes. This has been my policy almost since the beginning of the site.

Linking to the site is also encouraged; I make every effort to ensure my permalinks always work. However, linking directly to images is prohibited. If you’d like to use one of my images on your site, please make a copy and host it yourself. Someone has begun linking directly to one of my badges, which I consider a gross abuse of my bandwidth.

I have searched all over the offending website, and cannot find an email address for the site owner. Since I’ve never posted a linking policy, I would like to contact him and allow him to change his site before I take steps to prevent this. Therefore I’m posting this now, to allow him time to correct the problem. By this weekend I will be actively blocking this and possibly serving alternate content instead.

Subtraction

Jeremy at Alpha-Geek.com pointed today to Subtraction.com as a website design that stands out from the crowd (in a good way). He writes:

For some reason, I am really taken aback by the site. It’s so simplistic in design but so rich in all of the nuances.

Boy, do I agree. As I’ve noted before, I love clean, simple layouts (and do not hold up my own as a shining example). Subtraction is almost breath-takingly clean and stark, without feeling bare. There are so many nice things about this design.

I’ve been wanting to do a redesign for some time. As I first looked at Subtraction, I had a number of “I wish I’d thought of that” moments. With a design as distinctive as this, any thoughts of borrowing design elements are quickly suppressed by respect for the original. Take the links in the ‘Colophon’ section of the right column. They are text links with little pictures, so similar and yet so different to the ‘badges’ seen here and on so many other websites.

A couple of ideas which I saw were not completely new to me. I’ve actually been kicking around the idea of a black and white layout for a little while. Of course, if I do it now, it’ll feel like copying. The little image blocks at the beginning of many of the stories is another idea I’ve kicked around, but this implementation is much better than anything I’ve considered. The categorization scheme is also something I’ve been thinking alot about – Subtraction uses tags, like del.icio.us. I’ve already made plans to switch this site to a tag based system as soon as I figure out how I want to handle the permalinks.

Although the tag-category idea isn’t surprising, the four-column layout for displaying all of the categories is. As is the linkblog at the bottom of the page, which is more than just a link blog. Each item has a star-rating, and many of the comments are more extensive than the average linkblog. The layout is (like the rest of the design) understated and elegant.

Other details are subtle, but accumulate to great affect. The use of simple fonts, the font sizing, and the use of several shades of gray in addition to black and white form a whole greater than the sum of the parts. The use of horizontal lines also strikes me as stunning in a way I can’t quite put my finger on.

My only dislike is the use of done-to-death bright orange as the highlight color- in such a breakout design, it’s disappointing to see such a cliche. But this is a minor (and quickly forgettable) flaw, and every true masterpiece must have a single flaw.