Search Banter

SE Roundtable had a post today that said and I quote “bots are not obeying” the robots.txt files. That’s a little scary to me and I don’t even have a robots.txt file. But there are sites out there that have pages that should not be indexed by search engines under any circumstances. I’m sure Yahoo will be able to put these disobedient bots in time out for a while and get them to behave soon. I wouldn’t worry about that.

Trackback URI | Comments RSS