Ever done something stupid? Well, read this post on Google penalties, and then come back and finish reading here. Oh, and read the 3rd and 4th comments.
Yeah, stupid. I’ve found when you make a mistake the best thing to do is to own up to it, and then find something interesting you can learn from it. Aside from learning to check the stats on my blog more than once every three months, I’ve also learned that the robots.txt file actually works. Yeah, in fact it works really, really well. This graphic says it all:
That’s what happens when your robots.txt settings look like this:
User-agent: *
Disallow: /
Not all that different than:
User-agent: *
Disallow:
Wouldn’t you say? But it’s quite different. It’s the difference between having your site almost completely de-indexed or having your site indexing marvelously. So kids, learn from my mistake and don’t do this at home, at work, or anywhere else unless you actually want to avoid the search engines.
I know a “professional SEO company” that did this for 4 months on a high-end site :S
Well it works with Google anyway. There are so many spiders that completely ignore robots.txt, it’s really annoying. You stats should come back pretty quickly though in my experience.
I once offered to help a friend of mine with the site for his business. He’d done his best to do some things on his own, and despite his efforts, he just wasn’t seeing any changes in his search results.
So I made some changes on his site, got him listed in some great topical directories, executed a pretty nice link building campaign — and his site vanished entirely from the SERPs. I couldn’t believe it.
After checking virtually every element on the site, I finally thought I’d check the robots.txt (he had implemented this himself while I was working on the site after misunderstanding a tutorial). Upon discovery, I facepalmed a few hundred times, but thankfully the rankings did come back (and eventually improve) in pretty short order.