Warning: If you’re the type of person who would rather remain in ignorance because you think that makes you innocent of any crime because you don’t know if you’re committing one or not, then you probably won’t want to continue reading. Of course now that you’ve read this it’s too late.
Disclaimer: I’m not really calling you a liar, cheat, or dishonest person. But I had to get your attention somehow because I want answers to this question, and apparently the post title worked. And if you use one of the tools mentioned in the post title, there’s at least a chance that you are doing something dishonest. First, try out this simple self-test to determine whether you’re an honest person or not:
1. Do you steal things from stores?
2. Do you tell lies or misrepresent things?
3. Have you ever charged a client for SEO services you never provided?
4. Have you ever made outrageous claims you knew were false about your SEO services, just to get a client to sign up?
5. Have you ever signed a contract and then violated it because it was beneficial to you and you knew you could get away with it?
6. Have you ever used Web CEO, SpyFu, SEOmoz’s Rank Checker, or any other similar programs?
If you’re thinking that question #6 doesn’t seem to fit with the others, then you’re where I was a few months ago. But then I started doing some research and stumbled onto section 5.3 of Google’s Terms of Service, which reads:
5.3 You agree not to access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google, unless you have been specifically allowed to do so in a separate agreement with Google. You specifically agree not to access (or attempt to access) any of the Services through any automated means (including use of scripts or web crawlers) and shall ensure that you comply with the instructions set out in any robots.txt file present on the Services.
Tools like Web CEO, SpyFu, and SEOmoz’s Rank Checker (as near I can tell) all perform automatic queries of Google. They also display data from Google within an interface that is not provided by Google. Therefore, these tools violate the Google Terms of Service, as do you when you use these tools.
There seem to be five primary arguments for ignoring section 5.3 of the Google Terms of Service:
1. I’ll never get caught or be punished.
2. Everyone else is doing it.
3. It doesn’t hurt anyone.
4. Google doesn’t care anyway.
5. The intent of section 5.3 isn’t to stop programs like SpyFu or SEOmoz’s Rank Checker.
Let’s take these one at a time.
1. I’ll never get caught or be punished. The same rationalization is used by many, if not most, criminals. Would you say that somebody who has a foolproof way to steal cars and never get caught is an honest person? Of course not. Whether or not someone ever gets caught or punished has nothing to do with whether they are honest or not.
2. Everyone else is doing it. Just because everyone else is doing it doesn’t make it an honest act. The same logic has been used for downloading pirated music and movies, doing drugs, and killing Jews.
3. It doesn’t hurt anyone. To say something doesn’t hurt anyone is speculative at best, and merely a way to make one feel better about actions they are uncomfortable with but want to justify somehow. People use the same logic to steal from Wal-Mart. After all, they’re just a big corporation, so how does me stealing some batteries hurt anyone? That argument can be made, but it is, admittedly, a more difficult argument to make when it comes to automatically querying Google. But it doesn’t matter. Whether or not it hurts Google or anyone else doesn’t change the fact that automatically querying Google is a violation of their terms of service.
4. Google doesn’t care anyway. Just as with #3, the same goes for claiming that Google doesn’t care. First of all, you’re making an assumption that may or may not be true, and second, it doesn’t matter. Violating an agreement is breaking your word and being dishonest, regardless of how the person or entity you’ve made that agreement with feels about it. If they don’t care, then they can let you out of the agreement, but for someone to make an agreement and then break it based on the assumption the other party doesn’t care is still dishonest.
5. The intent of section 5.3 isn’t to stop programs like SpyFu or SEOmoz’s Rank Checker. This is the only way out, but it’s a hard case to make. In order to say that you are not breaking your agreement with Google, you would have to somehow claim that Google does not mean what they seem to be saying. That the words they are using mean something other than what those words generally mean. You would have to be able to prove that when Google says they don’t want you to “access (or attempt to access) any of the Services by any means other than through the interface that is provided by Google” that this does not apply to Web CEO. You would have to be able to read the words “You specifically agree not to access (or attempt to access) any of the Services through any automated means (including use of scripts or web crawlers)” and interpret that as not applying to how SEOmoz’s Rank Checker gets its data. If you can do that, I’d like you to explain it to me, because I can’t see any way around it.
So what is the intent of section 5.3? My opinion is that it truly is not intended to target services like Web CEO or SpyFu. I believe Google included section 5.3 as a safety measure to give them the ability to prosecute anyone doing such things if Google deems it worth pursuing. I can imagine a meeting between Google and their lawyer going something like this:
Google: “Hey, what about this section 5.3? Why do we need that in there?”
Lawyer: “Because you might need to use it some day.”
Google: “What do you mean? I can’t see any reason why I’d need to stop anyone from automatically querying us? I mean, I know these tools like Web CEO and SEOmoz Rank Checker are doing it, but we don’t care about that stuff.”
Lawyer: “You might not see a reason today, and even I can’t think of a reason, but you should put it in there any way just in case. Something may happen that you can’t foresee that damages your business and that section may be the only way you can prosecute the offender and stop it.”
I’m 99.99% sure if Web CEO or any other company talked to Google and Google were completely transparent, Google would say “Go ahead, we don’t care.” But of course Google can’t say that publicly, because that could be used against them in a court of law if they ever did want to sue someone based on section 5.3.
If my opinion is correct, then that’s pretty annoying, isn’t it? After all, how are we supposed to track rankings for our client? By checking them manually? What if we’ve got 50 clients and each one has 200 keywords they’re tracking? Well, then you’ve got a problem. But people have problems all the time. What if everybody said “Hey, I believe in being honest when it’s in my favor, but if it benefits me to break my word then I don’t think that’s dishonest.” That’d be a nice world to live in, wouldn’t it?
Honest people are honest whether or not they’ve got problems and regardless of the money involved. Here’s what real honesty looks like.
Jon Huntsman owned the largest private business in the world–Huntsman Chemical. He’s mostly retired now and is funneling his billions of dollars into curing cancer. In 1986 Jon Huntsman agreed to sell 40% of his company to Emerson Camp, the chairman and CEO of Great Lakes Chemical for $54 million. They shook hands on the deal, but then Emerson lagged on getting the paperwork together. Six months later, nothing had been done on the deal, and Huntsman’s company had done spectacularly well and some things had happened with the economy so that in that short time period that 40%, instead of being worth $54 million, was now worth $225 million, or $171 million more than it was worth just six months earlier.
Emerson came to Huntsman and said that he knew the business was worth about $200 million. Huntsman told him no, it was worth about $225 million. Emerson suggested they split the difference, and that he would pay $125 million for the 40% of the business. Huntsman told him no, he wouldn’t do that deal. He told him he had shook his hand six months earlier and told him he would sell him that part of the business for $54 million, and that’s what he was going to stick to because he had given his word.
So are you an honest person? If you like to think that you are, and you use these SEO tools, then what decision are you going to make that will allow you to sleep at night?
Now, is there some way out of this? Perhaps. A company can try to make an agreement with Google so that they have permission to automatically query the search engine. I’ve tried that myself, but have been unable to get a response. And so far I haven’t found anyone else who has been successful either. If you know of someone who has made such an agreement, I’d love to talk to them. Or if you see some other way around this that allows a company to automatically query Google without violating the terms of service, I’d love to hear about it, because I like these tools and would like to be able to use them without feeling like I’m a dishonest person. I would love to find a way around section 5.3, I just don’t see it.
Like many others, my early work history includes work at grocery and consumer electronics stores. There was usually a posting some where in the store that asked that you not loiter, you not take pictures, and in some fashion, make the experience unenjoyable for other shoppers – often in lawyerease to be sure…
Read the first example here:
http://arstechnica.com/news.ars/post/20081204-editorial-does-amazons-iphone-app-go-too-far.html
“In the official App Store description of its new iPhone app, the company suggests it can be used for “comparing prices on Amazon and 9,000 other merchants to those in the retail store you are visiting.”"
So… if the TOS says not to take pictures, guess you can’t take a picture for your pricing app to work or your dishonest, eh? Better just buy those printer cartridges and computer supplies from Staples rather than loitering there, taking up their floor bandwidth.
You can loiter as a special class, think elderly mall walkers. Hey, they got no place to go eh? But not as a group of kids. Hey, they got no place to go. Both take up “bandwidth” in real life and likely are loitering despite signs asking that people not do so.
Plus it is disingenuous for Google to say that it doesn’t want apps scanning their database, then they go and do the same thing to others. When they promote “net neutrality” and say that want the internet open to all, so that carriers can’t discriminate against competing apps… so what’s good for the goose is not good for the gander?
You could put a TOS on your site that says Google must pay you $1,000,000 if it scans your site. Dude, they’re not going to pay. Does this make them dishonest? Does this even mean that their scanning was “illegal”??
The TOS doesn’t make automatic querying of their site illegal, it means they have the right to sue for damages in court if the TOS is violated.
But my point doesn’t have anything to do with legality vs. illegality but rather honesty vs. dishonesty, and they don’t always overlap. The point is that you enter into an agreement with Google when you use their site, and that if you use these tools you are violating that agreement. Whether or not Google would win or lose in court or sue in the first place isn’t the point, the point is that if you knowingly violate the agreement you are breaking your word.
I suppose you bring up a sixth reason for ignoring the TOS:
6. Google does dishonest things to others, so why can’t I do it to them? That is, if Google violates the TOS of another website, why shouldn’t we be allowed to violate their TOS? Although somehow this seems fair, it doesn’t meet the definition of honesty. Dishonest actions against a certain party aren’t made honest if the targeted party is dishonest themselves. Stealing from a thief is still stealing.
So while I agree that it may be disingenuous of Google to have section 5.3 in their TOS when they violate in similar manner the TOS of other sites, that doesn’t change the point I’m making.
Point taken but aren’t you being dishonest yourself? You aay “I’m not necessarily accusing you of being a liar, unethical, or dishonest..” but in fact that’s exactly what you are doing. In fact you are being dishonest about being dishonest and that’s truly dishonest.
I’m not accusing somebody who doesn’t know anything about section 5.3 of being dishonest (which I believe covers most people). In addition, somebody might have an interpretation of section 5.3 that they feel doesn’t prohibit them from automatically querying Google. I don’t know how that’s possible, but I’m not ruling it out.
@Davit Temple – honestly… nice comment.
@Joshua Steimle – nice article. ranks well for some specific keyword sets – bravo! we write contracts all the time for specific contingencies for when a client is recalcitrant. most often, a client will violate these sections, but we don’t enforce them, because it was an “honest” mistake… or whatever. point is… well… not sure i have one.
In fair competition I think that it may be wrong to violate any TOS. However, when a company like Google who owns the majority of all search traffic and thus search traffic data so it is Monopolistic to not allow any sort of third party data analysis to openly accessible data.
Okay..I have been thinking about this issue a lot today as I found a way to use a browser called Lynx to “automate” some search queries I do in my keyword research potentially making things a lot easier for me if I use this approach.
A couple of points…
First of all a person cannot be forced to agree to a TOS by visiting a site. If Google wants to insist that everyone who visits their site agree to their TOS that is their right to insist on that of course but their insistance does not in any way, shape, or form obligate someone who visits their site publicly to agree.
That’s like saying that that everyone who steps into Wal-Mart MUST agree to their Terms of Visit (if there was such a thing) before being allowed to use the bathroom inside the store or some such thing. Or that everyone driving up the drive through at McDonald’s MUST agree to a McDonald’s terms of use before being allowed in their drive through.
Such a thing would be ludicrous in the real world and should likewise be ludicrous in the online world. If you make a site like Google and make it publicly available for all the world to visit whenever they want you cannot expect to have everyone who visits enter into a legally binding contract just because they visit you on your very public site.
Second point…they talk of not accessing their services through other than their interface. What is their interface exactly? Is it the Internet Explorer browser? NO! Is it the Firefox browser? NO! The browser is the browser which doesn’t even belong to Google. Their interface is that HTML page which can be viewed in a browser. Their interface uses the HTTP protocol. That is how you communicate with their interface. That is the prescribed manner of accessing their interface.
If I use Lynx like so….
lynx -dump “http://www.google.com/search?hl=en&q=black tuxedos” | grep Results | awk -F “of about” ‘{print $2}’ | awk ‘{print $1}’
I can get the number of pages that result from this query. Am I accessing their service through their interface? You bet! Would Google consider this automated or not? Hard to say. They don’t answer very quickly if at all so we are left to determine this ourselves.
What is automation? Without a browser one would have to enter HTTP protocol terms into a command prompt. If we use a browser instead are we accessing Google through automated means? Yep! We are. The browser automates our request. Would Google consider such a thing automated? Nope. Why not? Beats me.
Things are not as clear cut as you might think.
Carlos
Maybe Google is saying we’re all obligated to access their website through the Chrome browser
Yeah…it would be natural for them to wish that wouldn’t it?
I love Chromium, the underlying open source browser that powers Google Chrome. I don’t think I will ever use Google Chrome (even it it comes out for Linux). I don’t like agreeing to EULA’s if I can help it as I take my agreements seriously.
You know I was thinking of this subject a bit more.
What would happen if I put a TOS at my site that said something to the effect that by visiting and accessing my site (personally or through a bot representing you) you agree to release me from any agreement to abide by your TOS altogether? Hmmm…interesting.
Would that legally obligate Google to release me from their TOS? Hmmm…maybe so. It’s an interesting thought. I wonder how the courts would rule on such a conflict of TOS’s?
I think web site TOS’s are utterly ridiculous and probably unenforceable. It’s like if some city decided to write a TOS about walking on a public sidewalk and claimed that everyone who stepped on that sidewalk would enter in a binding legal agreement to…let’s see…mail them a $100 bill.
Would that be enforceable? Of course not!
Yet web sites actually expect visitors to enter into binding legal agreements no differently when “stepping” on their sites? Ridiculous.
The internet is a public information hiway. Just as real hiways do not have highway TOS’s prominently displayed at their entrance ramps that we agree to by using their hiway so likewise internet sites are publicly accessible through a public thoroughfare. Their TOS’s are as worthless in my opinion as a TOS for walking on public sidewalks or entering a hiway would be in real life.
Carlos
I’m sure you could put just about anything on your site as a TOS, the question, of course, is whether you have the funds to take it to court and win. Obviously Google does.
But again, my concern is not about legality, ethics, whether or not it’s ridiculous (I think it is, but that’s besides the point), or whether I’ll ever be punished in any way–it’s a moral question: Can I consider myself to be an honest person if I violate Google’s TOS?
There may be some way of looking at this that let’s me out of the moral quandary, but short of Google saying “No, we’re fine with services like WebCEO or SpyFu, that’s not what we’re talking about with section 5.3 of the TOS” I don’t see any way around it. I wish we could get Matt Cutts or someone official from Google to sound in on this.
Hi Joshua,
You bring up some good points but as you say the question you posed is “are we an honest person if I violate Google’s TOS”.
Honesty definitely implies a moral standard by which to judge one’s honesty.
Morally speaking if I agree to abide by something I must, if I am to be morally honest and walk with integrity, abide by that agreement. And I do.
Question is…when I go visit Google, am I agreeing to their TOS?
If I am not agreeing to abide by it then it cannot be rightly said that I am being dishonest by not abiding to something I never agreed to.
I do not agree with their TOS or any other TOS anywhere on the internet that forces upon me an agreement that I do not make when visiting their publicly accessible internet site.
They say I agree to it by virtue of visiting their site but I do not in fact agree to it. Such an agreement cannot be forced upon me either morally or legally.
Any more than I can be legally or morally be “forced” to agree to the whims of TOS writing lawyers who put TOS’s on things in real life like using public toilets, hiways, sidewalks, or even the very public air we breath.
There is nothing illegal or immoral or wrong about me visiting their site without agreeing to their TOS. They make their site publicly accessible over the internet. It is accessible to me whether I agree to their TOS or not, strictly speaking (though Google would of course like to limit such access only to those who agree to their TOS).
If they wish to limit access to their site to only those visitors who agree to their TOS then they need to limit access to their site only to those people who check mark an agree button on a login in type of form. Where such an agreement is not forced upon people publicly visiting their site just because they say it is but rather because a visitor makes a conscious and aware choice to agree to their TOS to get access to search through Google.
That is the way real world agreements are made in real life and that is the only way that such a TOS “agreement” would be legally enforceable. Morally speaking I am under no more obligation to abide by their TOS as a result of visiting their publicly accessible site than I am to abide by a cities made up sidewalk “TOS” for walking on a public sidewalk.
I mean I may need to rethink this but that’s the way I see it at present.
Not to mention that if I put a counter TOS up at my site (something I am seriously considering doing to show the ridiculousness of such TOS’s) and Google accesses my site then I am freed from even the slightest perception of my having to abide by their TOS just for visiting their site (when my TOS disavows such an agreement in a counter TOS).
Carlos
There are legitimate methods of accessing Google, generally via their published API. You can indeed with Google’s go ahead access their system. Not all features are available, and there can be cost (or limits) imposed on such access, but most of it exists legitimately. Go through the process of getting API keys and you will see what I mean.
That said, most major tools out there don’t use the API methods because they are inferior to scraping data off of pages the old fashioned way. Google needs to enhance access to their systems and charge a small fee for access. Problem solved…
Ty,
There are NO legitimate ways approved by Google for getting SEO valuable information from Google that one needs in the course of trying to get their sites to rank higher. Backlink analysis…Google’s backlink count is a joke. Number of competing websites that show up for a given phrase in a Google search…NOT provided. PR as an indicator of how competitive a site is and how tough it will be to outrank them…only provided through their invasive Google Toolbar as far as I know and then only for one site at a time.
Name a single metric that is useful to SEO’s made available through Google’s approved channels and I will eat my words (to the degree that I am wrong).
It’s a comforting thought that good ol’ Google, who can do no evil in the eyes of so many, makes available what we need publicly and without unreasonable restriction but…that just ain’t so my friend! They will not give out information that allows the rest of us to more easily game or manipulate their flawed algorithm to achieve a higher ranking. They won’t cut their own throat.
Carlos
Hi there,
Been looking for Google’s ToS for a while, in relation to exactly the sort of automated search and return query available from SEOmoz, etc.
Carlos made some great points about ToS, in particular I’d like to address this one:
“First of all a person cannot be forced to agree to a TOS by visiting a site….”
The relevant part of Google’s ToS is this one:
2. Accepting the Terms
2.1 In order to use the Services, you must first agree to the Terms. You may not use the Services if you do not accept the Terms.
2.2 You can accept the Terms by:
(A) clicking to accept or agree to the Terms, where this option is made available to you by Google in the user interface for any Service; or
(B) by actually using the Services. In this case, you understand and agree that Google will treat your use of the Services as acceptance of the Terms from that point onwards.
Google’s ToS 2.2(B) states exactly the opposite of what Carlos asserts: that you can, in fact, be bound by a service provider’s ToS IMPLICITLY, by simply USING the service offered – in this case, their search engine and ranked results outputs. Whether this is, in fact, in any way a legally binding provision (a la the “sidewalk usage clauses” of Carlos’ hypothetical council) is another matter. I would be most interested to hear what this might be, legally speaking, seeing as we are not paying for the service, nor entering into a written contract with Google, when we use their services.
Regardless of this, however, once the results are in your hands, Google is no longer providing you with a service: once they have provided you with a search output, it is a browser that displays that data. In theory, if you performed a search by a non-robotic, manual method, then used a robot to query the output – which is a file on your own computer/server – then Google has no cause for complaint. It would still be a bit painful to do for hundreds of terms, but it does sort of shows how ridiculous is Google’s provision: once they have executed the search for you, you ‘own’ the results. I also agree with Carlos that Google’s “Service” does NOT include the browser interface used to view the results provided by their Search Engine Service – or any other visualisation/presentation method, for that matter.
One other point: in Australia, the Telecommunications act provision for SPAM is that prior to sending any email, a direct advertiser must satisfy three conditions, one of which is providing an opt-out clause. As Google has failed to do this for its ToS, (ie, “acceptance of the Terms from that point onwards” with no option to opt out) I suspect that they could be in contravention of Australian Government regulations. Failing to provide an overt, transparent mechanism to enable SEOs to enter into other agreements could also constitute a breach of the Good Trade Practices Act, though I am not familiar with that one so don’t quote me.
Hi capdancer,
You said…
“I would be most interested to hear what this might be, legally speaking, seeing as we are not paying for the service, nor entering into a written contract with Google, when we use their services.”
That is the crux of the matter isn’t it? If we are legally and/or morally bound (for purposes of this discussion) to Google’s TOS (or any web site TOS for that matter) just by visiting the site I mean.
I personally don’t think that is legally defensively in a court of law. Anywhere.
If any court adjudicated that visitors to a site are legally bound just by virtue of visiting a site to a TOS agreement as a legally binding contract…well…it would open to the floodgates to all manner of TOS ridiculousness.
I might create a TOS saying that by visiting my site you are legally obligated to pay me $1 for every page you see on my site. That is my charge. And that is what you agree to based on my TOS. Just for visiting my site. That if you don’t like that you can go elsewhere.
If TOS agreements are legally binding contracts and if visiting a site is seen to legally be giving consent, even if just implied, to such an agreement then I am perfectly within my rights to exert such a TOS upon every single site visitor. The more the merrier.
I mean if Google’s TOS is a legally enforceable contract between them and visitors then my TOS would also be.
If the basis for entering into legally binding contracts now becomes just a visit to a web site then we have turned contractual law on it’s head. We have entered into a topsy turvy world of agreements by implication and not by willful decision to enter into agreements either by a signature or by at least checkmarking a box indicating that we agree over the internet (which itself may not be legally enforceable either).
And every one and their next door webmaster will have a different TOS to which we are then legally bound.
Can you imagine a situation where every house on a block would have a TOS agreement for visitors to the sidewalk in front of their house?
This house (i.e. website) would have this TOS and another house this one and a third one, a different TOS. On down the block. All legally enforceable contracts that would be entered into by implied consent just for walking on the public sidewalk in front of one’s home. Meaning that if one walked on the sidewalk one gave an implied consent to an agreement that was not even clearly laid out and shown to every sidewalk walker.
There isn’t even a link to Google’s TOS on the front page of their site anymore (at least on the front page of Google.com). Yet Google expects every site visitor to have given implied consent to a supposed legally enforceable contract that regular users (i.e. sidewalk walkers) don’t even see? That’s completely ludicrous.
It isn’t even an issue. Completely unenforceable as a legally binding contract in my opinion.
Regarding the automated issue…even that is full of holes. When I enter a search phrase into Firefox, say “back tuxedos” as an example, Firefox takes that phrase, adds appropriate Google query parameters to it, and sends the query “http://www.google.com/search?btnG=Google Search&q=black tuxedos” on my behalf to Google. If that isn’t automated I don’t know what is.
Even the browser is an automated means of accessing Google albeit one that Google has no problem with.
Like you rightly point out what Google means is to access Google through a script or crawler in a way that does not return the search results in a Google approved way. But once the results are asked for and returned I believe we can do what we want with them on our own computers.
Please bear in mind that I am not advocating that everyone go around and completely, willy nilly ignore all TOS rules. Not at all. I am simply pointing out the ridiculous state of current TOS agreements and expectations. I personally do try and respect the various TOS agreements, where I can reasonably do so but, even Google itself does not enforce it’s own. So why should anyone be expected to abide by the letter of their TOS? Doing so will simply allow competitors to use software to check rankings or whatever while Google looks the other way. Leaving the person who abides by the letter and the spirit of the TOS in the dust in the race for high rankings. That’s not fair or right.
Carlos