parislemon

Not Irrelevant

Q. Does Google need a deal to crawl Twitter? A. No. Google’s web bots crawl Twitter 120M+ times a day; 3 billion pages indexed & counting.
— Twitter Comms (@twittercomms) January 23, 2012

With all the Google Search+ talk over the past few weeks, there seems to be something very simple which has been lost: Google can access all public data on Twitter. The deal they had in place previously was simply for real-time bulk access. In other words, it’s more work, and less timely for them to access it now, but they still technically can.

There’s absolutely no excuse and no reason as to why they don’t include Twitter (or Facebook for that matter) profile results in their new Search+ social areas. Actually, there is a reason, but it’s something Google won’t just come out and say.

Update: After this post by Chris Dixon, it’s probably worth shedding a bit more light on the situation. From my understanding, the reason why the Twitter data deal was important to Google is that it’s very hard to regularly index all the content flowing into Twitter at any given moment. There’s simply too much data coming too quickly.

Twitter isn’t blocking access to any public tweets, but because Google’s “firehose” access is shut off, Google is likely missing some of that data. It’s a technical challenge, but it’s not about data being closed off, at least on the Twitter side of things.