This is a response to a Matthew Ingraham's post "Google helps newspapers––period." over at the Nieman Journalism Lab. I started this as a comment, but felt it needed some expanding, so here goes…
In the journalism world, there's a bit of a debate about whether Google is good or bad for newspapers. So let me just get this out of the way up front: Google IS bad for newspapers. Just not as is being currently debated. Here's how…
Before Google, information was a lot harder to find, and as such, the newspaper as a technology still held a sizable competitive advantage over the internet as an information medium. Less people used the web because it was less useful. In this pre-Google world, newspapers were creating a lot of value. The newspaper served as the choke point and filter for news and information for the masses. Because they were in this position of being the choke point, newspapers were able to extract exorbitant revenue from advertisers and hefty subscription fees from readers. Useful/relevant/high quality news and information were scarce––newspapers eased that scarcity for readers, and thereby created value. The same was true with regards to eyeballs and advertisers.
Remember, "All the news that's fit to print" used to mean something. Whereas once it was a point of pride, now it's an anachronism turned joke.
Because newspapers are now all in competition with each other for so many of the same eyeballs and ad dollars, they cannot afford, as Nick Carr has suggested, to turn off their Robots.txt file and go into hiding. To do so would be suicidal: it would mean to stop competing for these eyeballs that are the source of their cash. With 32% of all traffic coming from search, Google is the cheapest and primary of advertising for a newspaper site.
Google ended the scarcity of useful information on the internet and, to a certain extent, in the world. Gutenberg did something similar in 1440. In doing so, they both made the world a much better place, but killed off the scarcity that was early technologies (be they newspapers or town criers) source of competitive advantage and thus profitability. Nonetheless, value is always relative. 
The horse drawn carriage was an improvement over walking. The car was an improvement over the horse. And in New York, my bicycle and the subway beat 'em both. Here's the point: Businesses create value by easing a source of pain, and they earn profits when they create value in a way that is unique enough that consumers are willing to pay a premium for it beyond the cost of production. And you can bitch and moan all you want about technology, but newspapers have failed this test. Outside innovation has hurt them, but it is their own failure to innovate that is killing them.
Google change the news and information ecosystem, and it is these changes combined with newspapers' inability to adapt that are the root of the problem. Nonetheless, now that the world has changed, there's just no going back, regardless of what you do to your Robots.txt file.
In sum…Google: Bad for newspapers, good for the world.
