Scott Karp is going to use Twitter again, and has published a long post explaining why. I didn’t know that he had left. But, I’m glad he’s back.

If you are a developer of web content sites, then you must read Data and the Future of the Web by Scott Karp and Database Gods Bitch About MapReduce by Rich Skrenta. Scott provides the vision of where you need to go, and Rich provides an explanation of the new tools that are going to get you there.

Right now, most publishers provide commodity data (i.e. the same news that you can read on 1,000 other sites) without adding any value to either their users or their advertisers. As Scott notes, Google is the king of extracting commodity data. That has given them the power to also extract most of the revenue. But, there is another kind of data, the personal data that is created by a community of users on sites like Digg and Twitter: “it’s the data that’s still in our heads, the data that we have not put in digital form.” As Scott sees it:

“The future of the web will be determined by companies that can overcome people challenges — to bring EVERYONE’S data online, and make it useful. “

This is the primary challenge content producers face! How to mine the data their users provide them, in order to produce a better content experience that, in turn, provides more value to their users, and advertisers. The ability to do this will be the key to building a great content business in the web 2.0 era. And, it’s why I feel so strongly that content sites must embrace social media.

If Scott shows us the goal, Rich shows us the technical means to get there. Right now, most content producers have a database driven content management system (CMS), combined with a traffic reporting tool like Google Analytics. While this is perfectly good for serving content, and measuring your traffic, this combination will not allow you to do the kind of data analysis that will be needed in the future. The data is going to grow exponentially, and only a system based on technologies like mapreduce, HDFS, and Hypertable will allow your data analysis infrastructure to grow with it (at a cost you can afford).

Gathering increased amounts of data, and building the infrastructure that allows you to analyze and act on that data is the future of large scale content on the web. The only other alternative is content at an individual scale targeted at a niche audience (i.e a blog). At that personal level, the author can truly understand and respond to their audience. At any higher level, you need more, and the most successful publishers will be the ones who have the necessary tools.

Here’s a flash poem by M.

I have a guest post up on MetzMash called Three Reasons Content Publishers Need to Embrace Social Media. Thanks are due to Don Marzetta who generously shared his thoughts on the issue. Now, I eagerly await Dan Zarrella’s comments.

UPDATE: The post has now been picked up by Social Media Today as the choice of the day.

Guess this explains the post-dated email I got from Dreamhost: $7.5 Million Billing Error at Dreamhost.

Dan Zarrella has a guest post on Read Write Web on how you can use Twitter to boost your social profile. I have to say that my first reaction to this idea is highly negative. Can you tell from the post title? Call me naive, but what I love about Twitter is it’s utility as a tool for increasing social interaction between real groups of people who then (sometimes) build real relationships. As Terry once said to me, “I used to always follow everyone who followed me, but now it’s too many people.” I’ve found the same. Or rather, I’ve found that the majority of people who start following me are people who I have no chance of building a connection with, because they are following 2,000 others or they are a brand. So, I don’t follow them back. That’s regrettable, because that lack of interchange is the antithesis of what Twitter is about. What saddens me is that Dan’s article (and I should really emphasize I have nothing against Dan or his advice) is a marker to me for the growing commercialization of Twitter. We are in the middle stages of what we (assuming you are old like me) already went through with Usenet, and the web itself, and blogs. You knew it had to happen, but why did it have to happen so fast?

UPDATE: Here’s an even more jaundiced view from Nate Westheimer.

ANOTHER UPDATE: Dan, the author of the piece I criticized, took the time to come here and comment on my post. So, rather than replying in the comments, I decided to add another update: Yes, Dan, I am aware that you can unfollow, but it doesn’t really address what I am talking about. My title is both a little over the top, and (more importantly) misleading, because I am not really criticizing your advice (which is actually good for it’s purposes). My reaction is driven by the realization that Twitter is becoming as commercial as the rest of the web, and it makes me a little sad. In any event, you will get to call me a hypocrite, when my guest post on MetzMash appears later this month. It’s going to be about why web content publishers should have a social networking presence.

LAST AND FINAL UPDATE: Ok. We all knew it had to end this way. I started following Dan’s Twitter.

I went to another SF PHP Meetup last night. As always, it was highly entertaining, and well worth staying in the city until 10 PM. I love the fact that we host these at CNET now. Big thanks to Tougeron for running these meetings. As you can see from Touge’s account, Terry was in full effect (that’s a good thing). Next time, Mager should come instead of watching VT getting their ass kicked.

Greg Linden is predicting another dot-com crash in 2008. I have to agree that there’s a high possibility that he is right. I see a lot of web 2.0 companies getting funded, and I still don’t understand how they ever expect to make money. With the way the rest of the economy is stalling, it would be amazing if it didn’t hit the web.

Zed Shaw, the creator of Mongrel, has written an expletive-filled rant about the Ruby community. It was picked up by TechCrunch and Techmeme, and the comments have been flying. It’s a pretty amusing read. But, if I want some Ruby hate, I will stick with Terry.

So, I added myself to Tweeterboard. It’s an application that tries to measure your influence on Twitter. It’s not a bad service, but it still has a few problems to work out. In particular, it doesn’t seem to update the stats very often, and it misses a lot of links that I post. Still, a good initial effort, and worth checking out.