Accretion: the growing together of separate parts into a single whole.

Bonus definition! Excellence: the state or condition of being excellent.

Step One: Aggregation

I begin with the proposition that aggregation has been one of the key concepts of the past five years or so—a period that we might roughly call the “Web 2.0 era,” if we were so inclined.

You can’t discuss aggregation without touching on RSS. We’ve never seen the oft-forecast “year that RSS goes mainstream,” exactly, but that’s really because we spent a long time looking at the wrong metric. While the online population at large has never taken to RSS readers as a replacement for just visiting a bunch of sites, RSS feeds (often unrecognized as such) have become mainstream as a component of the Web sites we visit or even the engine driving those sites behind the scenes.

Digg and its brethren (whether social, curated, or hybrid) represent another take on aggregation: you’re visiting Web sites, but those sites are themselves explicit aggregations of content from across the Web. Recognizing that the amount of stuff being created on the Web increases at a dizzying rate, we have invested in tools to aggregate what is most interesting (for an arbitrary value of “interesting”) from many different sources and present it all in a single place.

And while it might not be immediately obvious, the API boom of the mid- to late-oughts also gave the aggregation mindset a significant boost. Because developers could directly access the data and functionality of a wide variety of Web sites “behind the scenes,” we got a new kind of aggregation, most clearly expressed in the mashup: “oh, you want the data from site X overlaid on the map from site Y?” No problem. “Now you want to see the pictures from site A and videos from site B that are related to the topics discussed on site C?” Easy enough, here you go.

Step Two: Accretion

Aggregation obviously has real value and isn’t going away, but I believe that in the coming five years we’re also going to be talking a lot more about accretion. Consider this: a few days ago I was in a room with about a dozen people from the New York tech/startup scene who were asked what they were interested in and working on. Three people (and I was not one of them) used some variation of the phrase “real time” in their answer.

I think that in the short term this means that a lot of people will be focusing on the “twitch reflex” end of real time, with tools that offer immediate, compact, and contextual feedback based on users’ input: think pitches along the lines of “it’s like {Twitter, Aardvark, Foursquare} but {for, with} {dog owners, movie times, hyper-local content, flavor crystals}.”

As we go forward, though, I expect that more attention will be paid to the idea that “right now” is only one facet of “real-time,” and that it’s very probably not the most interesting facet of real-time. We’re now used to aggregating data across services to the point where it’s almost taken for granted, but the growth of interest in (and technological capacity to support) near real-time services means that exploration of data accreted over time will become an increasingly big deal.

Any real time service must allow users to participate quickly and easily (reference once again Fred Wilson’s twenty seconds tweet), which means that each data chunk will be relatively small; what’s interesting about those little chunks, however, is that they aggressively blur the distinction between data and metadata. The concern for real time services is not getting users to provide a lot of data at any single point in time, but rather to provide tiny bits of data relatively frequently.

Foursquare is the easy example here: is a foursquare checkin “data” or “metadata?” Many—possibly most—checkins have virtually no content in the traditional sense. You’re not asked to review or even comment on the venue you’re at, you don’t have to say why you’re there or who you’re with…it’s just user, timestamp, and location (and you don’t even explicitly provide the first two). By the standards of a 2004 Web service this would be metadata, not data.

The value here comes in continuing use. A single checkin or single tweet doesn’t mean much outside of a very short time window, and that has been one of the common criticisms of real time services thus far: that the half life of each piece of data is too short and the “content” too thin for this stuff to have real significance.  And that’s true, as far as it goes.

It hasn’t gone far enough, though, because our first response has been to aggregate. Twitter’s trending topics and Social Great’s excellent trend tracking make a certain kind of sense out of this real time data, but I think that these are transitional. The big shift comes when we figure out how to make sense of the direction and velocity of change represented in these snippets of data that people are tossing out with a minimum of effort, consideration, and editing.

Web 2.0 brought us the 1% rule: of 100 people online, one will create something, ten will “interact” with that creation, and 89 people will just look at the results. If the next few years go as I hope and expect, those numbers are going to be upended. We’re going to be working with much, much more creation of much smaller things. We’ll still be interested in the aggregated “where are we all now?” but we’ll be paying just as much attention to the accreted “what routes did we all take to get here?”

Well while it’s taken us a while to get there, but Daryn has now rewritten the LOTD bot. While I was on vacation and virtually inaccessible, Twitter disabled the “replies” API call, which meant that the LOTD bot couldn’t access the lyrics that people were posting. And since I was without internet access and had intermittent cell phone coverage, the bot went down and stayed down until Daryn managed to contact me.

After an amusing little incident where I accidentally sent the admin password for the server to hundreds of people, Daryn got coding and wrote LOTD Bot, Mark II. In addition to general improvements, Daryn added one completely awesome feature: now, if the bot can’t get replies from the Twitter API it fails over to do a search on Summize to find new replies to post. Sweet, no? Here are the (slightly updated) requirements and info for LOTD Bot, Mark II:

Requirements
To run a variation of this @lotd script, you must be able to understand Perl well enough to make some basic modifications to the script, and must be able to set up a simple database table (phpMyAdmin is your friend). In addition, you need a server or account at a Web hosting service that:

• Allows you to set up a (small) database.
• Allows you to run perl.
• Has the XML::Simple and LWP::UserAgent perl modules installed, or will allow you to install them.
• Allows you to run scheduled jobs (i.e. cronjobs)

Overview
It’s a very simple setup: there’s a single perl script running on my server that gets the replies posted to @lotd via the Twitter API (or via a Summize search, if the API call fails), loads them into a database table, and then republishes the posts in the lotd account (again via the API). It’s scheduled to run once every 15 minutes, around the clock. The script uses the XML::Simple and DBI modules, but doesn’t have any other dependencies.

To run your own bot using this script, you’ll need to:

1. Create the Twitter account that will be the repost bot.
2. Create a database table as described in the next section.
3. Update the script below:
   • The DB connect information for your database. (Line 14)
   • The Twitter username/password information for your Twitterbot account (there are “read from” and “write to” values, but they’ll point to the same Twitter account in most cases). (Lines 16 – 20)
   • The regex to remove @YourTwitterBot from replies before it reposts them. (Line 40)
4. Upload the finished script to your server.
5. Set up a cron job to periodically run your script (@lotd runs every 15 minutes).

…and that’s pretty much it.

Get the Database Table Structure (MySQL CREATE)
Click the link above, copy and paste, and change the table or field names as seems appropriate.

Get the Example Script
Click the link above, copy and paste, and make the updates noted in the post above. If you change the database table or field names, make sure that you also update the script appropriately.

A digital native is a person who has grown up with digital technology such as computers, the Internet, mobile phones and MP3. A digital immigrant is an individual who grew up without digital technology and adopted it later. A digital native might refer to their new “camera;” a digital immigrant might refer to their new “digital camera.”

Description of “digital natives,” coinage generally credited to Mark Prensky

The idea of building competitors to Twitter on the same platform, or redistributing Twitter to multiple players reminds me of the idea that New York City should be rebuilt in Ohio because it would be cheaper. Or perhaps we could distribute a little of New York City in every state of the Union. New York City is what it is because of the people who live and visit there. Building another New York City in Las Vegas doesn’t result in the phenomenon that is New York City.

Echovar on decentralizing Twitter

There are roughly three New Yorks. There is, first, the New York of the man or woman who was born here, who takes the city for granted and accepts its size and its turbulence as natural and inevitable. Second, there is the New York of the commuter—the city that is devoured by locusts each day and spat out each night. Third, there is the New York of the person who was born somewhere else and came to New York in quest of something. [...] Commuters give the city its tidal restlessness; natives give it solidity and continuity; but the settlers give it passion.*

E.B. White, Here is New York (1948)

It took a week or so for these three things to snap together in my head, but when they did I was surprised how beautifully and accurately E.B. White described the population of the Internet.

I consider myself, and many of the people that I know, settlers: we have chosen to live a part of our lives online, and building online is important to us. Commuters have long since arrived—people who come online for work, or to do a little shopping, or in search of entertainment, but for whom the Internet is simply an interesting, useful, nice place to visit.

I’m not certain that there are yet many, if any, true natives, but I’m looking forward to seeing the (to me) strange changes engendered by people who can take the physical and metaphorical connectedness of the Internet for granted as a part of their daily lives.

* Personal note: my parents (both originally from the Midwest) moved to New York in 1968, and I was born in 1971. When I first read this passage, years ago now, I finally started to understand how my parents’ New York was different from my own—and that New York was important to them in a way that I think I can never entirely understand.

Update: Since Fred Twittered this post, it has reached a somewhat larger audience than expected and I’ll add an addtional note: there will be some interesting new stuff happening with @lotd. I’ve been chatting back and forth with Daryn Nakhuda, a gentleman and a real programmer, about possibilities, and when you put together his ideas, my ideas, and his ability to write sane, reliable code, cool things should result. Stay tuned.

While the logic of “hide your shame” should dictate that I never, ever reveal the code that runs the Twitter Lyric of the Day Club, there’s been enough interest that I’m just going to suck it up and publish it. For any programmers who end up at this post: yes, the code is a little eccentric, and yes, there are a number of ways that it could fall down and hurt itself. Believe me, I know. That said, here’s the rundown:

Requirements
To run a variation of this @lotd script, you must be able to understand Perl well enough to make some basic modifications to the script, and must be able to set up a simple database table (phpMyAdmin is your friend). In addition, you need a server or account at a Web hosting service that:

• Allows you to set up a (small) database.
• Allows you to run perl.
• Has the XML::Simple perl module installed, or will allow you to install it.
• Allows you to run scheduled jobs (i.e. cronjobs)

Overview
It’s a very simple setup: there’s a single perl script running on my server that gets the replies posted to @lotd via the Twitter API, loads them into a database table, and then republishes the posts in the lotd account (again via the API). It’s scheduled to run once every 15 minutes, around the clock. The script uses the XML::Simple and DBI modules, but doesn’t have any other dependencies.

To run your own bot using this script, you’ll need to:

1. Create the Twitter account that will be the repost bot.
2. Create a database table as described in the next section.
3. Update the script below:
   • The DB connect information for your database. (Lines 21, 43, 75)
   • The Twitter username/password information for your Twitterbot account. (Lines 6, 84)
   • The regex to remove @YourTwitterBot from replies before it reposts them. (Line 18)
4. Upload the finished script to your server.
5. Set up a cron job to periodically run your script (@lotd runs every 15 minutes).

…and that’s pretty much it. As I said in my first email to Fred about this, “the hardest part of making something like the @lotd bot is just having the idea — once you’ve got the idea, Twitter makes it easy to build what you’ve got in your head.”

Good luck, have fun, and let me know about anything interesting that you make!

Update: In response to a couple of questions, I guess I didn’t make it explicit—this stuff is released under the Woodie Guthrie license: “This song is Copyrighted in U.S., under Seal of Copyright # 154085, for a period of 28 years, and anybody caught singin it without our permission, will be mighty good friends of ourn, cause we don’t give a dern. Publish it. Write it. Sing it. Swing to it. Yodel it. We wrote it, that’s all we wanted to do.”

Get the Database Table Structure (MySQL CREATE)
Click the link above, copy and paste, and change the table or field names as seems appropriate.

Get the Example Script
Click the link above, copy and paste, and make the updates noted in the post above. If you change the database table or field names, make sure that you also update the script appropriately.

There’s a link to the actual report (PDF) at the bottom, if you want some more depressing reading.

[Note:  This item comes from friend Ken DiPietro.  DLH]

From: ken 
Date: March 20, 2008 3:49:54 AM PDT
To: Dewayne Hendricks 
Subject: The FCC has released their High-Speed Internet Status report.

Included in this report is some pretty interesting facts, for those of
us that follow this kind of stuff.

There are now over 100,000,000 high speed connections (as
defined by exceeding 200Kbps in one direction) in service in the
United States.
A little over 60,000,000 are connected to residential dwellings.

Of those connections only 5.6% have a greater throughput than
25Mbps.

The total number of connections that have speeds in excess of
100Mbps (in one direction) is a staggering 21,708 as opposed to
Japan which has already achieved close to 100% deployment of
100Mbps.

Over 95% of all lines are serviced by the duopoly. This would be
the same duopoly that does not exist, according to AT&T's
management.

And with a level of hubris that is beyond all concept of
reality, we find the FCC stating that 99% of all US ZIP Codes
now have, at least, one broadband provider, a statement that
Commissioner Copps called the ZIP code methodology "stunningly
meaningless." Even better, roughly 85% of all ZIP Codes
(estimated) to have four or more providers.

And in a move that I can only term, better late than never, the FCC has
decided that 200Kbps (in only on direction) is no longer a true
definition of broadband) and has voted to increase that rate to 768Kbps,
which coincidentally is the speed that many of the ILECs provide as
entry level DSL.

The FCC's report. titled, "High-Speed Services for Internet Access:
Status as of June 30, 2007" can be downloaded here:
http://hraunfoss.fcc.gov/edocs_public/attachmatch/DOC-280906A1.pdf

A reasonably good review of this report can be found here:
http://www.news.com/8301-10784_3-9898118-7.html

Give some thought to Kim Cameron’s work regarding identity, paying particular attention to “delegation coupons”.

Run through the writing that Dave Winer has done in the last couple of days on user generated content (UGC).

Consider the evolution of online community, with specific reference to the model suggested by Twitter and Tumblr.

Keep VRM in the back of your mind as you go.

The get back to me…there are some really interesting conversations to be had here.

Now as I noted a few weeks ago, even though most of my Tumblr content is just feeds from other stuff I’m doing (Twitter, Flickr, del.icio.us, last.fm), I don’t like the idea of including the feed from this blog in Tumblr…the “long form” character of it just seems wrong for how I’m viewing Tumblr right now. But I do want to get some reference to this blog in there, so what to do?

The obvious answer, of course, is to write a WordPress plugin that allows me to automatically submit a short-form excerpt from these blog posts as Tumblr “quote” posts; this is in keeping with the way I’m thinking about Tumblr (lifestreaming snippets that require little or no care and feeding of their own), and seems like a fun little project, anyway.

So the above represents where I was when Gwen started her nap about 45 minutes ago. Note that the “simple” answer above is only possible because:

And those aspects of the tools are just there by default…this is a really good thing.

So now the good:

Big points go to both WordPress and Tumblr: halfassed programmer that I am, in half an hour I was able to make a functional (though crude) plugin that would take a designated snippet from a WordPress post and correctly submit it as a Tumblr quote, with a link back to the blog post. Both products are just designed to let you do what you want to do. Very nice.

And the bad:

No points go to the hosting company that is currently running seamonkeyrodeo. I wrote and tested the plugin on a dev installation of WordPress that I have set up on my primary Web hosting account, where it worked flawlessly. When I moved it over to the live version of SMR? Not so much. I can’t use the damn plugin on SMR, because (due to laziness and intertia) seamonkeyrodeo is hosted by a different company…a company that in two thousand fucking eight still hasn’t upgraded to PHP 5.

So, happy Sunday afternoon, everyone, and remember: APIs good. Crappy Web hosts bad. I have to go manually submit a quote from this post to Tumblr now, damn it…

By taking advantage of the basic interconnectedness of the Web, using that interconnectedness to make it easy to combine information that already exists somewhere, things like the FeedBurner/del.icio.us combination get past the idea that everyone must be able to easily create HTML documents in order to “contribute” to the Web.

In the years since I wrote that, del.icio.us and Backpack have happily replaced my homegrown shell script approach to saving links and notes in an easily accessible way, and a whole new world of tools that take advantage of “stuff that already exists somewhere” has appeared…let’s talk Twitter, Tumblr, and some miscellaneous related items.

Chapter One: Twitter
I created a Twitter account around the time of the SXSW explosion, but hadn’t used it at all until a couple of weeks ago. I saw its appeal in a setting like SXSW (“hey, look, X is in the presentation next door—maybe we can meet up later”), but wasn’t particularly interested in broadcasting the minutiae of my day to day life.

Then a colleague convinced me to give Twitter a fair trial, in part by acknowledging that he had a hard time figuring out what to twitter (tweet?) himself. So I’ve been giving it a shot, and come to a strange realization: I like it.

You see, about ten years ago I had a writing regimen: among other things, I had to document one moment per day. It didn’t have to be anything significant, but at some point during the day I had to pull out the notebook and record something. The side effect of this was that I found myself going through the day in a more observant and thoughtful manner. I was looking and listening a lot more, thinking more about what was happening around me.

And now Twitter has started to do the same thing for me. Don’t get me wrong: I’ve already done my share of it’s raining, and my butt itches tweets, but to put it in Holmesian terms I’m observing more, rather than just seeing. I even like the 140 character constraint: it’s very much of the moment to say it, but paring a thought or observation down to 140 characters is a great exercise in creativity.

Twitter has passed my first test for social software: it’s now useful enough to me as an individual that I want to keep using it. I may not be using Twitter the way one is “supposed to,” and I have a hard time seeing what value I’m adding to the collective with my tweets, but I’m going to stay with it.

Chapter Two: Tumblr
What is Tumblr, you ask? Well, when you come right down to it nobody’s entirely sure. Fred Wilson posted some thoughts on the question (with visual aids!) a week or so ago, but wisely chose to say more on what it’s like than what it is. Fred and Bijan Sabet tend to describe it with some combination of the terms “social,” “blogging,” “lightweight,” and “look, just try it, okay?”

What is Tumblr to me, you ask? Well, I don’t know either, but I’m intrigued. [And you can find anagrams of "lightweight social blogging" here, if you're interested. My favorite is "a belching gigolo's light twig."] Anyway, the social/reblogging aspect hasn’t particularly grabbed me thus far, so I feel like I’m missing the point on that front, but the “lightweight” has me excited.

In the most general terms, one could describe the content creation end of Tumblr as a strongly-typed blogging tool: a Tumblr post is one of seven things: photo, link, quote, chat, audio, video, or text. What immediately intrigued me about this is that it’s implicitly telling users that they don’t have to do the whole blog thing:

• You’ve just taken a neat picture and don’t want to write anything about it? Cool. Photo. Done.
• You just read/heard a great quote, but don’t want to annotate and unpack it? Excellent. Quote. Move on.

The Internet is littered with long defunct, three-post blogs in large part because “blogging” is generally viewed as “writing,” and therefore every blog must apparently carry the burden of high school English classes: What’s my thesis? Have I developed my arguments? Spelling counts! Jeez, haven’t I written 500 words yet? Tumblr tries to eliminate some of that friction by scaling down the ambitions of blogging; if you just want to post a picture, post a damn picture and don’t worry about whether it constitutes a “blog post.”

But Tumblr’s structure adds something else, as well. You see, Tumblr accepts RSS feeds as input. My Twitter account already generates a feed of my little text musings. My del.icio.us account already generates a feed of links I’m interested in. My Flickr account already generates a feed of my pictures. My blog already generates a feed of my longer-form writing (but more on that another time).

Then consider that with Disqus and IntenseDebate in play I’ve got feeds of many of the comments I make out there on the Web. With last.fm added to the mix I’ve got feeds covering the music I’ve been listening to, sliced and diced in a variety of ways. With a little bit of Applescript, the Tumblr API, and a cronjob on top of last.fm, I can automatically give Tumblr one of the songs I was listening to yesterday as an “audio” post.

Tumblr is becoming fascinating to me as a way to capture all of the stuff that I’m scattering around the Web (and my own computers, for that matter) in a structured way. With only a relatively small amount of setup work, I can assemble my own little lifestream and just watch how it evolves.

There are already tools out there to generate “lifestream” blogs (check out Kieran Delaney’s slick Simplelife plugin for WordPress, for example), but “creating another blog” doesn’t appeal to me, somehow…and this, I’d guess, is where the social aspect will come in to play. Whether it’s true or not, aggregating this stuff within Tumblr’s framework feels different from, and more interesting than, creating yet another blog.

As with Twitter, I have a hard time seeing what I’m adding to the Tumblr community, but I’ll give it some time. Perhaps after a while “reblogging” something won’t feel quite so much like a transparent plea for the poster to validate my presence on Tumblr by following me. I’ll figure out (how||whether) to incorporate my Tumblr dashboard into an already big pile of input. I’ll work out answers to all of the little questions that come up when playing with a new tool.

But however it works out, three and a half years from now there will be another set of shiny new toys to play with. Be sure to come back then and read the next post in this series.

Seems like the Big Content executives would benefit from coming up with an equally engaging response. If only they had access to a group of people who were used to developing entertaining content based on current events on a very tight schedule…

Update, 7:23PM: This video has now been viewed 61,165 times. Sure, it’s no Lazy Sunday, but 22k views in six hours sounds like a decent start on viral uptake to me.

To summarize, if one were a recently converted DVR-less Heroes addict who finds it inconvenient to watch the show when it airs, one’s (entirely hypothetical) options would be:

1. Go with the NBC software, bearing in mind that there’s currently no way to schedule the download of an entire season ahead of time, that only the prior week’s show is available at any given time, that episodes expire seven days after they’re uploaded to NBC Direct (not seven days after they’re downloaded), and that downloaded shows must be watched within 48 hours of download or they expire. In addition, one can only watch the download on one’s crappy, aging, Windows laptop. This model does, however, offer the advantage of getting to watch un-skippable ads during the program, though.

2. Go with software such as Azureus (and its plugins), scheduling download of each episode as it becomes available and watching the episode whenever one damn well pleases (sometimes it takes a week or so for one to find the time to watch, or one wants to save up and binge on TV for an evening, you see). In addition, one could watch the episode on any computer, or burn it to a disc to watch on one’s TV.

Given these options, the only reasonable conclusion that one can reach is that there’s some bizarre alternate reality out there where NBC Direct actually offers some meaningful benefit to some hypothetical viewer. The only other conclusion—that NBC never even bothered to look at the mistakes made by all of the failed similar offerings that litter the history of the Internet and made all of the same mistakes themselves—seems uncharitable.