Searching for Search Stuff


I just came across a reference to Blinkx (on metafilter, though my FeedDemon watch for search related stuff actually brought it to my attention). I bring it up because I’m fascinated by what’s going on with search right now, on a couple of levels.

In part it’s interesting to consider how much Google has fed this trend: while it now seems obvious that search technology — if it’s good enough — can dramatically change how we look at information and the structures used to organize it, I’m not certain how much of that is 20/20 hindsight. Prior to 1998 it was equally obvious that data should be categorized and dropped into a formal, hierarchical structure, so that users could do a search and then look for related information in the appropriate place in the hierarchy. With Google’s success, suddenly everybody is about search, self-defining structures, and latent semantic analysis.

The other part that’s really interesting is trying to predict how effective search will be when extended into other areas. What Google does is hard, but you’ve got one very handy element: there are explicit links between many of the pieces of data that you’re analyzing. Rather than having to consider each data point (Web page) in isolation and arbitrarily decide which one is a “better” result for the search term alien abductions, you can check the explicit relationships (links) that exist between the data points that appear to be relevant. If one of those data points is a “hub,” pointed to by many other data points, it’s likely very relevant.

When you’re searching your hard drive for that document about email filters that you were working on a couple of months ago, or searching your email for the message that had the address of the party that you’re supposed to go to, you’re back to evaluating each document individually. Also problematic, you’re likely looking for one specific result rather than results regarding a general topic. As interest in these areas increases the tools will inevitably find new approaches to improve the process, but I suspect that it’ll take a while.

While both MS and Apple have been promoting improved search on the desktop, I don’t know of too much neat new stuff actually out there for the individual user. “Enterprise search” is one area where this tech is already being promoted pretty heavily, though: if anyone has any experiences in this area, I’d love to hear about them.