Lehrblogger

This post develops the conceptual language surrounding the product ideas in my previous post, Filtration as a Tonic for Internet Vertigo.

Towards the beginning of his aforementioned blog post, John Borthwick writes:

what emerges out of this is a new metaphor — think streams vs. pages

I think that this progression of metaphors is moving in the right direction, but it needs to be taken further. Streams consist of web content that is delivered directly to the user (to a Twitter client, to an RSS reader, etc), and this is in direct contrast to content that lives on specific pages to which a user must navigate in a web browser. Streams are dynamic, up-to-date and are delivered in (near) real time, while pages are static and not necessarily current.

Streams, however, are just collections of individual pieces of content, or packets. Tweets, status updates, blog posts, photos, mp3 files, and video clips are all discrete packets of content. These packets are the units which a user actually consumes as information, and streams are just a way to group those packets over time, usually based on on source (such as a specific blog) or topic (such as a search term on Twitter). But there are potentially more potent ways in which these packets can be organized than by their original source/topic, and this is important because these streams tend to be overwhelming in their aggregate. Borthwick continues about the future of content delivery via streams:

Overload isn’t a problem anymore since we have no choice but to acknowledge that we cant wade through all this information. This isn’t an inbox we have to empty, or a page we have to get to the bottom of — it’s a flow of data that we can dip into at will but we cant attempt to gain an all encompassing view of it.

I suspect that there is a more optimistic solution, however, and that there are better-than-random ways to organize the flow of content from our collections of streams. There will be some packets in these streams that are more important to individual users than others, so I want services that surface the best ones and hide the others. I predict a future in which streams are cut, rearranged, reordered and remixed into a single source of content that always has that moment’s most important/relevant/enjoyable packet at the front of the queue. The future of content on the web will be based on tools that focus on perfecting the delivery of these individual packets of information to users for consumption.

I agree with Borthwick, and think that the re-conceptualization of the destination web of pages into a real time stream of pages is the next (or current?) big thing. But I think the re-conceptualization of those streams as collections of individual operable packets is the big thing after that.

I’ve been thinking about this topic for a while, but was finally inspired to write it by a not quite recent but still very relevant blog post by John Borthwick (of betaworks, the startup accelerator associated with bit.ly, where I’m currently interning) about real-time distribution via social networks.

Update: I’ve written more about this topic in this subsequent post.

Anyone whose goal is “something higher” must expect someday to suffer vertigo. What is vertigo? Fear of falling? Then why do we feel it even when the observation tower comes equipped with a sturdy handrail? No, vertigo is something other than the fear of falling. It is the voice of the emptiness below us which tempts and lures us, it is the desire to fall, against which, terrified, we defend ourselves.
- Milan Kundera, Czechoslovakian novelist (1929 – ), in The Unbearable Lightness of Being

I have vertigo. My somewhat lofty goal is to read and digest all of the information that interests me, as it is created in real time, regardless of medium. My desire to fall is my desire to abandon all of my information sources, not bother keeping up with anything, and fall endlessly into ignorance. And this terrifies me; at the very least, the Internet could come equipped with better handrails.

I am interested in information from a variety of sources – blogs, people on Twitter, email lists, search terms in the NY Times, etc – and I subscribe to these things because I think they are worth reading. Although I wish I could read all of it, I know I can’t. But I want a better way to read only some of it, without having to face the infinities that I don’t have time to read, without having to make painfully arbitrary decisions about what to read and what to ignore, and thus without having the subsequent vertiginous desire to give up, declare email bankruptcy, and read none of it at all. So the question is, then, one of designing a sturdier handrail that I can grasp while observing information on the Internet as it streams by. And that handrail must be a tool for filtering content, not a source that recommends even more.

Recommendation sites/services such as Digg, Reddit and StumbleUpon have (as far as I know) user-preference modeling algorithms to make selections of what content to show to users, based on what those users and other users with similar past preferences have liked in the past. Netflix does something similar to make movie recommendations. They’re good systems, and have some cool machine-learning stuff going on, but I find their application to be fundamentally conceptually inverted.

I want to read the blogs of danah boyd, Jan Chipchase, and John Gruber. I want to follow Alex Payne, Jorge Ortiz, and 180 odd others on Twitter. Yet it’s too much. As Clay Shirky pointed out at the Web2.0 Expo, our filters have failed, not because of ‘information overload’ of ever-increasing magnitudes, but instead because of ‘filter failure’. Content was once primarily filtered by the editors and publishers, yet those systems are crumbling and I no longer have effective filters for this smorgasbord of carefully selected and professionally prepared feeds.

And I certainly don’t need Digg/Reddit/StumbleUpon to make additional recommendations. But given that I’m already not going to see all of the things I know that I care about, why can’t those same algorithms be used to filter incoming content instead? These information filtration systems would ideally have a few particular characteristics:

• They would be cross-platform and encompass all sources of information, including RSS, news, Twitter, Facebook, email lists (but not email addressed specifically to you), etc.
• The user could make adjustments to the aggressiveness of the filtration algorithm on a feed-by-feed basis.
• Filters would operate on a per-item level – i.e. I might be shown only one photo from an album of sixty, or one tweet from a day’s worth of a dozen. The content filtered out would not be less-than-worthwhile (you might really wish you could see all of your aunt’s vacation photos), but if the user simply doesn’t have time for everything, something must give.
• Users could build their preference models both passively and actively. The system would strongly take into account rating systems (number of stars, voting up/down, etc), but would also take into account how many seconds a user spent looking at a specific piece of content. Eye tracking would be preferable, but that’s tough and requires a camera at the very least. Instead, content delivery applications (such as RSS readers and mail clients) could measure how long a person spent with a piece of content open, adjust for word count and relative time for other pieces of content from that source, and assume that more time spent implies more engagement.
• Preference models would be partially social, but not exclusively. If a friend of mine liked something, it would be more likely to pass my filters. In addition, if some piece of content was enjoyed by a random other person who had shown similar past preferences to myself, it would be more likely to pass my filters too.

I don’t see this problem of perceived information overload (and consequent vertigo) getting any better on it’s own. Are there other solutions I’m not seeing? Anyone looking for a new giant software project?

Taken at the northeast corner of Rivington and Essex in the Lower East Side of Manhattan, facing west. Judging from the labels on the buzzer near the door just outside of the right edge of the frame, I believe the sign is there courtesy of a design firm named dress code. They’ve got a pretty awesome website, so go check it out, and then go check it out again a few hours later.

Click through for a smaller version on Flickr, and never sleep.

I’ve recently been using a Mac OS X tiling window manager named SizeUp. I had an idea about which I emailed the developer, and I thought it was worth sharing here too -

Hi,

I’ve dreamed about having a tiling window manager for years, and I was fortunate enough to see Alex Payne’s tweet making a similar lament; I found SizeUp as a result. I’ve been using it for a week or two now, and I love it. Both on my 15″ MacBook Pro, and when the MBP plugged into my 30″ external display. Nicely done.

I have a suggestion though – I wish I had slightly more control over how windows were sized. I understand how it could be a user interface and/or Mac OS X nightmare to let users specify custom sizes or behavior for how windows snap to other windows. But at the same time, I’d love to be able to assign windows to a grid location occupying less than a quarter of my screen. For example, Adium chats don’t need that much screen real estate, even if iCal and Firefox do.

Rather than use the arrow keys to indicate location, what if you used the physical QWERTY keyboard locations of the letter keys? In it’s most simple form, W S A D could correspond to Up Down Left Right. But you can do more interesting things too, and split a screen into a six- or nine-rectangle grid using Q W E A S D or Q W E A S D Z X C. Any one of those keys would place a window in the respective corner of the grid, so [shortcut combination]+Q on a six rectangle grid would make the window use one sixth of the total size, with a width equal to a third of the screen’s width and a height equal to half of the screen’s height. If you wanted a larger window, you could press multiple adjacent letter keys simultaneously – [shortcut]+Q+A would make the window take the leftmost third of the screen, and [shortcut]+Q+W+E would have the same effect as the current [shortcut]+Up. It might make sense to use shortcut keys on the right side of the keyboard with these letters (shift+option seem to be sufficiently unused in this context), or you could use a different set of letters and keep the shortcut keys where they were. In the SizeUp Preferences, users could even specify the number of rows and columns in the grid they wanted to use and the keys they wanted to assign to each screen location.

(This idea is somewhat reminiscent of a still-incomplete project I was doing a few months ago: a Javascript portfolio website that used the keyboard for navigation, had a representation of the keyboard on the screen, had all content exist on that visual representation of a keyboard but at different zoom levels, and used the physical keyboard keys and spacebar to zoom in and out on the on-screen keyboard to view content at different levels in the hierarchy. There are a few blog posts about it here.)

Sorry that was so long-winded, and please email me if you want to discuss further or need me to clarify.

Keep up the good work!

Best,
Steven Lehrburger

cross-posted at textonic.org
previously: Design for UNICEF – Detextive*, Mobile Tech for Social Change Barcamp, Design for UNICEF – RapidSMS and Mechanical Turk

Textonic.org is now live!

Recognizing that Textonic was a larger project than I was going to be able to finish in my free time over the summer (or that my group was going to be able to finish in our collective free time), I decided that the project would benefit from a more formal web presence than a handful of blog posts and a GitHub page. I registered a domain, set up WordPress, presented what we had accomplished, laid out what there is to be done, and tried to create a place where people could express their interest in getting involved.

The Conversation page on the new site is of particular interest. I’m using a Twitter search for the term ‘textonic’ as a sort of guestbook or message board. People who find the site and are interested can look to see who else has been there as well as when they expressed interest. Twitter itself can serve as a way for them to get in touch. The intransient* and public (since all of a user’s followers will also see the tweets) nature of these expressions of interest will help to catalyze the formation of a community around the project.

Credit for this idea goes to @n8han (who writes the blog Coderspiel) – I first saw it on his site for Databinder Dispatch. In addition, recent ITP graduate @joshbg2k created something similar for his Überbaster project.

*Note that Twitter’s search API only exposes that last ~3 months of tweets, so at some point I’ll need to archive the messages so that the entire conversation history is displayed.

I was awarded a student scholarship to attend the annual Apple Worldwide Developer Conference, so I’ll be flying to San Francisco in June. According to the website, the event “provides developers and IT professionals with in-depth technical information and hands-on learning about the powerful technologies in iPhone OS and Mac OS X from the Apple engineers who created them.” Many of the planned sessions sound interesting, and I’m also looking forward to the Apple Design Awards. I’ll probably spend the next month working on Meetapp (rather than Delvicious) so that I’m better prepared for the iPhone development labs.

I’m also planning to attend the Scala Lift Off on June 6th, a conference for developers using Scala and the associated web framework Lift. I hope to have more time to talk with some of the people who attended the Scala BASE meeting at which I presented my TwiTerra project in January. Martin Odersky, the creator of Scala, will also be speaking, and my friend Jorge Ortiz plans to attend as well.

I’m excited to go to both events and to visit other friends in California, so I’ll be sure to post about them again.

previously: project description, initial progress

David Nolen’s Little Computers class is now over, and I’ve made substantial progress this semester on Meetapp, the iPhone application I have been building for users of Meetup.com. Currently, it will fetch a list of a user’s groups from the API at that user’s request, make secondary API requests to fetch the events associated with each of those groups, and then display all of those events (i.e. all of those events that a user might be attending) in a table. It will also filter that table to show only those events that the user is organizing (rather than simply attending), and the user can drill down for more detailed information about both types of events. So far, I have -

• built a straightforward application that uses a navigation controller, a toolbar, and some tables to organize the information.
• implemented HTTP communication with the Meetup API using stored credentials for the user.
• created data structures for managing information about a user’s groups and events.
• designed and re-designed the interface so that it was simpler, cleaner, and better-suited for the functionality I was actually implementing.

The app has some use in it’s current form, but there’s a lot I still want to do before it’s ready to be posted for the public in Apple’s App Store.

The most recent code is on GitHub, and please feel free to contact me with questions about my progress so far or future plans. I hope to continue the project later this summer, but Delvicious will be my prioritized side-project, at least for the time being. I’m looking forward to watching the lectures from Stanford’s iPhone class to brush up when I jump back in to Objective-C in a couple of months.

project page here with overview and previous posts

http://delv-icio-us.appspot.com/

A very alpha version of Delvicious is live at the above link! You can currently sign in with your Delicious account (or use one of the ones I posted towards the bottom of the GitHub page), tell it to fetch all of your bookmarks (making a copy of the URLs in Google’s datastore), and then perform searches of those pages using a Google Custom Search Engine that is created on the fly* from XML files that are automatically generated and served by App Engine. I’ve had a little trouble getting the CSE’s to work the first time a user tries to use the search box, and I suspect that the search engines are not immediately ready for use after their (implicit) creation (which I think happens when the user first tries to do a search). One of the first of many next steps is to find a way to initialize that search engine automatically and inform the user (via email) when it is ready. If it’s not working for you, wait a little while and try again later.

There’s a lot of other functionality I want to add, and other next steps include:

• automatic updating of a user’s Delicious bookmarks, modifying only the necessary entries in the Datastore to reflect the changes on Delicious
• leveraging Delicious tags to improve search results – both by using them as refinements, and by being sure to return results that have the tag you are searching for (I’d love to have support for a “tag:tagname” operator in the search box)
• an option to leverage the number of other users on Delicious who have bookmarked a site to improve search results, since presumably heavily bookmarked pages are more useful
• Delicious results optionally displayed side-by-side with the Google results, both for comparison, and to make sure that all useful results are found (currently Delicious finds things that Google doesn’t) (I might use the Beautiful Soup HTML parser to scrape the results from the Delicious search pages since they aren’t exposed by the API)
• improve authentication with Delicious
• explore ways to support the searching of private bookmarks (but I’m not sure this is possible, since Google needs public XML files to create the search engines)
• do the graphic design for the various pages

I will continue to work on the project over the summer, and I hope to launch an initial version within the next month or two.

* I did this using Linked Custom Search Engines, and it’s worth noting that this was not the original plan. I had intended to ask users to authenticate with both Google and Delicious, and use the Google authentication for the creation of and interaction with CSE’s using HTTP requests. CSE’s only support Client Login authentication, however, which is intended for desktop/mobile applications and requires users to manually type in their usernames and passwords. I knew that I would not be able to convince users to give me their unencrypted Google login information (nor did I really want to possess it), so, without the ability to authenticate Google users on a google.com Google sign-in page, I was going to need an alternate solution.

project page here with overview and previous posts

I’ve had to focus on other projects for the past couple of weeks, but I finally got to turn my attention back to Delvicious. It will now:

• keep track of Delicious login information for a particular Google account
• fetch those bookmarks using the Delicious API and store them in the GAE Datastore
• serve the bookmarks back as an XML file to a user-specific URL

There’s enough functionality to make it worth visiting the appspot page here. Sign in with Google, enter your Delicious credentials, and fetch your bookmarks – you should see them displayed in a list. You could then make a custom search engine, go to the advanced tab on the left side bar, and add a URL of the form delv-icio-us.appspot.com/delvicious/annotations/[your_delicious_username].xml as an annotations feed. Your bookmarks should then start appearing in your search results. Test out the new version (that searches memento85’s small number of Delicious bookmarks) below –

I need to clean up the interface navigation, but the real next step is to dive into the CSE API documentation and automate the creation of the search engine so that one is automatically paired with a Google users Delicious account.

The problem with the plethora of social websites is not that people have to keep track of and maintain multiple profiles/identities – we do that in real life too (acting differently in different situations, talking about different things with different people, etc) and we’re pretty good at it. The problem is instead that we have to keep track of multiple profiles/identities for every person with whom we interact in more than one context. It is this social/cognitive task which we find so disorienting.

Note the inaugural use of the ‘thesis’ category! It’s not due for over a year, but this is the task, or at least an early formulation of the task, that I want to tackle with my final ITP project.