Lehrblogger

I’ve been thinking about this topic for a while, but was finally inspired to write it by a not quite recent but still very relevant blog post by John Borthwick (of betaworks, the startup accelerator associated with bit.ly, where I’m currently interning) about real-time distribution via social networks.

Update: I’ve written more about this topic in this subsequent post.

Anyone whose goal is “something higher” must expect someday to suffer vertigo. What is vertigo? Fear of falling? Then why do we feel it even when the observation tower comes equipped with a sturdy handrail? No, vertigo is something other than the fear of falling. It is the voice of the emptiness below us which tempts and lures us, it is the desire to fall, against which, terrified, we defend ourselves.
- Milan Kundera, Czechoslovakian novelist (1929 – ), in The Unbearable Lightness of Being

I have vertigo. My somewhat lofty goal is to read and digest all of the information that interests me, as it is created in real time, regardless of medium. My desire to fall is my desire to abandon all of my information sources, not bother keeping up with anything, and fall endlessly into ignorance. And this terrifies me; at the very least, the Internet could come equipped with better handrails.

I am interested in information from a variety of sources – blogs, people on Twitter, email lists, search terms in the NY Times, etc – and I subscribe to these things because I think they are worth reading. Although I wish I could read all of it, I know I can’t. But I want a better way to read only some of it, without having to face the infinities that I don’t have time to read, without having to make painfully arbitrary decisions about what to read and what to ignore, and thus without having the subsequent vertiginous desire to give up, declare email bankruptcy, and read none of it at all. So the question is, then, one of designing a sturdier handrail that I can grasp while observing information on the Internet as it streams by. And that handrail must be a tool for filtering content, not a source that recommends even more.

Recommendation sites/services such as Digg, Reddit and StumbleUpon have (as far as I know) user-preference modeling algorithms to make selections of what content to show to users, based on what those users and other users with similar past preferences have liked in the past. Netflix does something similar to make movie recommendations. They’re good systems, and have some cool machine-learning stuff going on, but I find their application to be fundamentally conceptually inverted.

I want to read the blogs of danah boyd, Jan Chipchase, and John Gruber. I want to follow Alex Payne, Jorge Ortiz, and 180 odd others on Twitter. Yet it’s too much. As Clay Shirky pointed out at the Web2.0 Expo, our filters have failed, not because of ‘information overload’ of ever-increasing magnitudes, but instead because of ‘filter failure’. Content was once primarily filtered by the editors and publishers, yet those systems are crumbling and I no longer have effective filters for this smorgasbord of carefully selected and professionally prepared feeds.

And I certainly don’t need Digg/Reddit/StumbleUpon to make additional recommendations. But given that I’m already not going to see all of the things I know that I care about, why can’t those same algorithms be used to filter incoming content instead? These information filtration systems would ideally have a few particular characteristics:

• They would be cross-platform and encompass all sources of information, including RSS, news, Twitter, Facebook, email lists (but not email addressed specifically to you), etc.
• The user could make adjustments to the aggressiveness of the filtration algorithm on a feed-by-feed basis.
• Filters would operate on a per-item level – i.e. I might be shown only one photo from an album of sixty, or one tweet from a day’s worth of a dozen. The content filtered out would not be less-than-worthwhile (you might really wish you could see all of your aunt’s vacation photos), but if the user simply doesn’t have time for everything, something must give.
• Users could build their preference models both passively and actively. The system would strongly take into account rating systems (number of stars, voting up/down, etc), but would also take into account how many seconds a user spent looking at a specific piece of content. Eye tracking would be preferable, but that’s tough and requires a camera at the very least. Instead, content delivery applications (such as RSS readers and mail clients) could measure how long a person spent with a piece of content open, adjust for word count and relative time for other pieces of content from that source, and assume that more time spent implies more engagement.
• Preference models would be partially social, but not exclusively. If a friend of mine liked something, it would be more likely to pass my filters. In addition, if some piece of content was enjoyed by a random other person who had shown similar past preferences to myself, it would be more likely to pass my filters too.

I don’t see this problem of perceived information overload (and consequent vertigo) getting any better on it’s own. Are there other solutions I’m not seeing? Anyone looking for a new giant software project?

I’ve recently been using a Mac OS X tiling window manager named SizeUp. I had an idea about which I emailed the developer, and I thought it was worth sharing here too -

Hi,

I’ve dreamed about having a tiling window manager for years, and I was fortunate enough to see Alex Payne‘s tweet making a similar lament; I found SizeUp as a result. I’ve been using it for a week or two now, and I love it. Both on my 15″ MacBook Pro, and when the MBP plugged into my 30″ external display. Nicely done.

I have a suggestion though – I wish I had slightly more control over how windows were sized. I understand how it could be a user interface and/or Mac OS X nightmare to let users specify custom sizes or behavior for how windows snap to other windows. But at the same time, I’d love to be able to assign windows to a grid location occupying less than a quarter of my screen. For example, Adium chats don’t need that much screen real estate, even if iCal and Firefox do.

Rather than use the arrow keys to indicate location, what if you used the physical QWERTY keyboard locations of the letter keys? In it’s most simple form, W S A D could correspond to Up Down Left Right. But you can do more interesting things too, and split a screen into a six- or nine-rectangle grid using Q W E A S D or Q W E A S D Z X C. Any one of those keys would place a window in the respective corner of the grid, so [shortcut combination]+Q on a six rectangle grid would make the window use one sixth of the total size, with a width equal to a third of the screen’s width and a height equal to half of the screen’s height. If you wanted a larger window, you could press multiple adjacent letter keys simultaneously – [shortcut]+Q+A would make the window take the leftmost third of the screen, and [shortcut]+Q+W+E would have the same effect as the current [shortcut]+Up. It might make sense to use shortcut keys on the right side of the keyboard with these letters (shift+option seem to be sufficiently unused in this context), or you could use a different set of letters and keep the shortcut keys where they were. In the SizeUp Preferences, users could even specify the number of rows and columns in the grid they wanted to use and the keys they wanted to assign to each screen location.

(This idea is somewhat reminiscent of a still-incomplete project I was doing a few months ago: a Javascript portfolio website that used the keyboard for navigation, had a representation of the keyboard on the screen, had all content exist on that visual representation of a keyboard but at different zoom levels, and used the physical keyboard keys and spacebar to zoom in and out on the on-screen keyboard to view content at different levels in the hierarchy. There are a few blog posts about it here.)

Sorry that was so long-winded, and please email me if you want to discuss further or need me to clarify.

Keep up the good work!

Best,
Steven Lehrburger

The problem with the plethora of social websites is not that people have to keep track of and maintain multiple profiles/identities – we do that in real life too (acting differently in different situations, talking about different things with different people, etc) and we’re pretty good at it. The problem is instead that we have to keep track of multiple profiles/identities for every person with whom we interact in more than one context. It is this social/cognitive task which we find so disorienting.

Note the inaugural use of the ‘thesis’ category! It’s not due for over a year, but this is the task, or at least an early formulation of the task, that I want to tackle with my final ITP project.

For my midterm project for Programming A to Z I decided to start working on a Delicious / Google Custom Search Engine mashup that I’ve been wanting to make for a few months. It will be called Delvicious, and a complete description of the project can be found on its project page. This post will primarily be about the initial implementation details and the progress I’ve made so far, but will conclude with some future plans.

I started off by looking at various options for getting a user’s bookmarks on the Delicious Tools page. I decided to use RSS feeds for the very first version, but those are limited to 100 bookmarks and I knew that I’d have to switch to something else later. I spent a long time familiarizing myself with Google’s Custom Search Engine tools – there are a lot of options for customizing the sites available to the custom searches. I ultimately decided that I needed the power and flexibility of a self-hosted XML file of annotations that would contain the URL’s of the sites to be searched. In addition, this seemed like a good project to start learning a programming language called Python.

I dusted off an old Delicious account, memento85, and added a few random bookmarks. I made a Python script that retrieved the most recent 100 bookmarks for a user as a JSON object and wrote the url’s from those bookmarks to a properly formatted annotations XML file. It took some trial and error, but Python was generally painless and you can see the script that I used here as a txt file. I then set it to run a few times an hour as a cron job on my server, and this made sure that my annotations XML file would be updated when my bookmarks changed. (Note that changes to the XML are not immediately reflected in the CSE, but this is ok – people can remember the sites that they’ve been to in the last few hours on their own).

Once that was working I set up a second CSE to use another Delicious account, lehrblogger, that had many more of my bookmarks imported. The annotations file made by bookmarksearch.py for this account looked like this. Adding this xml as the annotations feed in the CSE results in the following functional custom search engine – try it out below or go to the search’s homepage.

But why, exactly, is such a thing especially useful? Let’s say that I am looking for a specific site that I am sure that I had bookmarked a while ago and want to find again. I know it has something to do with SMS, but can’t be sure of any other keywords. If I do a search for ‘sms’ on my Delicious account, I get only one result. It is returned by the search because I tagged this result with ‘sms’, but perhaps it is not the site I was looking for and I am still certain that the other site is in my bookmarks. I could use the Custom Search Engine to search the full text of these same bookmarks, and this returns these four results, the one found by Delicious and three others. If I had been looking for, say, the first result, it would have been very difficult/tedious to find with only the tools offered by Delicious.

After that initial part of the project was both working and useful I started to think about ways in which it could be expanded. The Google CSE supports refinements, or categories of search results, which allow a user to quickly filter for results of a given topic. I thought there was a nice parallel between refinements and Delicious’ tags, and it seemed like a good next step to use the tags as refinements by pairing them in the annotations XML file with their respective URLs.

This feature also requires, however, that the main file that defines the CSE list all of the refinements. Google does provide an API for easily modifying this file, but a user must be authenticated with Google as the owner of CSE. I needed the updates to happen regularly as part of a cron job, and it would not work for each user to need to authenticate (i.e. type in her Google username and password) each time the CSE was updated. Even if I found a way to use authentication data as part of the cron job, I was concerned about storing that sort of sensitive information on my own server.

Thus it made sense to make a much larger jump forward in the project than I intended so early on: I decided to rebuild the application to run on Google’s App Engine. App Engine is a scalable hosting/infrastructure system on which to build rich web applications, and it offers substantial free bandwidth and CPU time as well as a competitively priced payment plan for larger/more popular applications.

App Engine uses Python and is well documented, so I dove in with the Hello World example. A good first step seemed to be to get the annotations XML file populated by the bookmarks returned by Delicious API call made by App Engine, and next I needed a way to serve that file at a persistent URL for the CSE to use. These things were more challenging than I expected – I had difficulty authenticating with Delicious, parsing the XML (as opposed to JSON) data that came back, and finding a way to serve those URLs as a properly formatted XML file. I initially looked for a way to write the URLs to a static file, but eventually found a detailed tutorial on writing blogging software for the App Engine, and I was able to adapt the RSS publishing portion of that example for my purposes.

The annotations XML files were now being published to URLs such as http://delv-icio-us.appspot.com/annotations/memento85 but currently it only works for that one user and you can actually put whatever you want after “annotations/”. Once I had App Engine making a call to the Delicious API and serving the resulting bookmark URLs in an annotations XML file, it was easy to set up a new custom search engine, this time for the handful of bookmarks of memento85.

Because the process of getting the above working involved so much trial-and-error, and because I intend to continue developing the project into a more complex application, I set up a GitHub project for the App Engine portion of Delvicious. There are many, many things left to do before the project is complete. I will need to:

• understand Google’s Datastore so that I can store information about the Google accounts of users and the Delicious accounts paired with them.
• also store the bookmark data for each user in the Datastore – too many API calls are required to fetch the entire list of bookmarks each time the XML is served, and it makes much more sense to store the bookmarks again and update them as new bookmarks are added. This also saves the need for the cron job – I can simply fetch new bookmarks whenever the CSE requests the annotations XML.
• design the various pages of the application, including signup and account management.
• develop the search page – ideally I can present the user with a single search box that will use both the built-in Delicious search and the Custom Search Engine and present the results side-by-side.

I’m excited about implementing these features and hope to continue this project for the remainder of the course. Delvicious will be a good opportunity to learn Python, familiarize myself with building web applications using the App Engine, and create a mashup that people might find truly useful.

This is my first post about David Nolen’s class Little Computers. We’re thinking about the iPhone as a computing device and learning to write applications for it.

I am going to build an iPhone application for users of Meetup.com for my primary project in Little Computers. Meetup is a pretty cool company – if you’re unfamiliar, it’s worth reading the following and checking out their site.

Meetup is the world’s largest network of local groups. Meetup makes it easy for anyone to organize a local group or find one of the thousands already meeting up face-to-face. More than 2,000 groups get together in local communities each day, each one with the goal of improving themselves or their communities.

Meetup’s mission is to revitalize local community and help people around the world self-organize. Meetup believes that people can change their personal world, or the whole world, by organizing themselves into groups that are powerful enough to make a difference.

I considered many alternative apps to build – xkcd, Diplomacy, a super simple app for exchanging contact info, a very complex app for music sharing, and others – but settled on this one. It seems like something that could be truly useful to a lot of people, doable within the course of the semester, and challenging and interesting to make. The slide deck and sketches I made to present the idea to the class is below. (Special thanks to my classmate Alex Kauffmann for the brilliant name – it’s much better than Unofficial Meetup.)