Lehrblogger

A WDYG post at last!

Where Do You Go provides Foursquare users with a dynamic heat map of the places they have visited on top of a standard Google Maps interface. Users can create snapshots of their maps and hotlink them as static URLs on their personal webpages, or they can use the simple WDYG wrapper pages to share their maps on Twitter. The maps will self-update automatically in the background as users continue to visit new places and checkin with Foursquare.

The idea initially came out of the difficulty I experienced in explaining to people the areas I tend to frequent in this expansive metropolis – “south of 14th Street and north of Delancey” is somewhat accurate, but really misses many of the nuances in my habits. It would have been relatively straightforward to make a static image of just my own checkins, but I wanted something that updated over time as I went to more places, and I wanted something that was also usable by other people.

WDYG was a project for the Mashups class I took last Fall, and I launched it on December 18th, just before I exhibited it at ITP’s Winter Show. Many people used the built-in “tweet this” button to share their maps with their friends, and then those friends saw those maps and wanted to create and share their own maps, and so on… this viral distribution contributed to much of the non-trivial amount of traffic the site has gotten, but I also got some decent press.

Below are two maps I made of my checkins in Manhattan and one from my trip last summer to Amsterdam (which was coincidentally one of the early Foursquare cities, although now you can check in anywhere), and there’s a fourth heat map embedded in the sidebar. The heat maps will update in the background as I check into new places, but it will skip checkins from the past 12 hours to protect my privacy, and the coloring is sufficiently vague as to make it ambiguous which places it is to which I am actually going.

Users are given a variety of options for customizing their maps before sharing them: they can adjust their maps to make them ‘hotter’ or ‘colder’ if there is either too little or too much coloring, they can easily jump to another city, they can select from a variety of color schemes, they can adjust the size of the map image, and they can select from one of several zoom levels. A user can then create a ’snapshot’ of his/her customized dynamic JavaScript Google maps that will have a static URL, and the user can then embed that .png elsewhere on the Internet or tweet the corresponding wrapper page (that has a conveniently auto-shortened bit.ly URL).

The Python code for the entire project is available on GitHub and the application runs on Google App Engine, a framework that abstracts away challenges related to server administration and scaling (i.e. worrying about getting and configuring more computers to run your application when you suddenly get a lot of traffic). It uses Foursquare’s API and John Wiseman’s wrapper library for it as well as the bit.ly API, the Google Maps API, jQuery, and Blueprint CSS.

The heat maps are created using a heavily modified version of the gheat-ae Google Code project, which is a port of the gheat Google Code project, without which I wouldn’t have known where to start (although it was mostly non-functional and un-documented when I found it). There are a few interesting things I’m doing to make the heat maps consistently attractive, so I’ll dive into the technical details a little.

Google serves their maps in pre-generated 256×256 pixel tiles, and they provide functionality in their JavaScript API for the client to overlay custom tiles on top of their map tiles. To display each overlay tile I query the App Engine datastore for the venues within that tile (and just outside of it, at a distance based on the zoom level, so that I get coloring from venues that are not within the tile but still bleed into it). Checkins are stored with the venue at which they occurred, so each venue has a latitude and a longitude and a number of checkins that the user has there (if two users check in to the same place, they each get a separate copy of the venue in the datastore). A heat map tile to be displayed on top of a Google maps tile might look like this:

The checkins are plotted on the map conceptually as dots that are darkest in the center and fade out towards the edges, and they’re drawn mostly independently of each other (i.e. I’m not doing any expensive distance calculations between the venues). To draw the tiles the application adds all the geo-located dots together to get a ‘darkness’ value for each of the pixels in the 256×256 pixel tile. I wanted a smooth gradient from the center to the edge of the dots, even if there were lots checkins at the same venue. I used the OS X application Grapher to experiment with functions that might create the desired effect and minimize hard edges of the stacked dots. I went with something similar to the highlighted equation below and implemented it in the calc_point function in the tile generation code – here x is the distance from the center of the pixel and y is the how dark that point will be when mapped to a pixel.

Next I needed to specify the actual color chosen from the color schemes for each pixel in the tile. The pixels correspond to a color that is a certain height up or down one of the below color scheme images (the cyan-red color scheme isn’t generated from an image, but we won’t worry about that here). Both the range and rate-of-change of the colors affects the appearance of the maps.

As described above, I needed to map the array of ‘darkness’ values for each of the 256×256 pixels in a tile to a height up or down one of these color schemes. I wanted the ‘darkest’ point on any map to correspond to the ‘hottest’ point in the color scheme (at the top of the image), and I estimated what that maximum level should be (at which the color scheme becomes ‘over-heated’ or ‘blown out’ or ‘over-exposed’) by calculating the total number of checkins and the total number of venues currently visible on the entire map. Then I needed a way to scale each darkness value so that it would always show some color for a checkin (in case a user had only one checkin somewhere away from his/her other checkins) but would be slow to reach the maximum level (i.e. the ‘hot’ end of the color scheme). I experimented with log functions in Grapher and settled on the formula in the scale_value function in that same tile generation code – here x is the value of the pixel resulting from the stacked dots and y is the level which will be mapped to a point on the selected color scheme.

Please let me know if you are interested in explanations of other aspects of the tile generation or the code in general – I think the above tricks are particularly cool, but there’s lots of interesting stuff going on within the application that I’d love to talk about.

And last but not least, the inspiration for the name of the project (finding a good name is, of course, always the hardest part of anything) – thanks to Jorge for the suggestion!

http://wanderli.st – wander the internet, bring your friends

I have 787 friends on Facebook. On Twitter I am following 216 people and am being followed be 285 people. I have 1,190 cards in Address Book, all of which are synced with my Google Contacts and my iPhone. I have 49 friends on Foursquare, 33 connections on LinkedIn, 18 friends on Goodreads, 8 contacts on Flickr, 1 contact on Vimeo, 0 friends on Yelp, and 146 buddies on AOL Instant Messenger.

If I want to sign up for some new website, it’s not at all easy to re-use these existing relationships: I can go through and add people individually; I can ignore the security risk, enter my Gmail login information, and selectively choose which (or all) of my 1,190 Google Contacts need an email invitation to the website; I might be able to connect with my Twitter account, but the nature of the information shared on Twitter results in the people I’m following being a strange subset of my social graph; I might be able to connect with my Facebook account, but I rarely want to publish a summary of my activity on the new site in the news feed of every single person I know on Facebook.

My social life on the Internet is somewhat of a mess, and it’s becoming increasingly unmanaged and unmanageable. Social networking websites are not going away, and I want better tools to consolidate and manage these myriad representations of my real-world relationships as I wander the Internet…

(photo of me at the xkcd book party by insunlight on Flickr | CC BY-NC 2.0)

Wanderli.st will be my attempt to solve this problem. I want to take my existing friendships and relationships with me wherever I go on the Internet. I want more powerful tools for managing my contacts, I want this private information to sync with a giant social graph in the cloud, and I want websites to access subsections of this social graph based on the permissions I grant them.

More specifically, I want Wanderli.st to help me organize everyone I’ve ever met using a simple system of custom tags (‘ITP classmates’, ‘bit.ly coworkers’, ‘Scala programmers’, ‘SXSW 2010′, etc.) and lists that are combinations of these tags (‘all of my family and photography friends, but none of my ex-girlfriends’), and then let me use those lists to automatically specify my relationships on social websites. I want an intuitive yet powerful address book application with standard fields for phone numbers and mailing addresses but also with dynamic fields for usernames on social websites. I don’t want Wanderli.st to bother with actual content – let other websites specialize in the sharing of photographs, videos, status updates, long blog posts, short blog posts, and restaurant reviews – Wanderli.st can simply be a social graph provider.

I want my social data to be device- and website-independent, and I want to be able to export all of it to a standardized XML file. But I also don’t want to worry constantly about importing and exporting, and instead I want to be able to make one change in one place whenever I make a new friend, and I want that change to be pushed automatically to all of the applicable social networks.

I’d like to be able to sign up for a new social photography website, assign that site a list (i.e. some combination of tags), and then have the option of inviting friends to that website based on some other combination of tags (perhaps I have a tag named ‘people it is okay to invite to random websites’). If I make a new friend who is interested in photography, then I want it to be sufficient for us to only have a) exchanged email addresses and b) associated our usernames on various photography websites with our Wanderli.st accounts – it will then seamlessly create our connection on those websites and automatically add those usernames to each other’s personal address books, with no “Steven has added you as a friend on Flickr” emails required.

I want to have the option of managing my privacy simply and intuitively at the level of the website, and not at the level of the individual piece of content: you can see the pictures of me drinking in college if we are friends on the site on which they are posted, but if I don’t want you to see them then I simply won’t be your friend on that site, and I can use a second site (or second account on that same site!) to share my other pictures with you.

Wanderli.st will also make it easier for me to move among social websites. Both established and fledgling websites will benefit from this because it will be easier for them acquire new users and provide existing users with the best possible social experience. Furthermore, there have been mass diasporas of users in the past as people have moved on from Friendster and MySpace, and I predict Facebook faces a similar future (more on this in a future blog post). I’m willing to re-create my social network only one more time after I’m ready to move on from where I am now (and Facebook still won’t let me export my data), but after that I want my data to be open and portable and mine so that I never have to re-friend a thousand people again.

I also intend to make Wanderli.st my ITP thesis. I have been thinking about the project for several months, and wrote up and presented an early draft of the idea in Kio Stark’s When Strangers Meet class last Spring. I think that Wanderli.st should be compatible and complimentary with existing standards and upcoming proposals (OpenSocial, Portable Contacts, WebFinger, etc.), but I think it is important that the project be a new site in and of itself that hosts the data and popularizes the platform through actual successful use cases.

I’ve read that the best software is made by people who are building the tools for themselves, and I’m excited to create Wanderli.st and improve how I socialize on the Internet. If what I’ve described here sounds like something you’d like to use as well, comment below or enter your email address at http://wanderli.st – I’ll let you know when it’s ready for beta testing.

http://webideas.uservoice.com

Nearly a year ago I posted about this blog’s “web idea” category, and wrote:

During conversations with friends, I regularly have ideas for websites, services, or other technologies. These conversations happen in person, over email, on instant messenger, via text message.

[...]

Ideas, I have found, are relatively commonplace; the real work is in their execution. Sometimes people guard their ideas, keeping them secret out of fear that they will be taken and implemented by someone else first. None of that here. Please, take these ideas, and please, bring them to life. I have more of them than I’ll ever have time for, and even if I were to eventually have time for all of them, they would have long-since lost their relevance. If you find yourself with sufficient knowledge, time, and interest to start work on one of these, I would love to talk about it and hear what you are thinking about what I was thinking.

I’ve continued to have these ideas, and I scrawled some of them in my notebook, left some of them in instant messenger conversation logs for later searching, and saved many of them in Things. I’ve been dissatisfied with how private all of these storage media are, and have been inspired by Alex Payne’s posts on http://ideas.al3x.net/. I’ve considered doing something similar, but I’ve worried that the formality of the text boxes on a proper blog of any sort (whether it’s here or on Posterous or Tumblr) would discourage me from regular and casual posting.

UserVoice to the rescue! Built as a robust user-feedback tool for websites, I realized that I could re-purpose it as a tool for organizing these ideas in a public and collaborative way. Anyone can post ideas, anyone can comment on them, and anyone can allocate one of their limited number of votes to indicate that they like an idea. I hope that this will provide a mechanism for bringing the best/most-desired ideas to the top of the list and act as a useful metric for prioritizing projects.

(Note that there are several other sites that do this sort of thing, but the others are either not free to use or do not have the same collaborative vibe. Full disclosure: betaworks has invested in UserVoice as well as in bit.ly, the startup for which I work.)

There isn’t much on http://webideas.uservoice.com yet, but I’ll be adding more as I migrate over my old ideas and come up with new ones. Feel free to contribute, and please take one and build something!

The Internet, when I first started using it, felt like a social desert. People had email, instant messaging, chat rooms, and discussion forums, but these channels for communication felt separate from everything else and more like alternatives to existing real-world channels. Communication was isolated from destinations for the consumption of content, from tools for the creation of content, and from platforms for the publishing of content.

Eventually, sites for the social sharing of content appeared, and each of these maintained separate representations of the social graph. Over time people collected contacts on Flickr, friends on Facebook, and followers on Twitter, and such sites became oases of social functionality. At first these patches of green built walls to keep out the marauding hordes of anonymity, but as they grew larger they also grew more open, and they started to trade content amongst themselves.

Each real-world individual, however, was forced to maintain a separate existence in each of them simultaneously. It was difficult for people to travel with various aspects of their digital identities between walled oases, and it was nearly impossible for them to take their friends with them when they did. As a result, people were forced to duplicate their selves and their relationships. Some walled gardens tried to build roads to connect with (and undermine!) the others, but nothing really improved. Everyone had to maintain a copy of themselves in each oasis in which they wanted to produce and share content, and life was a mess for everyone. It was time to build something new, a sort of subway under the blossoming desert, so that every aspect of every person could be wherever it was appropriate for it to be, all the time, all at the same time…

- – - – -

The Internet has outgrown its social infrastructure. It’s becoming increasingly inconvenient and infeasible to create and maintain multiple copies of our networks, with all of their social complexities (friends, family, coworkers, acquaintances, ex-girlfriends, etc), and with all of their nuances of interaction (friend requests, @replies, emails, wall posts, blog comments, etc). Tools such as Facebook Connect are hopelessly hindered by over-saturated social graphs, pre-existing notions of privacy, and misguided attempts to pull content back into single cluttered interfaces. Identity and content aggregators such as Chi.mp, Plaxo and Friendfeed don’t provide the tools for web-wide social graph management. Put simply, we need new tools for the modern social Internet.

What will they be like?
Who will build them?

This post develops the conceptual language surrounding the product ideas in my previous post, Filtration as a Tonic for Internet Vertigo.

Towards the beginning of his aforementioned blog post, John Borthwick writes:

what emerges out of this is a new metaphor — think streams vs. pages

I think that this progression of metaphors is moving in the right direction, but it needs to be taken further. Streams consist of web content that is delivered directly to the user (to a Twitter client, to an RSS reader, etc), and this is in direct contrast to content that lives on specific pages to which a user must navigate in a web browser. Streams are dynamic, up-to-date and are delivered in (near) real time, while pages are static and not necessarily current.

Streams, however, are just collections of individual pieces of content, or packets. Tweets, status updates, blog posts, photos, mp3 files, and video clips are all discrete packets of content. These packets are the units which a user actually consumes as information, and streams are just a way to group those packets over time, usually based on on source (such as a specific blog) or topic (such as a search term on Twitter). But there are potentially more potent ways in which these packets can be organized than by their original source/topic, and this is important because these streams tend to be overwhelming in their aggregate. Borthwick continues about the future of content delivery via streams:

Overload isn’t a problem anymore since we have no choice but to acknowledge that we cant wade through all this information. This isn’t an inbox we have to empty, or a page we have to get to the bottom of — it’s a flow of data that we can dip into at will but we cant attempt to gain an all encompassing view of it.

I suspect that there is a more optimistic solution, however, and that there are better-than-random ways to organize the flow of content from our collections of streams. There will be some packets in these streams that are more important to individual users than others, so I want services that surface the best ones and hide the others. I predict a future in which streams are cut, rearranged, reordered and remixed into a single source of content that always has that moment’s most important/relevant/enjoyable packet at the front of the queue. The future of content on the web will be based on tools that focus on perfecting the delivery of these individual packets of information to users for consumption.

I agree with Borthwick, and think that the re-conceptualization of the destination web of pages into a real time stream of pages is the next (or current?) big thing. But I think the re-conceptualization of those streams as collections of individual operable packets is the big thing after that.