Lehrblogger

previously: midterm progress, initial idea, Twitter/Scala experiments

For my final project I plan to integrate my previous work with NASA’s World Wind, Twitter, and Scala into more recent work with PHP and MySQL. I will continue to focus on Twitter ‘retweets‘ and will use a PHP script to search for recent retweets (using a variety of search terms – ‘RT’, ‘Retweet’ and ‘RTWT’ as well as ‘Retweeting”) and store them all in a MySQL database. I’ll use the database information and additional searches to link retweets to previous retweets that had the same content and to the original source tweet.

I will then use the twittervision API or the Twitter API to associate geographic data with each of those tweets, and I will filter the database to keep only linked chains of tweets for which all tweets have an associated location. These chains of tweets can then be further filtered by distance between Tweets, since larger distances will create more interesting visualizations. They can also be filtered by complexity of the chain – I have some expectation of seeing tree-like branching, in which multiple people retweet a single tweet, and then other people retweet those retweets.

I’ll then visualize these retweet trees on a globe. The globe will rotate itself such that the location of the first tweet is visible, that tweet can be marked with a dot and displayed with an annotation, and then a line can be drawn to the next level of tweets and the globe can rotate itself again to illustrate the progress of the idea. After the leaf nodes of the tree are drawn, the entire tweet can fade slightly (but still remain visible), and the process can repeat for another tree. I will use the database information to always have enough retweet trees to create an appealingly dense web around the world.

I also hope to create a web visualization, using a similar but simpler Flash globe visualization with the Poly9 FreeEarth.

The third and final component will be more interactive, and will be based on Twitter’s current feature of ‘nudging’ users who have not tweeted in the past 24 hours (and have expressed that they wish to receive the reminders). I will create a new Twitter account which interested users can follow, and it will watch their tweeting patterns to determine if that user has not posted a new tweet in a certain amount of time. The twitter account can then automatically suggest a much-retweeted tweet for that user to retweet as well. Thus that user can have an awareness of current popular ideas on Twitter, can participate in the viral spread of these ideas, and can interact with the trees of retweets as they grows.

I’ve figured several things out with World Wind that I was having trouble with in the previous version, but there aren’t enough updates for a new video. Current versions of the files, however, are below:

• WWJApplet.scala
• TweetHandler.scala
• RetweetPair.scala
• Retweet.scala
• Tweet.scala

I set up a local MySQL database to store the information about the keys, and I made two web pages to interact with that database. The first displays a table of the current contents of the database (currently, what text is associated with each key) and the second allows you to add, update, or delete database entries for the keys.

Much of the work involved was learning how to use PHP and MySQL , and the functionality I implemented is effectively just a tiny subset of what you might find in phpMyAdmin. It will be necessary later, however, for dynamically re-generating the HTML page for the actual keyboard site whenever I update the content; this will allow search engines to index the content in their crawls. In addition, I can also generate the images of the keys (based on their content) that will be used to create a smooth key resizing effect.

I’m not going to create a web-hosted database and upload the files and get everything working for this update, but the new files can be viewed on github: http://github.com/lehrblogger/keyboard-portfolio/

recently: current upload
previously: initial idea, implementation experiments, early progress

I’ve been having some issues with browsers other than Firefox, and I think most of them were stemming from my use of console.log for debugging in Firebug. David Nolen (the course instructor) posted a solution to this issue, and all past projects should now be working. I also fixed a few other Safari-specific issues in the keyboard project.

I decided to use Git, a popular version control system, for the continuation of the project. This will allow me to keep track of changes and progress in a sane and flexible way. I’m also going to use GitHub, a collaborative code hosting service, to make it easier for others to comment on and contribute to the project as it moves forward. You can visit the GitHub repository of the project here: http://github.com/lehrblogger/keyboard-portfolio.

I began to explore the powerful Flickr API, and plan to use it as the source of the photos for the portfolio. Provided that it proves to be fast enough, it should simplify image uploading, facilitate various file sizes, and create a unified place for comments. Image names and captions on Flickr will also be displayed on the website. Image categories and projects will be tracked using tags. (Each category and tag will probably have a ‘slug’ in the database (more on that later), and the tag should use that slug.) I’ve used ‘category2′, project1′ and ‘image1′ in this example, which just pulls a single image from Flickr using the API – http://lehrblogger.com/nyu/classes/fall08/ajax/hw6/part2.

I started to think in detail with David about how I was going to implement various parts of the site, and I set up Apache and a MySQL database on my local machine to facilitate testing. All of the page content will be managed through interaction with the database (possibly through a content management page). Saving this page will automatically generate new Javascript and HTML for search engines to crawl and index.

Each standard keyboard key will always have an entry in the database, and this won’t be changeable from the CMS (content management system). Each key will have one of four possible modes – category, project, blank, and hidden – and other available table entries will depend on which type of key. Most punctuation keys will be hidden, but I will provide support for them in case I need the space for additional content later. Blank keys will have no other data associated with them, and will likely be greyed out on the site. Project keys will have the most associated data – the project will need a title, descriptive text (with limited HTML formatting), and some number of associated Flickr photos. Category keys will have a category name but no other content.

Note that there will be no difference in how each of the latter two types of keys behave. At the second level of zoom, the focused key and the surrounding keys will all be visible. At the third level of zoom, only the primary key will be visible. At the fourth level of zoom, for viewing the images, the interaction is handled through the number keys (1,2,3,4,5,6,7,8,9,0 – for up to ten images), and these won’t do anything on keys without images. A user can press a project key from the top level, but that user would need to press it again to be able to read it.

The automatic generation of the HTML and Javascript will be a challenge to get working, but they will make it much easier to manage the site. It is likely that the best way to get the text resizing to work will be to replace that text with an identical-looking image, resize the image, and then replace it with text at the new zoom level (or not, for smaller zoom levels). Those images would need to be automatically generated from the text saved to the database via the CMS, and (with David’s guidance) I’ve started to look into using Cairo for doing this on a server.

That might be the trickiest part of this whole thing, but if everything works well together it will be really cool. I’m not quite sure that I’ve been able to fully convey with clarity my idea of melding content and interface, and I’m excited to have a working example.

For my midterm project, I started to combine the work with Twitter retweets from last class with a map/globe-based visualization. Since many Twitter users have geographic information associated with their accounts (made particularly easily accessible through the TwitterVision API), I wanted to explore mapping certain conversation patterns on Twitter to their actual locations. My intention was to show how retweets create links between different geographic locations over time, perhaps even forming chains or trees. I considered using a two dimensional map, but I thought the globe was much better for conveying the connectedness of the world.

NASA has made their World Wind globe available as an open source project, and it functions much like the more common Google Earth. After a fair amount of struggle (and assistance from Jorge), I got it to work with Scala in Eclipse. I then integrated my Jumping Tweets code from the previous week, and was ultimately successful in displaying these as Annotations at (random) locations on the map, with Polylines between a specific location (arbitrarily set to NYC) and each tweet’s location. For next week, I hope to get the actual location data, improve the line quality, and add other options.

Since I did not use Processing at all for this project (instead relying on World Wind for the graphical interface), I was not able to export a embeddable Java Applet. Instead, a video I screen-captured is embedded below. I have also posted the Scala files, and feel free to contact me if you have any questions about how I got this working in Eclipse.

ICM Midterm – Jumping Tweets from me on Vimeo (you can watch a high definition version there)

(Sorry for the graphic strangeness at the beginning of the video – I think that happened during the import to FCP, and I’m not sure how to fix it (or if it’s worth the trouble).)

• WWJApplet.scala
• TweetHandler.scala
• Tweet.scala

Another post is coming soon about my plans for continuing this for my final project.

I made substantial progress on the main layout and animation for my keyboard portfolio. The keyboard is built dynamically from what is now an array of letters, and I used the MooTools tween/morphing functionality to handle the animation. Tweening of both the size and location of keys on the full keyboard at 24 fps seems to be more than my browser can handle, however, and I need to figure out a way to improve performance. There are three versions below – the first is without any tweening, the second is with tweening for only a part of the keyboard, and the third is with tweening for the whole keyboard.

In all versions, clicking on or typing a letter will zoom in on that letter, and spacebar zooms out keeping that letter in the upper left. I’m using Firefox 3.0.3 to view these, and I’m not sure if other browsers will work.

(It was much trickier than I expected to get the keys to change and move correctly, so I didn’t have time to modify the Flickr/GoogleMaps/YouTube/Twitter examples – I will later this week.)

HW6 Part 1 (no tweens): http://lehrblogger.com/nyu/classes/fall08/ajax/hw6/part1_notweens
HW6 Part 1 (partial keyboard): http://lehrblogger.com/nyu/classes/fall08/ajax/hw6/part1_partial
HW6 Part 1 (all): http://lehrblogger.com/nyu/classes/fall08/ajax/hw6/part1_all/
HW6 Part 2: http://lehrblogger.com/nyu/classes/fall08/ajax/hw6/part2/