Lehrblogger

project page with previous posts here

I resolved a couple of bugs in the line drawing animation, and now my database calls are made for each tree before it attempts to start drawing lines (allowing for a smoother animation). I also got the built-in View state iterator to work and make the globe rotate. Although much needs to be done before the project is ready to present, i think I’ve found solutions to the major challenges. A video is below:

TwiTerra Milestone from me on Vimeo.

I am hoping to do a number of things before presenting it in class next week, but for now I need to spend a few days working on my AJAX keyboard portfolio project. This list includes:

• the spinning of the globe should follow the animation of the tweet lines, rather than spin continuously (this should eliminate the unwanted pause between rotations)
• the lines between tweet coordinates are not interpolating correctly due to the way that latitude and longitude correspond to globe positions – this just needs a little bit of math to fix
• I need to show the text of the tweet, either as an Annotation object on the globe or somewhere else
• I want to filter so that I am showing the most interesting trees, based on tree complexity, the distance between tweets and the locations of the tweets

project page with previous posts here

I rewrote my Scala code from the Midterm to access a MySQL database for its tweet data instead of the Twitter search API’s XML feeds. I used Lift, the Scala web framework, to access the database – this tutorial and this one were helpful, but I would not have been able to get it working without Jorge Ortiz’s help. (Thanks!)

The current version is relatively rudimentary, and simply iterates through each of the root tweets in the database (those without a parent) and displays them. It uses the parent_id fields of the children to recursively build trees for each root, and then draws the lines corresponding to the locations of the tweets in that tree on the globe. I have functionality for filtering these trees by the minimum distance between tweets, but later I want to improve it to show only the most interesting/compelling tweets. I also need to figure out how to animate the globe to show the tweets in a smooth and appealing way.

The submission deadline for the ITP Winter Show is Monday December 1st, and I have re-posted the text of my submission over on TwiTerra’s project page – I’ll be likely be updating it as the show approaches.

previously: project proposal, midterm progress, initial idea, Twitter/Scala experiments

I’ve made substantial progress on my Twitter/globe project for ICM. I rewrote and improved the Twitter search code that fetched new retweets in PHP to work with a MySQL database on my Dreamhost web server. For each retweet found by that PHP script, it uses the @username syntax and text of the retweet to find the original retweet. If another retweet is found (i.e. there is a chain of retweets) it recurses back until it finds the original. If all of the tweets in the chain have valid location data, it stores each tweet with a unique ID (assigned by Twitter), the author’s username, the text of the tweet, the latitude/longitude, and the time.

It also stores the number of retweets that a tweet has – in a three tweet chain, the original tweet will have 2 retweets, the first of the two retweets will have 1 retweet, and the final retweet will have 0 retweets. It also stores the ID of the tweet’s parent tweet – in that same example, the original tweet will have no parent, the first retweet will have the ID of the original as its parent, and the second retweet will have the ID of the first retweet as its parent. This data will make it relatively simply to choose and display chains and trees of retweets on the globe by filtering for interesting structures and long distances between tweets.

You can see that PHP script here (sorry for the lack of comments, I’ll add them later), and it’s been running a few times an hour for the past several days. I have over 10,000 tweets in my database; about 5,500 of them are leaf nodes and have no retweets, 4,400 of them have one retweet, 500 have two retweets, 100 have three retweets, 30 have four retweets, and 25 have five or more retweets. I’m getting good data and will be able to draw interesting chains and trees on the globe, especially after it has been running for a few weeks.

As a side note, it’s been interesting to watch the sort of things that people are retweeting. For example, there was a flood of heavily retweeted content after the terrorist attacks in Mumbai. The three most heavily-retweeted tweets are (with 11, 14 and 11 retweets, respectively):

• chrisbrogan: 15 year old girl needs a kidney. She’s dying. Can you help? Can you at least retweet? http://tinyurl.com/6bue7f
• SantaClaus25: Please re-tweet and make history “Ho Ho Ho @Santaclaus25 and all of us here at the North Pole wish your family a Happy Thanksgiving”
• zigzackly: #mumbai sortable list of injured/dead at http://tinyurl.com/5q23h6 – please retweet

In addition, I’ve finalized the name (formerly Retweet Tree) to TwiTerra. I registered the twiterra.com domain, and for now it currently redirects to a WordPress page for the project. A project description and one-line pitch are coming soon, since the submission deadline to be in the show is December 1st.

previously: midterm progress, initial idea, Twitter/Scala experiments

For my final project I plan to integrate my previous work with NASA’s World Wind, Twitter, and Scala into more recent work with PHP and MySQL. I will continue to focus on Twitter ‘retweets‘ and will use a PHP script to search for recent retweets (using a variety of search terms – ‘RT’, ‘Retweet’ and ‘RTWT’ as well as ‘Retweeting”) and store them all in a MySQL database. I’ll use the database information and additional searches to link retweets to previous retweets that had the same content and to the original source tweet.

I will then use the twittervision API or the Twitter API to associate geographic data with each of those tweets, and I will filter the database to keep only linked chains of tweets for which all tweets have an associated location. These chains of tweets can then be further filtered by distance between Tweets, since larger distances will create more interesting visualizations. They can also be filtered by complexity of the chain – I have some expectation of seeing tree-like branching, in which multiple people retweet a single tweet, and then other people retweet those retweets.

I’ll then visualize these retweet trees on a globe. The globe will rotate itself such that the location of the first tweet is visible, that tweet can be marked with a dot and displayed with an annotation, and then a line can be drawn to the next level of tweets and the globe can rotate itself again to illustrate the progress of the idea. After the leaf nodes of the tree are drawn, the entire tweet can fade slightly (but still remain visible), and the process can repeat for another tree. I will use the database information to always have enough retweet trees to create an appealingly dense web around the world.

I also hope to create a web visualization, using a similar but simpler Flash globe visualization with the Poly9 FreeEarth.

The third and final component will be more interactive, and will be based on Twitter’s current feature of ‘nudging’ users who have not tweeted in the past 24 hours (and have expressed that they wish to receive the reminders). I will create a new Twitter account which interested users can follow, and it will watch their tweeting patterns to determine if that user has not posted a new tweet in a certain amount of time. The twitter account can then automatically suggest a much-retweeted tweet for that user to retweet as well. Thus that user can have an awareness of current popular ideas on Twitter, can participate in the viral spread of these ideas, and can interact with the trees of retweets as they grows.

I’ve figured several things out with World Wind that I was having trouble with in the previous version, but there aren’t enough updates for a new video. Current versions of the files, however, are below:

• WWJApplet.scala
• TweetHandler.scala
• RetweetPair.scala
• Retweet.scala
• Tweet.scala

For my midterm project, I started to combine the work with Twitter retweets from last class with a map/globe-based visualization. Since many Twitter users have geographic information associated with their accounts (made particularly easily accessible through the TwitterVision API), I wanted to explore mapping certain conversation patterns on Twitter to their actual locations. My intention was to show how retweets create links between different geographic locations over time, perhaps even forming chains or trees. I considered using a two dimensional map, but I thought the globe was much better for conveying the connectedness of the world.

NASA has made their World Wind globe available as an open source project, and it functions much like the more common Google Earth. After a fair amount of struggle (and assistance from Jorge), I got it to work with Scala in Eclipse. I then integrated my Jumping Tweets code from the previous week, and was ultimately successful in displaying these as Annotations at (random) locations on the map, with Polylines between a specific location (arbitrarily set to NYC) and each tweet’s location. For next week, I hope to get the actual location data, improve the line quality, and add other options.

Since I did not use Processing at all for this project (instead relying on World Wind for the graphical interface), I was not able to export a embeddable Java Applet. Instead, a video I screen-captured is embedded below. I have also posted the Scala files, and feel free to contact me if you have any questions about how I got this working in Eclipse.

ICM Midterm – Jumping Tweets from me on Vimeo (you can watch a high definition version there)

(Sorry for the graphic strangeness at the beginning of the video – I think that happened during the import to FCP, and I’m not sure how to fix it (or if it’s worth the trouble).)

• WWJApplet.scala
• TweetHandler.scala
• Tweet.scala

Another post is coming soon about my plans for continuing this for my final project.