Oct 11

Do you want your service to keep state for you? Comparing the Twitter and Facebook APIs

Tech with tags: , , 6 Comments »

twitterfacebook

I love the Twitter for iPad application and how Loren et al have built an experience that feels right in your hands. There is one bug though that can drive me crazy, and that is the fact that the unread state of direct messages is broken. It feels like 99% of the time I will open up the application and the last 20 messages will have unread:true, even though I know I have read those messages. The bug is particularly irksome because it is so in your face. The messages tab is highlighted as though there are no items even though I know there is nothing new for me (and I just need to go through again and mark them all as read? ugh).

It made me wonder how Twitter was handling the state of direct messages. In creating Facebook for webOS we built out FB messaging, and the state was handled for us. The Facebook design has the notion of threading baked in (as apposed to email for example) and is as follows:

So, where is the unread flag kept? You may think that it is at the message level… after all, it is a message that is either read or not. However, the thread actually controls the flag for Facebook. This means that in a conversation there is basically an int count that shows you how many messages are unread. If zero, then all messages in the thread are read. There are trade offs on having the unread flag be a count and not a flag on the message layer, but the high level point is the core platform is keeping track of what a user has done. Whether you go through the website, or any application that speaks the Facebook API, as long as your application doesn’t cache too aggressively, you will have a good result. The key becomes: as the platform developer you very much understand that the service is the master, and you need to be smart about caching data to give your users a responsive interface, but make sure to sync on the data to make sure it is accurate.

echofon-sync

The Twitter API on the other hand doesn’t appear to keep any unread state for messages. Echofon is a cross platform client that showcases its support for syncing this kind of data. It is a nice feature for them to be able to differentiate on, but man….. it sucks that they have to write it!

I hope that Twitter starts to own the state so we aren’t in the current situation of no one really knowing my state, and it has been interesting to see the different state of affairs, and have you thinking about what the platform should be doing, and what should be left to the clients of said platform.

Dec 31

Not just social history, actual information from Twitter

Security, Tech with tags: 12 Comments » 
<html>
<head>
<script>
 
document.write("This page should show you your twitter info if you're logged in. (If you see a login box make sure you're logged into Twitter)<br/><br/>");
 
// forgive the document.write ugliness
function orly(data)
{
document.write("Your username is "+data[0]['user']['screen_name']+"<br>");
document.write("Your real name is "+data[0]['user']['name']);
 
}
</script>
<script src="http://twitter.com/statuses/user_timeline.json?count=1&callback=orly"></script>
 
</head>
<body>
 
</body>
</html>

What is that is all it took to grab your username and even real name out of Twitter? Try it, it works.

Yowser?!

(via Bill Zeller)

Oct 30

Using FriendFeed for it all, broadcasting to Twitter for the users

Tech with tags: , No Comments »

FriendFeed is marching on. I prefer it to Twitter these days, but of course the users are still mainly on Twitter. The best of both worlds for me is to run FriendFeed as my “client” yet have Twitter users happily following and not knowing or caring that, that is the case.

We may have the pieces to actually do this right now:

Posting to Twitter

FriendFeed added support for posting activities to Twitter for you. People immediately worry about spamming Twitter with this stuff, so you should think about that and turn off a bunch of services. I currently use TwitterFeed, which I will turn off if I use this feature (as FriendFeed will handle that as I turn on my blog service).

Seeing your @replies

FriendFeed is one of the few companies who have access to the XMPP firehose. They should be able to grok the replies (e.g. @dalmaer) and merge those into the comment stream.

In the meantime, DeWitt has a great hack:

I created an imaginary friend on FriendFeed and subscribed it to http://search.twitter.com/search.atom?q=%40dewitt

I did this, registering “To Dion” as the friend. It is ugly as it doesn’t merge into your flow, you can’t easily tell who has replied, and it is one big link… but it is good enough.

Friend / Follower Syncing

The people I follow on FriendFeed and Twitter isn’t in sync, and it should be. I haven’t caught up on FriendFeed. The beauty of the imaginary friends feature is that you can add Twitter folks even if they don’t have FriendFeed accounts yet. Ideally, I could tell FriendFeed to do the syncing and followee management for me.

For now though, I could try this sync script and see if it does the trick.

Good times for FriendFeed. I also think that it shows how great their real-time Web experience it is, auto updating in front of my eyes. If they implement these features natively, I will be a total convert.

Oct 01

Small things; How Facebook and Twitter share new features

Tech with tags: , , 5 Comments »

This is incredibly anal, but I appreciated it. When Facebook launched their new look and feel, it started out on a beta site, and then migrated to become the default. For what feels like a looooong time, I have seen this header:

Facebook New

The problem is that I get it. I know now. Let me get rid of the header please!

Twitter on the other hand did a really good job with this. When they launched the election subsite I started to see this:

Twitter Election Header

Once I internalized that I could click on the ” X ” close icon and it would minimize to this:

Twitter Election Minimized

Minimized allows me to get back to it if I want, but without taking much real estate. There have been some issues where it doesn’t seem to remember that setting, but in general I like this kind of implementation. On the right hand side they have a little box for their new look and feel that says “Notice something different? Read about the changes.” I wish that had an ” X ” too, without minimize, as again… I know it is new, and I don’t need to read about it anymore :)

Jul 11

FriendFeed vibe compared to Twitter

Tech with tags: , 5 Comments »

First, I apologize for writing a post with a title that has both FriendFeed and Twitter in it. I understand that most people will hit delete as soon as they can as they are sick of hearing about them.

However, I feel like writing this so I will.

I have dabbled in FriendFeed, but although I thought I should really like it, I never got into it and stayed in Twitter land. It took me awhile to really get the right feel for Twitter though, so I decided to really try to spend time with FriendFeed this week and see how I feel at the end of it.

I think I found the zen a little, but there are good and bad things:

Participation

Once you participate in conversations on the items you see how much nicer it is compared to @replies thrown this way and that in your stream. This is a little like going to conversations in Gmail giving you real threading in email instead of random replies.

One side effect is that you don’t do as many posts yourself, and do more comments on other peoples.

The down side though, is the way the river flows through. With Twitter it is easy, it flows on up. Here though, again a little like email, threads jump around as people comment. I find this pretty weird, as suddenly an item from 4 months ago shows up just because some random bloke “liked” it.

I do find that I actually click on “like” here though, whereas I never “favourite” on Twitter. I don’t know why. I think it is because you actually SEE when people like something so you are motivated to play the game.

Go Away A-List!

One nice viral (organic @kmarks?) feature is that they show you items one friend away (friend of a friend). This seems great, and can make a nice way to find people that you would like to follow. Seeing their stuff allows you to judge that, whereas “find friends” doesn’t.

The downside here though is that I can’t get away from Robert Scoble. No offense to Robert. The problem is that even after I unsubscribed to Robert because most of the posts had something to do about him…. I still get his posts because of this:

FriendFeed Scoble

Since so many people connect to him and other A-lists, chances are that some of my friends are in that chain too. Argh! There doesn’t seem to be a way to explicitly mute these, and using ‘hide’ a lot doesn’t seem to tell the engine to do it itself. “Oh, wait, Dion has hidden this guys stuff a lot so lets lay off.” The net effect is that the A-list folk have way too much of the feed.

Another couple of random things. For some reason I couldn’t get Flickr to work as a service. The interface asks for your nickname or email, so I tried both: my Flickr username (dalmaer), and my email, both personal and the Yahoo! one. No cigar. Finally, I went over to Flickr and duh, the email was [email protected] in my settings. I plug that in, and FriendFeed found it.

Finally, I am still looking for a bit more. I still want The River to be a tool that I live in, instead of something that sits over there. I hope the good folk at FF push ahead :) I do find that I am very rarely in Facebook now. My social group isn’t really there anymore, and my Dad is on there a lot more than I. The new iPhone app is awesome though, and could become your contact list.

Thanks for putting up with me for this post. I keep thinking that I sound like Steve Gilmor going on about @replies and the power of track.

Jun 22

Endpoint Resolver: Getting tinyurl out of the Twitter stream

JavaScript, Tech, Web Services with tags: , , 11 Comments »

Sometimes you can get in the zone just enough to be productive on a plane. On my flight to Mexico City yesterday, I created Endpoint a project that contains a server proxy, JavaScript client, and Greasemonkey Script with a mission. The mission is to take a URL, work out if it is a redirect (via a Location: header), and then return the final endpoint for it.

Why did I do this?

I was brainstorming functionality for a Twitter client with James Strachan (he is working on gtwit) and we talked about how annoying tinyurl / is.gd / snurl / you name it URLs are. They don’t tell you where you are going, and you could get Rick Rolled (if you are lucky) or much much worse.

So, I wanted to create a library, and one client (Greasemonkey) to test it out. Then anyone else could use it too to resolve directly from their Web pages.

How does it work

You load up the JavaScript via script src and then you can call resolve, passing the URL and a callback that will get the result. A few examples:

// Simple version
Endpoint.resolve('http://snurl.com/2luj3', function(url) { 
  alert(url); 
});
 
// Using the original URL to work out if it has changed
Endpoint.resolve(
  document.getElementById('testurl').value, 
  function(url, orig) { 
    alert(url); 
    alert(Endpoint.isRedirecting(url, orig));
  }
);
 
// How it is used in the Twitter Endpoint Resolver
Endpoint.resolve(url, function(resulturl, originalurl) {
  if (!Endpoint.isRedirecting(resulturl, originalurl)) return;
 
  newtext = newtext.replace(originalurl, resulturl, "g");
  jQuery(el).html(newtext);
});

Under the hood, a bunch of stuff is happening. I would love to be able to just use XMLHttpRequest to dynamically hit the URL and look at the headers, but the same-origin policy stops me.

This is why I have the server proxy, which returns a JSONP callback.

When you call resolve(url, callback) the script tag is created on the fly and added to the DOM. The callback function is all handled to allow multiple calls, and then the chain unravels.

Here you can see it all in action, showing how my Twitter stream will go through and the URLs will dynamically change from their tinyurl versions to whereyouaregoing.com:

I wanted to use App Engine to host the server proxy, but unfortunately I can’t work out how to do that yet. You have access to the URLFetch API to access resources from App Engine. Unfortunately for me, one of the features is that it understands redirects and just goes on through to the full resource itself, with no way to get the endpoint from the headers in the response.

It was also interesting to read Steve Gilmor talk about these services all be it in a post that is hard to actually understand ;)

Also, Simon Willison just put up a simple service on App Engine, json-time, that “exposes Python’s pytz timezone library over JSON.” I think that we will see a lot of these types of mini-Web services hosted on App Engine. Taking Python utility and making services from its goodness is an obvious choice.

May 26

Twitter is getting a Mom

Tech with tags: , 8 Comments »

Twitter MOM

@al3x told us about the Twitter architecture, and what says it all is:

Twitter is, fundamentally, a messaging system. Twitter was not architected as a messaging system, however. For expediency’s sake, Twitter was built with technologies and practices that are more appropriate to a content management system. Over the last year and a half we’ve tried to make our system behave like a messaging system as much as possible, but that’s introduced a great deal of complexity and unpredictability. When we’re in crisis mode, adding more instrumentation to help us navigate the web of interdependencies in our current architecture is often our primary recourse. This is, clearly, not optimal.

Our direction going forward is to replace our existing system, component-by-component, with parts that are designed from the ground up to meet the requirements that have emerged as Twitter has grown.

Amid the huge number of “oh no twitter is down. make it faster!” posts, we have some good ones.

The answer isn’t “Use PHP” ;)

If I was Twitter I wouldn’t be looking for Erlang as the answer, but I would be interested in talking to Joe Armstrong. I wouldn’t jump to Java as the answer, but I would be reaching to talk to Cameron Purdy of Tangosol Coherence (now Oracle). These people have seen systems that make Twitter look like a toy in comparison, and it is the knowledge that is more valuable than any technology.

If you think about contorting a typical LAMP stack to run Twitter you quickly shudder. Having a database layer, even with master/slaves, is scary.

Twitter needs a Mom, and it looks like it is finally getting one. With true message oriented middleware, and money to get the systems they need, they should be fine. As Cedric says this isn’t an original problem.

The system of messages shouldn’t be living in bottleneck databases. Instead, they can be flowing through a river of distributed cache systems. The Jabber side of the house shouldn’t be able to “bring down” the entire website. The beauty of publish subscribe and messaging is that you can throttle things nicely. You shouldn’t be “running out of database connections.” You can tune the number of listeners for the “track” command for example, and if it is getting abused you limit its resources. Sure, this may mean that you get messages to people a little later, but who cares. If messages got a little slower would people even realise? Compare that to the birds lifting the animal to safety message.

In fact, if you think about systems such as the stock exchange. You will realise that you rarely get truly real-time access. Most of the time you are on a delay of some kind, and that is fine. Through the distributed caching architecture you can push out messages to other systems to do their work. One of these systems will be the website itself. Twitter.com is just another client to the messaging backbone. Even if the backbone is in trouble, the website can still show the current view of the world, and could even still batch up work to be done.

I was talking to another startup that is migrating away from a database backed system, and soon the entire real-time world will be in a huge distributed cache. I am sure that Twitter will be moving there too.

Currently, I still feed bad for the engineers. I have been there; The point where you are the limits of your current architecture and you know it can tank at any time. You are firefighting all day and night, and thus don’t even have much time to fix anything at all. It is hard work. It is tiring work. It is demoralizing work.

However, I know that Alex and the rest of the crew will pull through their current situation, which after all came about thanks to the amount of love that its users have for the service, and one day the new architecture will be there in a way where we will look back and remember the early days, where downtime was such an issue.

Thanks for all the hard work guys. I can’t wait to be tweeting on a fully loosely coupled architecture, talking to one of your Moms!

May 23

TwitterFone: Voice recognition can be dangerous here

Comic, Tech with tags: , , 1 Comment »

Twitter Phone

I like the idea of TwitterFone. It would be nice to be out and quickly call in a tweet. I got an invite, and tried it out this week, and then remembered that voice recognition is really hard. Doesn’t it always seem to be one of those technologies that is “10 years out.”

My test tweet occurred after Man U beat Chelsea for the Champions League trophy.

I said:

I’m really excited that Manchester United won again today but I can’t believe that it went to penalties.

and I saw the following in my twitter stream a few minutes later:

I’m really study the dimensions unit one again today but I can’t believe that it went to penalties.

To be fair, they did pretty darn well. It is hard to expect them to know something like Manchester United. Unfortunately, even though they did well, it would be hard for anyone to understand what I was saying.

I then thought back to Twitter Translate. In that case we are translating between languages, and it is also often “off”. However, you can normally work it out, and there is a key difference. I expect that if I translate something that it will be a bit off, and I will cope with that. However, something like TwitterFone is happening without the consumer knowing, and thus it comes across as gobbledy gook as they don’t have the context (other than seeing “twitterfone” if they look carefully).

It seems that there are many use cases where it is OK that translations of some kind are “good enough”, and others where it does matter. You could get into trouble too, such as the comic above which messes up the “d” for direct message and changes it slightly so you say something totally different.

You could go from: “I like yoghurt and peas” to: “It hurts when I pee”.

Then there is the 140 character issue. It is impossible to do the math as you are speaking a sentence, so you will often run on.

Nice idea, but we need the technology to get to the next level for this kind of input to be used in a case like this. Using it for GOOG-411? Perfect… you can keep saying the same thing again.

May 20

Using downtime for publicity

Comic, Tech with tags: 9 Comments »

Twitter downtime

Every other day you see a post like this or this.

I wonder if the downtime that Twitter has, may have been a good thing in some ways. It gets people riled up, and we all rant about how it is crucial that we have a Twitter that is up. But, it isn’t down enough to make people switch to Pownce, which doesn’t seem to go down (or at least you don’t hear about it) so it therefore doesn’t get in the news.

I wish we could have a parallel universe that could compare the growth of a service with zero downtime versus a touch of it, in this case.

Apr 28

Twitter Translate: Automatically convert tweets to your language

JavaScript, Tech with tags: , , , , 13 Comments »

Twitter Translate

I am having a lot of fun with the AJAX Language API. Last week I talked about the translation bookmarklet that lets you translate anything that you select in the browser.

This time, I whipped up Twitter Translate, which watches the tweets in the page, and if the content isn’t your native tongue automatically converts it and replaces it inline. It then adds a mini logo with “translated from …” which you can click on to see the original text:

Twitter Translate Example

It is probably easiest to quickly see it in action:

I think that I am so excited about this API as it is a vertical service that you can just use for free. Think back on how you would be able to integrate language into your applications in the past. You would either:

  • Have to work on the discipline yourself, which is crazy if it isn’t your core business
  • Find a vendor that has some product that you can use, all of which are very expensive.

Now, we can use this service for free, and the best part? It will keep getting better behind the scenes without us having to do a thing!

Previous Entries