Jun 22

Endpoint Resolver: Getting tinyurl out of the Twitter stream

JavaScript, Tech, Web Services with tags: , , 11 Comments »

Sometimes you can get in the zone just enough to be productive on a plane. On my flight to Mexico City yesterday, I created Endpoint a project that contains a server proxy, JavaScript client, and Greasemonkey Script with a mission. The mission is to take a URL, work out if it is a redirect (via a Location: header), and then return the final endpoint for it.

Why did I do this?

I was brainstorming functionality for a Twitter client with James Strachan (he is working on gtwit) and we talked about how annoying tinyurl / is.gd / snurl / you name it URLs are. They don’t tell you where you are going, and you could get Rick Rolled (if you are lucky) or much much worse.

So, I wanted to create a library, and one client (Greasemonkey) to test it out. Then anyone else could use it too to resolve directly from their Web pages.

How does it work

You load up the JavaScript via script src and then you can call resolve, passing the URL and a callback that will get the result. A few examples:

// Simple version
Endpoint.resolve('http://snurl.com/2luj3', function(url) { 
// Using the original URL to work out if it has changed
  function(url, orig) { 
    alert(Endpoint.isRedirecting(url, orig));
// How it is used in the Twitter Endpoint Resolver
Endpoint.resolve(url, function(resulturl, originalurl) {
  if (!Endpoint.isRedirecting(resulturl, originalurl)) return;
  newtext = newtext.replace(originalurl, resulturl, "g");

Under the hood, a bunch of stuff is happening. I would love to be able to just use XMLHttpRequest to dynamically hit the URL and look at the headers, but the same-origin policy stops me.

This is why I have the server proxy, which returns a JSONP callback.

When you call resolve(url, callback) the script tag is created on the fly and added to the DOM. The callback function is all handled to allow multiple calls, and then the chain unravels.

Here you can see it all in action, showing how my Twitter stream will go through and the URLs will dynamically change from their tinyurl versions to whereyouaregoing.com:

I wanted to use App Engine to host the server proxy, but unfortunately I can’t work out how to do that yet. You have access to the URLFetch API to access resources from App Engine. Unfortunately for me, one of the features is that it understands redirects and just goes on through to the full resource itself, with no way to get the endpoint from the headers in the response.

It was also interesting to read Steve Gilmor talk about these services all be it in a post that is hard to actually understand ;)

Also, Simon Willison just put up a simple service on App Engine, json-time, that “exposes Python’s pytz timezone library over JSON.” I think that we will see a lot of these types of mini-Web services hosted on App Engine. Taking Python utility and making services from its goodness is an obvious choice.

May 13

WS-* is AOP

AOP, Tech, Web Services 3 Comments »

I had to have a little Friday chortle when reading that the WS-* stack are aspect oriented protocols.

Personally, I would agree that they are AO, but not AOP :)

May 05

WS-*: Where is the community?

Tech, Web Services 1 Comment »

Web services has been over-hyped for a long time. Book sales are poor. You don’t hear too many people talking about it other than “REST vs. WS-*” arguments.

I realised that there doesn’t seem to be a community here. I think this may be similar to there not really being a community that revolved around JRMP or IIOP itself?

People want to get together to talk about meaty stuff:

How can I get work done

Open source software often has great community, as the software that is successful normally comes out of pragmatic need. It also does well as the group often encourages participation, and you feel like you can make a difference. If you find a bug, you want to put it in JIRA as you know someone will be in there. How many times do you have to fight for this with larger commercial companies, who get to it after they finish up bugs that have come in from their largest clients (NOTE: lots of good commercial companies take care of their lil guys too).

The interesting thing to note is that Web services is happening. More and more projects have a need to use some form of them for inteop, or other reasons. However, it is often a fringe part of the project. “I need to do X, Y, Z, and need to expose a service A that can be consumed safely”.

My main participation with a WS community is finding bugs and issues where interop is a painful broken promise :)

It is interesting to compare this to the communities that jumped up when EJB was brought into life. A lot of developers were forced into that world, and quickly developed hacks (patterns) to make it usable. We haven’t really seen the same take off in the WS space.

With the EJB mess behind us, we are able to focus on writing good software, with real OO models, and yes with services of some kind.