techno.blog("Dion")

Dealing with JavaScript scope issues; The tale of Alex kindly indulging me

JavaScript is about run-time. Run-time is great and all, but especially when dealing with the browser, and how your Web page has to bootstrap the entire world on every load, Ajax developers have to think about issues that other people don’t. Problems that others can compile away, or know that “that happens once when I start up the puppy on the server” are here for us to stay.

This often seems to mean that we have to deal with writing our applications in ways that aren’t as clean as we may like. We run across problems where for the Nth time someone was bitten as somewhere code did a “for in” that wasn’t guarded with hasOwnProperty and then someone throws up there arms. Never Again. Dojo does a lot of things out of this experience. It is out of real-world pain that choices were made in the toolkit. One of these is how they are very careful not to pollute the global namespace. This is great in that you don’t run into collisions, especially in a world where code is being sucked in from who knows where (e.g. some Ad code is sucking in things). As the author of some JavaScript code, you don’t actually know what else may get into your global area when running, so you need to guard against it.

The problem is that this means that you can lose out. Prototype feels so right to me in many ways as it is less of a “JavaScript library” than a “way in which JavaScript should have evolved”. We have seen some of its goodness get into ES 3.1 (e.g. bind()) but at that rate of progress we will get four more methods in 20 years ;)

"some content   ".trim(); // feels right
dojo.trim("some content   "); // doesn't feel right

I have to take a second to tease here. Did you notice that there are two trim methods in Dojo? dojo.trim and dojo.string.trim. This is a good example of the crazy things that we have to think about. It appears that dojo.trim is lean code, so it gets into base, but dojo.string.trim is more code, but it runs faster. Wow :)

Once you get a lot of code going, you suddenly realise that the word “dojo” appears 50 times per screen of code. It makes me want to create a Bespin plugin that change the shade of that one word. It feels like Java. Unlike Java, you can’t import away some of the pain. You can try to var foo = some.really.big.package; but that gets annoying quickly.

So, to the point of the blog post. When I saw Alex Russell at a nice wine bar in San Francisco last week, I told him how I would die for the ability to have the best of both worlds of Dojo and Prototype:

I want to be able to write code the way that feels right, without the verboseness, but ALSO not run into the scoping issues. I would even take a performance hit for this. What if I could have:

runSomeCodeWithMagic(function() {
    // in here I am in the lovely land where "some content   ".trim() works
    // and I can just forEach instead of dojo.forEach
    // but the outside world isn't affected
});

Well, Alex listened to me blather on, probably thinking I was a total idiot, and then went on to quickly indulge me by giving me a little of my wish by giving me dojo.runWith:

dojo._runWithObjs = [];
dojo.runWith = function(objs, func){
	if(!dojo.isFunction(func)){
		console.error("runWith must be passed a function to invoke!");
		return;
	}
	var rwo = dojo._runWithObjs;
	var iLength = rwo.length;
	// console.debug(func.toString());
	if(!dojo.isArray(objs)){
		objs = [ objs ];
	}
	var catchall = dojo.delegate(dojo.global);
	var fstr = [ "(", func.toString(), ")()" ];
	objs.unshift(catchall);
	var locals = {};
	objs.push(locals);
	objs = objs.reverse();
	dojo.forEach(objs, function(i){
		var idx = rwo.length;
		rwo.push(i);
		fstr.unshift("with(dojo._runWithObjs["+idx+"]){");
		fstr.push("}");
	});
	(new Function(fstr.join("")))();
	// allow us to GC objs passed as contexts, but don't rewind
	// further than we started (allowing nested calls)
	rwo.length = iLength;
	// TODO:
	//		iterate on locals and look for new properties that
	//		might have been assigned. Maybe give the with-caller a
	//		way to handle them or specify a policy like "make
	//		global"?
};

To use it I can do something like this:

<html>
    <head>
        <script src="http://o.aolcdn.com/dojo/1.3.0/dojo/dojo.xd.js"></script>
        <script src="runwith.js"></script>
        <script>
        dojo.addOnLoad(function() {
            dojo.runWith([ dojo ], function() {
                var sum = 0;
                forEach([1, 2, 3, 4], function(i) {
                    sum += i;
                });
                byId("result").value = "The sum is: " + sum;
            });
        });
        </script>
    </head>
    <body>
        <h1>Run with Wolves</h1>
 
        <input type="text" id="result">
    </body>
</html>

Very nice! Only a couple of dojo’s in sight!

Here are some of the thoughts from Alex himself:

Note/warning about the runtime cost: If your browser parses things fully in it’s JS engine up front, this function may hurt. A lot. It de-compiles a function using the toString() method, meaning that it does an uneval + eval + string concat + with() call. Each of these operations alone might be painful in a slow engine. Together they could be fatal. On the other hand, if you’re using these functions in, e.g., Chrome/V8, this could turn out to be relatively cheap, particularly as this is run-once kinda thing. The runtime cost involves namespace misses on locals, and that can be significant. I dunno. You’ll have to test to find out.

Note that you won’t easily be able to define globals from here by dropping a “var”. This might be a feature or a bug, depending on how you think about it.

Anyway, hope it’s useful. I’d imagine that you’d structure your files like this:

// something.js
dojo.provide("thinger.something");
dojo.require("thinger.blah");
 
dojo.runWith([ dojo, thinger ], function(){
       ...
});

What about getting the magic on the core objects? Again, only within the magically land of that scope do we want String to have the trim method…. and code outside of it shouldn’t see it. Can we swap onto the objects and their prototypes? Alex has some thoughts here too:

I think I can proxy intrinsics at some additional cost. I’d like to make it a protocol, though, so that you might be able to have a list or function that handles your extensions. E.g.:

var contextObj = {
    ":intrinsics": {
        "String": { ... },
        "Number": { ... },
            // ...
    }
};
 
dojo.runWith([ contextObj, ... ], ... );

This would give us a way to un-install them later, but I don’t have a solution for aync code as we do with with(){ … } which actually binds definitions at declaration time.

I might try to proxy intrinsic prototypes somehow, but I need to spend more time thinking about how to get that to work well. I’d like these things not to place big constraints on how you think about or use this system.

Wicked. This also ties into Pete Higgins and some of the very interesting work he is doing in similar but different veins with his dojotype and plugd which munge Dojo into interesting forms.

Since Dojo has pretty much everything that any other library has, you can start thinking about how you can bend it to look and feel like others, take on their APIs when it makes sense, and bend it to your whim.

With research like Alex’s we could see an interesting view when you can create worlds which don’t affect each other, let you have a view that makes sense for your code, but doesn’t affect others.

Even if you poo-poo some of it for the performance aspects, this is why it is incredibly exciting to see the latest JavaScript Vm work. With these guys running, they can optimize a lot of this a way, and things that used to be bottlenecks in the code will cease to be.

Thanks to Alex and Pete for indulging me, and taking the time to listen and produce really interesting solutions!

Updated: Using the Dojo Loader

James Burke has a very cool follow up on how to give a solution to this kind of problem using the Dojo Loader itself:

Setup your locals

dojo.provide("coolio.locals");
 
dojo.setLocalVars("coolio", {
 trim: "dojo.hitch(dojo, 'trim')",
 $: "dojo.hitch(dojo, 'query')",
 id: "dojo.hitch(dojo, 'byId')"
});

Now setup your actions that will use the locals:

dojo.provide("coolio.actions");
 
coolio.actions = {
 init: function(){
  $("#trimButton").onclick(coolio.actions, function(evt){
   id("trimOutput").value = trim(id("trimOutput").value);
  });
 }
}

Finally, use HTML to auto load:

<script type="text/javascript" src="dojo/dojo.js" djConfig="require: ['coolio.locals']"><script>

In former times (I love how Germans say that) I spent time working on event driven systems and busses. Those unfortunate enough to work with Enterprise JavaBeans were happy when Message Driven beans were announced as they could tell the pointy hair chaps that they were indeed doing EJB, and would continue working in their MOM architecture.

Now I fast forward to my days with JavaScript, and I remember getting excited to see the common JavaScript libraries support custom events. I now had a very nice way to work with my library to use event driven architecture on the client side, which is actually quite natural to do too. We see this pattern in desktop APIs such as Cocoa, and it helps enforce the practice of responsive applications for end users.

I may have taken this notion to the extreme a little in Bespin.

We have a general event system to put general work and you will also find that most of the components of Bespin also have their own events.js. A component will often load up the events by doing something like this:

// from within the editor itself, pass into the event system so the events are in scope
new bespin.editor.Events(this);

The event class will take in that component, put it in scope, and then an event can just use it. Note the use of editor here:

dojo.declare("bespin.editor.Events", null, {
    constructor: function(editor) {
        bespin.subscribe("editor:openfile:opensuccess", function(event) {
            editor.model.insertDocument(event.file.content);
            editor.cursorManager.moveCursor({ row: 0, col: 0 });
        });
    }
 
    // ... all of the events
});

As new events come into play for the editor, this is where they will naturally be placed.

There have been some really nice fall outs of this approach. It has been really nice to have the notion of “this just happened, if anyone is interested, here is some info.” There is no need to find an object and call a method on it…. the event bus just takes care of it all for you.

There has been a nice side effect in the case of sharing commands across different parts of the application. Take the dashboard vs. the editor itself. Both have a command called newfile that looks a bit like this:

bespin.cmd.commands.add({
    name: 'newfile',
    takes: ['filename', 'project'],
    preview: 'create a new buffer for file',
    completeText: 'optionally, name the new filename first, and then the name of the project second',
    withKey: "CTRL SHIFT N",
    execute: function(self, args) {
        if (args.filename) {
            args.newfilename = args.filename;
            delete args.filename;
        }
        bespin.publish("editor:newfile", args || {});
    }
});

The core piece is that the command basically just sends an event of type editor:newfile. On the other side of the coin, who is listening to this event? Well, in the case of the dashboard, it changes the URL to the editor itself, but if you are already in the editor itself, you don’t need to do that, and can just clear the editor buffer and file state.

Having commands at this high level means that they are easy to share.

We do need to do some more thinking about when it makes sense to promote something to be a full fledged event. In general I like the idea that people can write plugins and extension code that only requires them to grok some events and they are in and out in a jiffy.

However, some team members have brought me down to Earth and I realize that if we go too far here we will be reinventing method dispatch itself, and in a way that is only asynchronous :)

We don’t want to go that far. The no-brainer use cases are for typical call backs such as “a file has opened, care?” Something that an event asking for something to be done (e.g. newfile above) is a more subtle item and shouldn’t be over used.

Another side effect from using this model is that I remember exactly how much I like named parameters, which is why you will see the pattern is publish(eventName, argumentsHash). Dojo allows you to pass in multiple arguments, and in fact makes it harder to only send in one object as you have to dojo.publish(eventName, [ one ]). The [ ... ] looked ugly, so we wrapped the mechanism as bespin.publish(...). We also did this because we wanted to be a layer above the implementation. The Prototype version used Element.observe() and Element.bind() and we had to change all of that work in the port. Hopefully we won’t have to do this if a major change happens in the future.

Fun times! I will also write-up the effects of this, and how extensibility kicks in with Bespin on commands, your configuration, and plugins.

Courtesy SierraBlair

I had a rather long post on the pain of dealing with the system clipboard in Web applications now that Flash went and fixed some security hole :/

The solution in place at that time worked well in Safari, with the on[cut|copy|paste] and unfortunately not great with Firefox due to one crucial big of data being omitted.

Then Tom Robinson and a couple of others made the great point that I could get ALMOST everything I wanted with just the hidden textaraea trick. Instead of tying the trick to the on[cut|copy|paste] events, I can just manually grab the Cmd/Ctrl-C X V commands. The only downside to this is that if the user goes to the Edit menu and chooses something, it won’t work. Annoying, but that’s the world of the hacky Web sometimes.

I added in this new tactic, and copy and paste works OK for Firefox in the latest version of Bespin in code (not deployed to bespin.mozilla.com at the time of this writing):

dojo.declare("bespin.util.clipboard.HiddenWorld", null, {
    install: function() {
        // * Configure the hidden copynpaster element
        var copynpaster = dojo.create("textarea", {
            tabIndex: '-1',
            autocomplete: 'off',
            id: 'copynpaster',
            style: "position: absolute; z-index: -400; top: -100px; left: -100px; width: 0; height: 0; border: none;"
        }, dojo.body());
 
        var grabAndGo = function(text) {
            copynpaster.value = text;
            focusSelectAndGo();
        };
 
        var focusSelectAndGo = function() {
            copynpaster.focus();
            copynpaster.select();
            setTimeout(function() {
                dojo.byId('canvas').focus();
            }, 0);
        };
 
        this.keyDown = dojo.connect(document, "keydown", function(e) {
            if ((bespin.util.isMac() && e.metaKey) || e.ctrlKey) {
                // Copy
                if (e.keyCode == 67 /*c*/) {
                    // place the selection into the textarea
                    var selectionText = _editor.getSelectionAsText();
 
                    if (selectionText && selectionText != '') {
                        grabAndGo(selectionText);
                    }
 
                // Cut
                } else if (e.keyCode == 88 /*x*/) {
                    // place the selection into the textarea
                    var selectionObject = _editor.getSelection();
 
                    if (selectionObject) {
                        var selectionText = _editor.model.getChunk(selectionObject);
 
                        if (selectionText && selectionText != '') {
                            grabAndGo(selectionText);
                            _editor.ui.actions.deleteSelection(selectionObject);
                        }
                    }
 
                // Paste
                } else if (e.keyCode == 86 /*v*/) {
                    focusSelectAndGo();
 
                    setTimeout(function() { // wait just a TOUCH to make sure that it is selected
                        var args = bespin.editor.utils.buildArgs();    
                        args.chunk = copynpaster.value;
                        if (args.chunk) _editor.ui.actions.insertChunk(args);
                    }, 1);
                }
            }
        });
    },
 
    uninstall: function() {
        dojo.disconnect(this.keyDown);
    }
});

Because of the issues, I took out the UI buttons for cut/copy/paste, and am in fact wondering if the editor needs that row at all. I wonder if we can consolidate the header to one line, giving us more vertical space. A code editor for developers is not like Google Docs for average Joe users, so having the visual cues probably doesn’t matter in the same way for items like copy.

There are a few subtle annoyances such as running an action like killLine (Ctrl-K) which cuts the selection but has to do so in a non-work with the clipboard way.

End result: getting there, but still need to work on making this generally viable for any application on the Open Web Platform. What do you think?

UPDATE: Used the copynpaster variable throughout, and added a setTimeout around the paste operation as I found some people needed to hit the paste key twice as the focus()/select() was taking a little too long.

Do you ever write random bits of code and wonder why you are doing it? Something that seems worth it at the time and then you really wonder?

I just did this with a class that I call TokenObject. The goal of this trivial class is to take a string, and give me a nice way to get pieces out of it. Normally I would use split and grab the right piece, but I was doing so much conversion from “command line string” to an object, that I ended up with something that lets me do this:

var userpass = "dion password";
var up = new TokenObject(userpass, {params: "user pass"});
// up.user;
// up.pass;
// Can also get via: up.param('user'); up.param(0);

Funny huh? :)

/*
 * TokenObject: Given a string, make a token object that holds positions and has name access
 */
 
var TokenObject = function(input, options) {
	this._input = input;
	this._options = options;
	this._splitterRegex = new RegExp(this._options.splitBy || '\\s+');
	this._pieces = input.split(this._splitterRegex);
 
	if (this._options.params) { // -- create a hash for name based access
		this._nametoindex = {};
		var namedparams = this._options.params.split(' ');
		for (var x = 0; x < namedparams.length; x++) {
			this._nametoindex[namedparams[x]] = x;
 
			if (!this._options['noshortcutvalues']) { // side step if you really don't want this
				this[namedparams[x]] = this._pieces[x];
			}
		}
 
	}
}
 
TokenObject.prototype.param = function(index) {
	return (typeof index == "number") ? this._pieces[index] : this._pieces[this._nametoindex[index]];
}
 
TokenObject.prototype.length = function() {
	return this._pieces.length;
}

I have had the pleasure of spending time with members of the V8 team such as Kevin Milikin and Kasper Lund, and Brendan Eich of Mozilla and TraceMonkey. I haven’t had a chance to talk with the great folks behind SquirrelFish Extreme yet, which looks very good too.

It has been great to talk to these guys about the way their implementations work, and to look at differences too. What has been particularly exciting has been how they have loved each others work. There is a huge amount of respect between all of the teams, and they are jazzed to be making the JavaScript world scream.

One common issue that I have found has been benchmarks. They all use slightly different benchmarks and can all agree that:

The JavaScript benchmarks suck!

SunSpider has known bugs in it which are not changed because they have a baseline with IE. Are you kidding me? It is also very Regex heavy, and also focuses on other areas. The methodology of just adding up all of the timings also doesn’t help, and totally skews results (you could do the important things fast, and the obscure things slow, and still look bad). IE can’t even run the tests correctly, and thus their numbers are messed up too.

Dromeo is interesting, but has flaws too. The timing seems to be broken as only V8 gives you precision that doesn’t get rounded down, and thus they get penalized. Also, it uses libraries which do “if browserX do X elsif … else ….” which means different engines actually go down totally different paths.

This is the tip of the iceberg. Benchmarks are notoriously hard to build in a way that measures practical performance versus micro meaninglessness.

It seems like everyone acknowledges that JS benchmark life is pretty bad, so we can either keep that in mind, or we can get everyone together to have an attempt to do a better job. Doing the practical thing can be tough too, as there are too forms of practical:

• How we do things now to get past browser bugs
• How we could do things if there were those silly browser bugs

Time for SPEC JSVM?

Brendan Eich promised that trace based JIT’s will give us killer JavaScript speed and now we have seen the fruit of his labour with TraceMonkey (adds trace smarts to Tamarin-tracing).

A lot has been said already, but I am really excited about much more than the “look at how we run Sun Spider”.

The DOM matters

Although it may not sure in todays code drop, Brendan gets that you have to care about the DOM and not running while(1) { cheasyTest(); }:

• He gets it: “As we trace more of the DOM and our other native code, we increase the memory-safe codebase that must be trusted not to have an exploitable bug.”
• If DOM code comes out of native code and into JavaScript, it matters: “TraceMonkey advances us toward the Mozilla 2 future where even more Firefox code is written in JS. Firefox gets faster and safer as this process unfolds.”

Canvas

You know Canvas? It’s days are coming… fast, as John Resig mentions:

One area that I’m especially excited about is in relation to Canvas. The primary thing holding back most extensive Canvas development hasn’t been rendering – but the processor limitations of the language (performing the challenging mathematical operations related to vectors, matrices, or collision detection). I expect this area to absolutely explode after the release of Firefox 3.1 as we start to see this work take hold.

Opensource matters

None of this could be accomplished without the great side of opensource. TraceMonkey uses Tamarin-tracing, which enables it to build on amazing code.

TraceMonkey itself is opensource, which enables the SquirrelFish team say, to check it out. And, vice versa. This will keep the competition going along nicely.

This kind of competition is phenomenal for us, the consumers of the Web. Performance is key in 2008, and Web developers get to laugh all the way to the bank.

Thanks guys!

After listening to Brendan talk about typing on the latest Open Web Podcast episode on ECMAScript Harmony it got me thinking again about optional typing.

It has always bugged me a little to think of type information in my nice clean dynamic languages (Ruby, JavaScript, Python, Perl, etc.).

Looking at even the simplest of code like this (taken from Mike Chamber’s XMPP server in ActionScript 3)

var room:Room = new Room(connection);

It irks me. Just work out that it is a room already won’t you? I know you can do it? You CAN do it. So, leave it out in this case.

Although the type information is optional, it seems that a lot of the code that I have seen in AS3 puts types in all over the place. I somewhat like the idea of using types when you really need them, such as for clear documentation of a core library, or some performance issue that you find is an issue (note: that becomes an issue, not one that is assumed!)

Once you open the door though, can people play that way? Is giving the programmer a bit switch of “shall I put the type in here?” every few seconds a good thing? Especially when the tools try to put the type in all the time?

I also found it very interesting when Brendan talked about how adding type information can actually hurt performance, sharing an example of when var foo:int doesn’t do what you actually need. For me, performance is pretty much out of the window, now, and definitely in the future.

So, I lean towards not needing it. But then I flip over the cards and see stuff like this (note: a mock example):

/**
 * @type Room
 */
var room = new Room(connection);

Ugh. Now we have the worst of all words. We are documenting the code out of band. This is often done for some jsdoc-like tool that will generate docs for you. The problems are:

• The language itself has no idea about this information. The compiler. Nor, runtime. If there was something that could be done for performance, it couldn’t be
• How easy would it be to get this out of date. If you change the type no one is going to complain. There is no validation here.
• Way more verbose!

This alone makes me think that I would rather have optional types just to avoid any hacks like this.

You?

I had a post in my queue relating JavaScript 2 to Perl 6. They are both in very rough spots indeed, yet JavaScript 2 has a couple of features that have me a lot more worried than I am about Perl 6:

• No offense to Perl (a language I really love, and I have code in CPAN) but JavaScript is an immensely important tool on the Web
• Unlike Perl, there are no dictator groups who can clean up. The politics are politics. If a certain someone doesn’t implement, you are in big trouble.

I have been hacking more and more on JavaScript (1.0, Brendans toy!) recently, and the more I do so, the more I like it. I worked out some years ago that it was the DOM and cross browser issues that I didn’t like, not the language.

I do wish that there was a standard library for JavaScript (Dojo does a good job). I do find myself reinventing the wheel a lot when it comes to the Object world. This week I will be in a Class.create() kinda mood. Other times I would fancy some new function() magic, and other times a bit of (function() { functions; return { ... } })(). Now I try to use the right tool for the job, but the problem is reading other peoples code.

Take a peek at the Firebug source and you will see the following at the top of each module:

FBL.ns(function() { with (FBL) {

Joe Hewitt likes with() :)

This is where it starts to feel a lot like Perl. Many ways to do one thing, and when that one thing is so core as packaging and modularization, it shows through. This is also a reason why we are in pain with components. If someone writes a great component for Dojo but you are using Prototype, you curse. What is the real difference? The darn packaging! dojo.hitch() versus function.bind(). These changes go so deep though, that it would be quicker to write from scratch.

I would love an import/include statement. Finding myself dynamically creating script elements to do a script src is starting to get a little silly. The language itself should just support that.

JavaScript 2 is helping us with packaging, but in my opinion, it used to have too much in it. Programming units, packages, namespaces, that is just too much to wrap your head around.

I am also not a fan of the optional static typing. That is fine for ActionScript, but I don’t think JavaScript needs it. This is a scripting language, something that can fit in a onclick="..." and I think the world is moving more towards the dynamic world (there is still a place for static). How about just allowing ActionScript to be written in the browser? Why try to boil the ocean with JavaScript, as Java has tried to do in the past. Work on the runtime, and allow multiple languages. This is already being done with IronMonkey and the like…. and Microsoft is doing a great job with that tactic in Silverlight. Work on an object model that can be shared, and then we can all go to town.

var self = this;

I see that too much too, even with the binders and the hitchers. The magic of var foo jumping to the top of the scope is weird too. But, all in all, JavaScript is pretty damn good! In fact, JavaScript 1.8, and what I hear about 1.9 is really nice too. I am all for cleaning up what needs to be cleaned up, and working on other stuff.

At this point in my draft I sat in the corner and cried, as I thought JS2 was in epic trouble, and that the walls were going to fall down. I didn’t know what to do. We had Brendan++ on one side, and Doug/Microsoft on the other (simplified version). I sit in between. Without some Harmony, what would happen to the Web?

Well, the reason that I can post this is because I heard that the ECMAScript meeting in Oslo was very promising indeed, and some Harmony may come out of it. Hopefully the news will get through the official process, and out in public as soon as possible, as there is new hope for us. I am sure there is still a long road, but with great people working together, we’ll get ‘er done.

Ever since Aaron Boodman uses the back and forth saw animation to demonstrate how the Gears WorkerPool can keep your UI responsive with the browser, I have used it for testing just that. It turns out the human eye is quite good at the “oh man look at how badly that jumps” kind of test.

To make it simple, I have componentized it, allowing you to drop in a kitt.js and a new Kitt() to be on your way.

This is the minimum that you need to kick one off. The new Kitt will look for an element with the default id of ‘kitt’. Since it isn’t found, one is created and appended to the end of the body.

What about expanding this?

You can tell Kitt to do more, such as giving it an explicit id, colors to use, whether to use the automatic start and stop buttons, etc. Here is an example that turns off the buttons and programatically starts and stops the animation.

var evilkitt = new Kitt({id: 'evilkitt', animateBackgroundColor: 'red', hideButtons: true });
evilkitt.start();
setTimeout(function() { evilkitt.stop(); }, 6000);

With this simple little component at work, I can drop it in on the variety of performance testing work that I am doing more and more of these days (hopefully something to share one of these days).

Now, I need to make it use SoundManager2 and the sound of Kitt! ;)

For more information, check out the Google Code project that hosts the code.

Sometimes you can get in the zone just enough to be productive on a plane. On my flight to Mexico City yesterday, I created Endpoint a project that contains a server proxy, JavaScript client, and Greasemonkey Script with a mission. The mission is to take a URL, work out if it is a redirect (via a Location: header), and then return the final endpoint for it.

Why did I do this?

I was brainstorming functionality for a Twitter client with James Strachan (he is working on gtwit) and we talked about how annoying tinyurl / is.gd / snurl / you name it URLs are. They don’t tell you where you are going, and you could get Rick Rolled (if you are lucky) or much much worse.

So, I wanted to create a library, and one client (Greasemonkey) to test it out. Then anyone else could use it too to resolve directly from their Web pages.

How does it work

You load up the JavaScript via script src and then you can call resolve, passing the URL and a callback that will get the result. A few examples:

// Simple version
Endpoint.resolve('http://snurl.com/2luj3', function(url) { 
  alert(url); 
});
 
// Using the original URL to work out if it has changed
Endpoint.resolve(
  document.getElementById('testurl').value, 
  function(url, orig) { 
    alert(url); 
    alert(Endpoint.isRedirecting(url, orig));
  }
);
 
// How it is used in the Twitter Endpoint Resolver
Endpoint.resolve(url, function(resulturl, originalurl) {
  if (!Endpoint.isRedirecting(resulturl, originalurl)) return;
 
  newtext = newtext.replace(originalurl, resulturl, "g");
  jQuery(el).html(newtext);
});

Under the hood, a bunch of stuff is happening. I would love to be able to just use XMLHttpRequest to dynamically hit the URL and look at the headers, but the same-origin policy stops me.

This is why I have the server proxy, which returns a JSONP callback.

When you call resolve(url, callback) the script tag is created on the fly and added to the DOM. The callback function is all handled to allow multiple calls, and then the chain unravels.

Here you can see it all in action, showing how my Twitter stream will go through and the URLs will dynamically change from their tinyurl versions to whereyouaregoing.com:

I wanted to use App Engine to host the server proxy, but unfortunately I can’t work out how to do that yet. You have access to the URLFetch API to access resources from App Engine. Unfortunately for me, one of the features is that it understands redirects and just goes on through to the full resource itself, with no way to get the endpoint from the headers in the response.

It was also interesting to read Steve Gilmor talk about these services all be it in a post that is hard to actually understand ;)

Also, Simon Willison just put up a simple service on App Engine, json-time, that “exposes Python’s pytz timezone library over JSON.” I think that we will see a lot of these types of mini-Web services hosted on App Engine. Taking Python utility and making services from its goodness is an obvious choice.