Feb 26

Supporting the system clipboard in your Web applications; What a pain!

Bespin, Tech with tags: 26 Comments »

copypaste

When we launched Bespin people noticed that our copy and paste experience wasn’t the best. It wasn’t integrating with the system clipboard in most cases. An editor needs clipboard support. Needs it. Essential. 101.

It turns out that this can be a royal pain to work across various browsers and platforms. I thought it may be good to share our story, where we were, where we are, and where we want to be. I also am looking forward to hearing your ideas, as we may have missed something.

First, what are we talking about here?

With our canvas based editor we needed access to the system clipboard to be able to copy data into, cut, and paste from the outside clipboard.

The first version that we had, used Clipboard Copy which accesses the clipboard APIs available in Flash. Unfortunately, our parade was rained on. With Flash 10 Adobe changed the access model to these APIs (due to security issues). Now a user has to explicitly click on a SWF control to get the access. To help, zeroclipboard tricks Flash, allowing you to at least fake out the user and have them click on any element. zeroclipboard hides a translucent SWF control on top of the element so the user thinks they are clicking on the “copy” icon, or something like it. This is all well and good, but we obviously need to tie into the key combinations that we habitually know (Cmd/Ctrl C, X, and V).

With Flash not able to do what we need (as Flash 10 will be well supported soon, and it broke it) we needed to take a fresh look at the problem.

What APIs do we have available in the browsers? In typical fashion, each browser does things differently.

IE had an early API to give you access. I won’t go into detail on this puppy because we aren’t interested in supporting IE yes, until we can get Canvas/fillText running in a performant manner. Then we will care.

What about Firefox?

Firefox has a rich clipboard API in the XUL platform itself via the nsIClipboard interface. It has everything we could possibly need, and we can implement copy and paste easily:

copy: function(copytext) {
    try {
        if (netscape.security.PrivilegeManager.enablePrivilege) {
            netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect");
        } else {
            clipdata = copytext;
            return;
        }
    } catch (ex) {
        clipdata = copytext;
        return;
    }
 
    var str = Components.classes["@mozilla.org/supports-string;1"].
                              createInstance(Components.interfaces.nsISupportsString);
    str.data = copytext;
 
    var trans = Components.classes["@mozilla.org/widget/transferable;1"].
                           createInstance(Components.interfaces.nsITransferable);
    if (!trans) return false;
 
    trans.addDataFlavor("text/unicode");
    trans.setTransferData("text/unicode", str, copytext.length * 2);
 
    var clipid = Components.interfaces.nsIClipboard;
    var clip   = Components.classes["@mozilla.org/widget/clipboard;1"].getService(clipid);
    if (!clip) return false;
 
    clip.setData(trans, null, clipid.kGlobalClipboard);
},
 
data: function() {
    try {
        if (netscape.security.PrivilegeManager.enablePrivilege) {
            netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect");
        } else {
            return clipdata;
        }
    } catch (ex) {
        return clipdata;
    }
 
    var clip = Components.classes["@mozilla.org/widget/clipboard;1"].getService(Components.interfaces.nsIClipboard);
    if (!clip) return false;
 
    var trans = Components.classes["@mozilla.org/widget/transferable;1"].createInstance(Components.interfaces.nsITransferable);
    if (!trans) return false;
    trans.addDataFlavor("text/unicode");
 
    clip.getData(trans, clip.kGlobalClipboard);
 
    var str       = new Object();
    var strLength = new Object();
    var pastetext = "";
 
    trans.getTransferData("text/unicode", str, strLength);
    if (str) str = str.value.QueryInterface(Components.interfaces.nsISupportsString);
    if (str) pastetext = str.data.substring(0, strLength.value / 2);
    return pastetext;
}

Verbose isn’t it? Hardly the most succinct API ever for the common case, but it does the trick. Or does it.

You can see the real problem here:

try {
    if (netscape.security.PrivilegeManager.enablePrivilege) {
        netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect");
    } else {
        clipdata = copytext;
        return;
    }
} catch (ex) {
    clipdata = copytext;
    return;
}

For security, we have to ask the user to give us permission to use copy and paste. You get one of those ugly dialogues, and the user will be asked repeatedly for each session.

Not only is this a pretty poor user experience, but we can’t even get this far. When we ask for the priviledge, depending on the users settings, changes are that you will never actually get asked to grant them. You have to sign JavaScript and the like to even get close. This is pretty crazy. I don’t even get a CHANCE to ask the user for extended privileges (unless you get users to do magic like: user_pref("signed.applets.codebase_principal_support", true);). Here is where the cliff of the Web platform raises its ugly head. We need to fix this!

Time to get subtle here. Our use case isn’t about being able to sneakily get access to the clipboard without the user knowing. All we want it to actually get access based on the user doing something. When they Cmd/Ctrl C, X, or V. When they click on a copy button, then do the action. This is very different than getting access to the clipboard at any time.

In the code above, notice what the exception cause is doing: clipdata = copytext; This is our worst case scenario. If we can’t get access to the clipboard then we keep an internal clipboard that only allows you to cut/copy/paste within the editor itself. As soon as you want to copy something from a web page and paste it into the editor? Outta luck. This isn’t good enough.

Now, let’s take a look at WebKit. WebKit does a decent job and implementing the initial Microsoft work in a nicer DOM way.

There are a set of DOM events that have pairings that tell you before* “getting ready to do the action so set things up if you need” and then the action itself.

  • beforecopy
  • copy
  • beforecut
  • cut
  • beforepaste
  • paste

In our world with the editor, we use the before events to set things up, and we have to do something pretty hacky to make it happen. The copy event itself only actually goes through if you are on an element that supports it. There are hacks around this too. For example, if you want to be able to get a copy event on a div, you need to turn on contentEdible and set the tab index to -1. Strange huh?

To get around all of this, we use a hidden text input, which can of course accept these events. Then, in the before event we focus over to that hidden element. Here is an example for beforecopy:

Event.observe(document, "beforecopy", function(e) {
    e.preventDefault();
    $('copynpaster').focus();
});

You will notice that we make a call to preventDefault. This tells the system that we are in control and are handling the copy ourselves. This turns out to be important is some subtle ways too. For example, The OS will often grey out a menu choice for “Copy” if it doesn’t see anything selected. Since we are in control of the selection, we need to override that behavior and continue to pass through the event. In fact, I noticed a few weird things happening if you don’t get this all just right. For example, in WebKit, if you select the “Edit” main menu, it would fire a beforecopy right away, and then once again when you selected copy. Strange.

But, now we have gotten beforecopy to focus us in the text input, we are ready to implement the copy event which will not get passed through. Here we need to do OUR magic to get the selection that we want to put into the clipboard, and for that we use e.clipboardData.setData(type, data). In our case the type used is just plain text (test/plain) but you could use other rich data formats if you need.

You will also notice at the bottom of the code, we focus back on the canvas to shoot us back to where we were. This completes the cycle:

Event.observe(document, "copy", function(e) {
    var selectionText = _editor.getSelectionAsText();
 
    if (selectionText && selectionText != '') {
        e.preventDefault();
        e.clipboardData.setData('text/plain', selectionText);
    }
 
    $('canvas').focus();
});

The cut action is identical bar the step that gets the data into the clipboard. In the case of cut we also add a step to delete the current selection:

_editor.ui.actions.deleteSelection(selectionObject);

Paste is also quite similar, but instead of adding to the clipboard we are getting the data out, and for that we use:

e.clipboardData.getData('text/plain');

Now we are good to go. You can see the code in its entirety here.

You may notice some of the wiring up of the two styles (Firefox/WebKit).

We cheat and do a simple test:

setup: function() {
    if (Prototype.Browser.WebKit) {
        this.install(new Bespin.Clipboard.DOMEvents());
    } else {
        this.install(new Bespin.Clipboard.Default());
    }
}

We favour feature detection over browser detection of course, but here it is a little painful as the real test is to to see if the event in a copy/cut/paste/ has a clipboardData object associated with it. That’s a little too deep into the onion layers, so we punt.

We would love to use this standards approach with Firefox too, and we almost can. MDC has docs on support for this, such as paste but the rub is in this bug that states how onpaste doesn’t let you getData() back, which makes it useless for all bar signaling when the user wants o paste something. PPK does a good job talking abou the various quirks with these events in general.

We still have some problems though. We handle the copy/cut/paste menus and keyboard shortcuts perfectly, but our UI has our own icons that try to kick off these actions. This is where we need to get zeroclipboard going, and then we are done (in WebKit).

Phew. That turned out to be a long explanation. Fun times on the Open Web! What am I missing? What tricks have you run into? Inquiring minds want to know!

More articles from Bespin

We are learning a lot from Bespin and beyond. Here are posts from the team and community that you may find interesting:

Feb 24

Browsers are finally catching up; It isn’t the Future of Web Pages is it

Tech 7 Comments »

Ben and I are speaking at the Future of Web Apps in lovely Miami. Having just been in the midwest, heading to some nice weather is welcome. I spoke at the event in London and thoroughly enjoyed it. At the time I was speaking on Gears. This time, Ben and I are talking about the state of the Web, and it obviously excites us. The timing is perfect as we give the talk on the back of the recent release of Bespin, the project that let us play with the bleeding edge. We were unencumbered by the chains of old browsers, and it shocked us a little. These chains keep us in the stone ages, and if we just suck it up and live with them the platform will pale in comparison to those that rev.

It is 2009 yet the majority of us still have to work in a world of browsers that were built for web pages. This show isn’t called the Future of Web Pages though, is it!

Through innovative and tiresome work the community evolved what we finally coined Ajax. It looked a little like this:

Ajax hackery

It was a series of hacks. A series of quirks. Painful. But, just like the early planes, who cares? The beast does fly! This cobbled together set of technologies gave us our flight in the form of giving us the ability to build applications. The small changes that we got in XHR and the like meant that we could create interactive experiences on the Open Web.

But, the browsers are finally changing. The new crop come with technologies that show that the browser vendors are thinking about building a platform for desktop quality applications. The Chrome comic book was full of this. We highlight some of these technologies:

  • Super fast JavaScript. Other platforms have smart JIT VMs and we should be no different. With the latest VMs that browsers are integrating we get orders of magnitude in performance gains. This doesn’t just make things a little faster, it means that the game has changed and you can do things that you previously couldn’t
  • Offloading the processing. Fast JavaScript is great, but you still have a bottleneck in the old single threaded architecture of the browser. Web Workers gives us a nice API (no threading APIs here!) to offload the work. As we think that we can do more (due to the faster JS) we find ourselves with more code, and we need to let the browser do its thing while this code runs.
  • Multiple processes. Even with fast JavaScript and Web Workers, we need to isolate our worlds. If you have Gmail, a calendar, Bespin, and three other real Web applications running, all polling for updates and the like, it is too easy to have them collide as they all ask the CPU to do work. With the new multiprocess architectures we side step this issue. When I am in Bespin I want it to have as much nice as possible.
  • Rich graphics. We are used to having text, boxes, images to work with. That’s kinda it! Isn’t it amazing to think about what we have done with those simple building blocks? With Canvas, SVG, and the great new CSS primitives (border radius, css animations, css transitions, css shadows, and more) we get to think outside of the box.
  • Desktop services. We need to reach out. Real applications need to talk to the clipboard, get geolocation, access your address book, etc. We need to solve this problem to get past the cliff that we currently experience where people say “but to do X, you need to use a native desktop framework”. We need a trust model that works, and to get the APIs into the browser.

What we end up with is this a metamorphosis from the first planes to the jet:

Web Apps Future

This is why I am excited to work on new developer tools for the Open Web. I am jazzed that the community is making this happen. Browser vendors, web developers, together we can make a big difference.

Of course, when the koolaid wears off I am always acutely aware of the problems that we have on the platform. There is a long way to go, and we can’t just ignore old browsers. Fun times ahead though!

Feb 21

Exactly what is wrong with education and government; Input vs. Output

Personal 5 Comments »

Taking sexual harassment training is good stuff. I get it. I support it. However, the implementation is so poor that I had to comment on it.

The law states that you need to take 2 hours of training. See anything wrong with that?

As I took the online class, after the first section (where I got every question right) I faced the following screen:

basedoninput

I was “ahead of schedule” so they made me sit and wait for minutes so I would “catch up.”

Doesn’t it make more sense to have the expectation that: “You will understand US sexual harassment law” with an exam that proves it, instead of just “you sat around for 2 hours…. and we all know that why you waited in between you could have just been playing Wii Mario Cart.”

Can we please start to measure on output and not input? If someone answers 100% of questions in 3 minutes, good on them. They know the law.

Feb 20

Headless Eclipse, and Smalltalk browsing with Bespin, thanks to Avi Bryant

Bespin, Tech with tags: , No Comments »

Nothing like seeing people doing things that make you saw “Wow.” and they are using your project!

First, I posted on the Mozilla Labs blog about Boris Bokowski and Simon Kaegi and their work with Bespin fronting a headless Eclipse. Their demo is exciting to us as they were able to implement the Bespin Server API in short order, and use Eclipse to run Java compilation on the server, thus returning errors back to Bespin. This is exciting as this would enable a huge variety of languages and functionality.

I was fortunate enough to meet Boris, Simon, and an Eclipse crew today. It was a real pleasure to talk tech and share ideas. Amazing when you are surrounded by smart engineers who have been doing tools for a long time!

bespin-eclipse

Then, as I sit at the airport, I see a message from Avi Bryant of Seaside and DabbleDB fame (as well as many other things).

He did something very cool indeed. He backed Bespin with Squeak:

I work in Smalltalk, which deals with source code at method
granularity rather than file granularity. I spent a few minutes today
playing with a Bespin backend for Squeak that exposes the package/
class/method hierarchy as if each method were a separate file, which
works very nicely, and feels very familiar (since column browsers
were, AFAIK, invented for Smalltalk-80 in the first place :).

bespinsqueak

He points out the issues of the current Dashboard to Editor design, which we agree with. Our plan is actually to unify the two, and now we want to do it even faster to help his use case. What if you can be in the editor and quickly bring down the horizontal file pane (moving down the editor). Then you can select the methods and the code will change below. Excited to get that going. We also need small things like having the dashboard remember state.

Thanks to the people that have spent time playing with Bespin. Very much appreciated!

Feb 13

Launching Bespin; Feeling light as a cloud

Bespin, Mozila, Open Source, Tech with tags: , 5 Comments »

Talk is cheap. Shipping code is important. I have always felt that to be the case, so I always look forward to the first time that I ship something. At my latest adventure I got to do that yesterday.

It was a fun ride to go from the technical challenge of “can this be done” to having the experiment out of Mozilla Labs and into the hands of the wider community. One of the reasons I am so excited to be at Mozilla is that I get to develop in the open. You can watch our source code repository and see a community on irc and join us in our news group. Running an open project well is part art, and is incredible hard to do well. I have deep respect for those in the open source community that have succeeded. I am really looking forward to that challenge with Bespin and beyond. I want to make sure that we are truly transparent (no hidden agendas and back channels). I want to raise the profile of anyone who contributes to the project so people really know who the people behind Bespin are. I want to try hard to get designs out there early in the process so decisions can be shared, but I want to make sure that a vision drives the project forward.

Foolish chaps and companies have come to me in the past thinking that open source will be a silver bullet for “getting other people to do our work.” Those that have been involved in open source know that it isn’t the case. It is often more work. But, it is worth it. I have no doubt that the community that we hope to grow will come up with amazing ideas and contributions. I am humbled by the contributions even a DAY after launch. I am stunned that people think our experiment is worthy of their time and thought.

The wind is at my back, but I know that announcing Bespin is the beginning and not the end. The birth of the project. Now we get to see if we can have the kid grow up.

Not sure what Bespin is? Here is some info from the announcement and more. Thanks again to all of the kind words from people across the Web. It means a lot:

Bespin proposes an open extensible web-based framework for code editing that aims to increase developer productivity, enable compelling user experiences, and promote the use of open standards.

Based upon discussions with hundreds of developers, and our own experience developing for the Open Web, we’ve come up with a proposed set of features along with some high-level goals:

  • Ease of Use — the editor experience should not be intimidating and should facilitate quickly getting straight into the code
  • Real-time Collaboration — sharing live coding sessions with colleagues should be easy and collaboratively coding with one or more partners should Just Work
  • Integrated Command-Line — tools like vi and Emacs have demonstrated the power of integrating command-lines into editors; Bespin needs one, too
  • Extensible and Self-Hosted — the interface and capabilities of Bespin should be highly extensible and easily accessible to users through Ubiquity-like commands or via the plug-in API
  • Wicked Fast — the editor is just a toy unless it stays smooth and responsive editing files of very large sizes
  • Accessible from Anywhere — the code editor should work from anywhere, and from any device, using any modern standards-compliant browser


View Introduction to Bespin

Credits

There have been a lot of people that we can thank for getting us out there today. Firstly, our new team of Kevin Dangoor and Joe Walker. Secondly, the great new colleagues that we have at Mozilla. Our Labs team members have been inspiring. We are building on the shoulders of great work. We are not only working closely with the Ubiquity team (Atul Varma, Aza Raskin, Jono, and others) to make sure the command line and Ubiquity are integrated, but we use Atul’s code illuminated to house the documentation for Bespin code. The Weave team has provided guidance for a future where Bespin data can be housed in their scalable infrastructure, which excites us. Whenever we chat with a Labs team we see places for integration, and we can’t wait to get there.

We care about design, and have been fortunate enough to have input from two great designers: Sean Martell and Chris Jochitz.

Other Mozilla folk have helped a lot too. You will notice that Bespin makes heavy, heavy use of canvas. Vladimir Vukićević has given far too much of his time to let us run through ideas and profile the canvas performance. We have also already seen contributions from outside of Mozilla. A few issues have been put into Bugzilla by beta testers, and even code patches (for example, thanks to Ernest Delgado for his canvas skills).

We have only just begun. We really wanted to get this tech preview out as soon as we could to embrace the community and experiment heavily. We hope to have your name in the credits soon!

Get Involved

There are many ways to get involved with the Bespin project and the Developer Tools lab. You could start by giving us feedback on the product (via comments, in our group, on irc in channel #bespin, or in Bugzilla).

Have a feature you would love as a developer? Fancy sharing a design concept? (We like those). We would love to hear from you on all fronts, from ideas to design to code. One of the reasons that we are excited about Bespin is that it is written for the Web platform, on the Web platform. This means that your Web skills can be applied to your tool. Want a nicer syntax highlighter? With that we had support for a version control system that we don’t support? Wish that there was interesting Python support? Help us build it!

Bespin has been built with extensibility in mind. We want you to be able to tweak your tool. Bespin Commands are just one example. Would you like to embed the Bespin editor into your own project? We want to enable these kinds of use cases.

Feb 09

Will the real WebKit please standup

Comic, Tech, Web Browsing 6 Comments »

webkits

I recently heard a conversation in Palo Alto which had the following in part of it:

  • Chap A: Are you testing in WebKit?
  • Chap B: Which WebKit?

I have heard this before. I am running into this issue myself right now. Something that our team is playing with uses some bleeding edge standards, and some recently were added to WebKit nightly. Unfortunately, that doesn’t mean that they are in Chrome, AIR, Mobile Safari (in this case it is), or the other people who use and fork WebKit (from Nokia to Titanium).

On the one hand companies are highly leveraging each others work when they choose WebKit. Forking isn’t necessarily an evil thing anymore, especially if everyone is giving back to the cause as much as possible. The WebKit team implements CSS Animations, and you hope that it will be placed in browsers that use WebKit in the future. Watching the WebKit community deal with the many large companies will be interesting indeed in the coming couple of years. Luckily, the core team is top notch, and continue to push forward.

Feb 06

Internet Explorer: Microsoft and the EC

Microsoft, Mozila, Tech 2 Comments »

NOTE: It goes without saying, but to be clear, none of my thoughts here are related to Mozilla itself

Mitchell Baker has a post on the EC prelim conclusion on Microsoft and IE:

From EC: “Microsoft’s tying of Internet Explorer to the Windows operating system harms competition between web browsers, undermines product innovation and ultimately reduces consumer choice.”

Mitchell has some very interesting points, and the two high level ones that stick out at me are:

They did a bad bad thing

In my mind, there is absolutely no doubt that the statement above is correct. Not the single smallest iota of doubt. I’ve been involved in building and shipping web browsers continuously since before Microsoft started developing IE, and the damage Microsoft has done to competition, innovation, and the pace of the web development itself is both glaring and ongoing.

Bloody hell, coming up with a remedy that is fair and good for the Web is tough!

There are separate questions of whether there is a good remedy, and what that remedy might be.

The extent of the damage is so great that it makes it difficult to figure out an effective and timely remedy. I believe it’s worth some effort to try.

Thinking about the users, you end up at PPK’s point: 0 or 5. Hard indeed.

Also, take a good look at the entire Mitchell post as it is really interesting to see how Firefox has grown despite of the barriers, and that Microsoft shouldn’t be able to use it as a “see, look what can be done.”

Bizarre times indeed.

Feb 05

Gmail Multi Pane changes workflow

Google, Tech with tags: 14 Comments »

gmailmultipane

The new multipane view for Gmail by Vivi will change my workflow for the better. I have placed views to the right of my inbox that show me:

  • My starred items (is:starred)
  • Emails sent to me (to:me)
  • Notifications. This is a big one. These items are things that I kinda want to see fly by but don’t care too much about. In the past they would clutter my inbox as I didn’t want to “skip inbox” and not see them (I never pro actively check out labels). Now that I can have the list scanning by, I CAN change my filter to skip the Inbox which will keep it cleaner!
  • Skipped content. I have a bunch of mailing lists that I tell to skip the Inbox. The end result is that I rarely see it, so it is really quite silly. Now though, I can have a small window into that world.

Very cool indeed. There are pain points though. If you are not plugged into a nice 30″ monitor, your real estate gets cramped fast. You can choose to put items on top, bottom, or right, but right is the only option that makes sense to me.

I want to be able to have fine grained control here. Gimme a splitter so I can choose how much space to give. Let me put some on the right, some on top, some on the bottom. Let me show and hide items so I can shoot open my inbox to be full screen when I need too.

Selecting items on the other panes is strained. You don’t have checkboxes and the arrow keys to help you over there. I wish that I could click on a space on the very left and have the “>” item show up there so I can move around and select things. I hate the mouse.

I am curious to see how I like this, and how it changes me Gmail usage. Do you like it?

Feb 04

Google Latitude; Get your explaining ready

Comic, Google, Tech 3 Comments »

googlestripclubs

Google Latitude is cool indeed, and will have folks from Loopt and beyond watching (see Dodgeball didn’t die, it’s just slowly becoming a feature of Maps!).

Even with all of the privacy controls, you KNOW that there are going to be fun cock ups in the future, just like we see on Google Maps Street View. It just has to happen. But, it’s worth it :)

Shame the iPhone version didn’t come out on day one, and the Palm Pre version will be cool :)