techno.blog("Dion")

Jason Robbins and Jenan Wise have released a new Ajax source code browser for Google Code project hosting:

We recently launched a new source code browsing tool as part of Google Code’s project hosting feature. This new tool makes it easy to navigate through a project’s Subversion repository. Key features include: fast directory browsing tree, syntax highlighting, history of changes, and easy-to-read diffs. See it yourself under the “Source” tab of any project that we host.

The new system certainly feels fast, and uses jQuery to flip around in short order, jumping through the revisions of your system, expanding into new directories, etc.

This change required a change to my Google Code greasemonkey script that adds a direct link to the trunk. It now takes you right to the trunk of the code in the new browser!

One of the interesting differences between a liberal opensource project, and a commercial endeavor is that you don’t know who is using your stuff.

If someone wants to extend or integrate beyond your work, they don’t need to deal with you to make it happen.

The fun side-effect is when you get the code drop. Out of the blue, someone sends you a message and says “oh, by the way, I just ported X to work with Y”.

This really keeps you on your toes, and it gives you a gift. The “oh man, that is so amazing that they did that work, now I don’t need too!” gift.

I personally believe that Googlers will produce more open source in 2008 than prior years, and this post will discuss why. The same reasons apply to many other companies, which leads me to believe that we will see an increase in open source across the board.

Google does a lot of open source

When I am on my travels I get enough questions about Google and open source that has me realize that Google does a bad job in showing what a good open source citizen it really is.

Many people do know about the fantastic programs that Leslie Hawthorn and her team puts on which include: the Summer of Code and the Google Highly Open Participation Contest. This is the tip of the iceberg with respect to what the open source group does though.

As I talk to more and more people at Google, I find out about more and more people like Jeremy Allison, or the Samba team, who gets to spend much of his time hacking on the Samba project. There are Linux kernel hackers, Firefox contributors, and then you get to smaller pieces such as Hibernate Shards and MySQL tools.

If you have a butchers on Google Code itself you will find 135 current projects from Google, and this doesn’t include the magnitudes more that Googlers contribute too. 20% time really helps here of course, and it is a popular choice for engineers to dabble in the open source world one day a week. I can imagine the situation of working on some internal project, potentially something that is hush hush and secret, and having the option of spending some time on something very public indeed would be appealing.

So, we contribute a lot of open source code to various communities, we put out a bunch of our own code, and of course there are a number of high profile projects that have open source components to them: Gears, OpenSocial, and Android.

Erm, so if that is a lot… how come more?

Now I have hopefully shown a glimpse of the top of the iceberg for the amount of open source development that Google does, you may be wondering why I think 2008 will be ever better. Is it a numbers game based on the fact that we are constantly hiring more top notch developers? Well, that will hopefully help.

The core issue is that “doing open source” can mean many things. The code side of it is often the easiest part. Then you have the legal crud. What can be really challenging though is the process, especially when you have legacy code and procedures which everyone but the newest startups have (and the startups will have them soon too, as things change again).

Google is known for its engineering practices, and tools that help make them happen. I think that we do a really good job at providing a process that makes sense when you have as many programmers as we have, but not being heavyweight. A constant tight rope walk for sure.

One of the tools that has been talked about before is Mondrian, a Web based tool written by Guido himself, to help the review process of code. It is the kind of tool and process than if I left Google I would want to replicate, and would miss.

It is well known that Google uses perforce, yet most open source projects are using Subversion, or Git, or [insert your favourite new shiny toy]. This leads you to a problem when you decide to open source some code. How do you manage the fact that your code AND TOOLS are for X, when the open source world wants Y? If you are lucky, you can do a dump and have everyone using the other land. That sounds like the simple solution, but what if that code is being used in other projects within your company? This is where the real world gets complicated.

In many enterprise companies that I have had the pleasure to work with:

It is common for these process issues to be so painful that doing the open source thing isn’t worth it!

A lot of companies are still working out how best to handle these situations. It is easy for a small new company to come along and base its work on open source. You use what the open source world uses, and your life is easier. This shows how it can make sense to go down this path and favour open source software where appropriate. If you think that you may open source some of your code later, or want to integrate with open source, why make it hard for yourself?

One way to help with the friction is to open source more and more of your own code. You quickly find that although you thought you were just open sourcing Foo, you actually need to open source a slew of dependencies, and it would help to open source some of the tools too!

I only just ran across Review Board, which looks a little like Mondrian with support for various VCS. Promising indeed (Django/Python app).

Google is getting better and better and handling these issues, more and more tools and libraries are open, which means the path to open sourcing code in new project gets easier.

This is why I think that 2008 will be a fantastic one for open source at Google, and throughout the development community, as other large companies get better and better at it too.

What do you think? Have you run into these issues?

Today, I was able to launch a new section of Google Code that is focused on open source code related to the Mac. Since I am passionate about both open source AND the Mac, it was a really fun little side project to work on.

The core Google Mac team, and other developers at Google, actually have a fair amount of Mac related open source code. From small pieces of helpful code such as the Quartz Composer patches for Leopard, to large projects like MacFUSE to fun tools like iPhoneDisk to a new meta status updater called Statz. This is just the beginning of course, and now there is a home for this code.

One of the pleasures of working at Google is the slew of great people that you get to meet. On this project it was a real pleasure to work with Amit Singh (MacFUSE and much more), Nicholas Jitkoff (Quicksilver), and Dave MacLachlan (Adobe and much more).

Congratulations to Bob, Cedric, Romain, and the many many engineers that worked on Android.

Today the Android SDK was released, and along with it a raft of video content and documentation.

For all intents and purposes for developers Android == the SDK right now (until killer phones ship in short order). The development experience for Java programmers will be a dream, and I really like the architecture. They have really learned that declarative markup for UI is a Good Thing &tm;.

If you want to get a high level look at what this is about, I would start off by seeing what the phone can do:

Then, if you want to write some code, start by watching Dan build a simple application on Android and then delve deep into the Androidology that shows you the full architecture. Learn what .dex files are. See how cool Intents are. Check out the markup.

devphone.com: phones are decent now

I am really excited to see the bar being pushed by Apple, Google, and other players out there. You know that when the iPhone came out, Nokia had a lot of meetings and engineers got a better budget for doing innovative processes on their phones. With Android pushing the bar too, in a different way, I think that we can safely say that the phone that we hold in a year or two is going to be amazing.

I am quite astounded at how little Emily uses her laptop since she got an iPhone (apart from Scrabulous. She uses her laptop for her scrabble addiction.). You can see the future today by visiting Europe and Asia. Ben and I were so excited about this, that we went looking for a community to rally this excitement, kinda like Ajaxian for mobile. We were surprised that we couldn’t find it. It seems like most developers hang out in the forums and such of the various platforms. Since we are interested in development that transcends one implementation, we decided to start devphone as a place to throw all of the ideas into. It just launched and is very raw, so who knows what will come of it. Check out our welcome, an interview with Joe Hewitt, and subscribe to the feed

One of the fascinating effects of working for a company that so many love to keep a look on is that you get to be on the inside watching the thoughts of “analysts”, press, and random folk.

Watching the speculation around the “GPhone” has been fun. I particularly loved it when people would come up with suggestions such as:

• “The GPhone will read ads into your ear before each call”
• “The GPhone will have scrolling ads through the screen”

Riiiiight. That would go down really well wouldn’t it! I love how some think that Google has to literally put ads everywhere to make it worthwhile. Google needs the web to keep expanding and to have more people on more devices on it. If that happens, Google will do well.

Android

So there isn’t a GPhone, but instead there is the Open Handset Alliance, or Android (for a more fun name), which is an Apache licensed open source stack for mobile. No more walled garden. This is pretty huge. I can’t wait for the SDK to get out in the open on November 12th. I wish we could have gotten more information out there today to be honest. There are a bunch of usual suspects that people who read my blog also read that are a major part of this, so I am really happy for them that this is getting out in the open!

What I am looking forward to

The applications of course. Smart location-aware services will be fun, but what I really wish I could get is for a mobile digital wallet so I don’t have to use cash/credit cards. I want to use my phone for this just as they are able to do in parts of Europe and Asia. With an open platform that anyone can build applications for, I know it is going to happen.

Here is the fluffy look at Android, but for developers, please think about what apps you would like to see on a phone, and come back on November 12th for the real announcement that we care about… the SDK itself so you can see what you can do!

I played around with getting Ruby to work in the browser while watching some cheesy TV this weekend.

I was able to get things slightly working, and put this work in progress up on Google Code here.

That link sends you to a page that will evaluate Ruby for you, both from a script tag, and from your own input.

If you take a look at the top of the file, you will see:

<script type="text/ruby">
#class Foo; def initialize; @foo = 'whee'; end; end; f = Foo.new; f
Time.now
</script>

This script is found by the Ruby in the browser system, and is run through JRuby via a little applet. The page takes the output at the end and throws it into the container HTML on the right.

You can also edit the text area at the bottom left and run your own Ruby code, and the output will be placed on the right too.

Now, this is just the first baby step to get to something feature full like:

<script type="text/ruby">
document.ready do |dom|
dom["table tr"] <<"<td>test</td>"
end
</script>

Just being able to run some Ruby doesn’t mean much. We need to give it rich support for the browser to be able to interact with the DOM and Ajax libraries to do cool things, using the power of Ruby.

So, there are a lot of TODOs here:

• The applet is a download of JRuby complete, which is huge, and isn’t all needed. I need to work with the JRuby folk to produce a slimmed down applet version.
• Come up with a solution that allows you to puts data out (maybe put that data in a div somewhere), and more importantly, talk to the DOM directly from Ruby. Since the applet can contain Ruby files themselves, we can create a Ruby module that lets you do DOM stuff, that becomes JavaScript which will be eval’d to run in the browser. Or, we just give direct access. I am still playing with what makes sense there
• Only tested in Firefox. There are some known issues in Safari.
• I am lazily using an applet tag instead of the magic object/embed stuff that would make this work in other places

Any direction make sense to you?

Mike kindly started the presentation with a consuming warning, letting us know in advance that he was going to be pimping JIRA (because this was going to be case study-esque).

These days JIRA uses Lucene for “Generic Data Indexing”: Fast retrieval of complex data object. This isn’t about text searching for “dog” sorted by relevance. The statistic pages all come back from a Lucene index, not from the DB.

Lucene has a way for you to write your own Sort routines via Sort, SortField.

I have seen the “viral Lucene” pattern apply in a variety of projects. You start out using it for /search, and then you see that you can use it for other things. Slowly your DB is doing less, and your Lucene indexes are growing. This is a killer open source project, even if the API is a little weird.

Hadoop: Open Source MapReduce

I had a couple of people ask “why Google hasn’t open sourced our MapReduce?” They didn’t know about Hadoop:

Hadoop is a framework for running applications on large
clusters of commodity hardware. The Hadoop framework
transparently provides applications both reliability and data
motion. Hadoop implements a computational paradigm named
map/reduce, where the application is divided into many small
fragments of work, each of which may be executed or reexecuted
on any node in the cluster. In addition, it provides a
distributed file system that stores data on the compute nodes,
providing very high aggregate bandwidth across the cluster. Both
map/reduce and the distributed file system are designed so that
node failures are automatically handled by the framework.

The intent is to scale Hadoop up to handling thousand of
computers. Hadoop has been tested on clusters of 600
nodes.

Hadoop is a Lucene sub-project
that contains the distributed computing platform that was
formerly a part of Nutch. This
includes the Hadoop Distributed Filesystem (HDFS) and an
implementation of map/reduce.

For more information about Hadoop, please see the Hadoop wiki.

The great efforts of Christophe Bisciglia of the open source group revolve around UW classes where Hadoop is used in the curriculum.

It was great to see an email from Eddie O’Neil from BEA, announcing the Apache Beehive final 1.0 release.

I got to work on an Apache Beehive project, and was very impressed with certain parts and pieces. This may be a 1.0 release, but remember that it has been around for awhile in production projects such as Portal, and Workshop.

Nice work lads!

We are pleased to announce the v1.0 release of Apache Beehive!

This release provides the following major features:

• NetUI (Page Flow + JSP tag library)
• Controls framework
• System controls for accessing JDBC, JMS, and EJB resources
• Many bug fixes
• Significantly enhanced documentation
• Additional samples

Binary and source distributions can be downloaded via an Apache mirror from:

http://beehive.apache.org/releases/release-1.0.cgi

Documentation for v1.0 is here:

http://beehive.apache.org/docs/1.0

A few notes about this release:

* WSM and the web service control are not included as WSM has not passed the JSR 181 TCK yet. Both will be available in a subsequent release.
* v1.0 does not include XMLBeans due to licensing issues with the JSR
173 API JAR; however, XMLBeans can still be used with Beehive and is available from http://xmlbeans.apache.org
* Changes in v1.0 from v1.0m1 can be found here:
http://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&pid=10570&fixfor=12310123
* Thanks to everyone who submitted bug reports, patches, testing, and feedback. All of these contributed significantly to this release.
* Please send comments / questions about this release to [email protected] and development-related issues to [email protected]
* Bugs found in 1.0 should be filed in JIRA at http://issues.apache.org/jira/browse/BEEHIVE
* This version was built at SVN change 291405.

We welcome your feedback.

On to v.next…

The Beehive Team

SourceLabs has released a search engine called Swik which is:

Swik brings together concepts from wikipedia, delicious, and blogs to provide a service that is both open and structured.

More and more services are popping up to help out in this area, such as O’Reillys CodeZoo, and the interactive JDocs.

The fact that it is a wiki is interesting, and the tagging makes a lot of sense. As always, there are pros and cons to the “wiki” effect.

It is also good to see that “ajax” is a top term ;)