Nov 07

Refactoring tools are fantastic and overrated

Tech 137 Comments »

Refactoring tools are fantastic. I am an IDEA fan, and I think their refactoring support is second to none.

However, as great as it all seems, I think that refactoring tools can be overrated compared to other important factors.

For example, I was watching someone on a typical Java project with a particular stack, and watching their deploy cycle was making me cringe.

Sure they tried to do a lot out of the container, blah blah, but at some point they had to keep testing in the browser, which meant rebuilds of their project, and tomcat doing reloads for non-trivial changes.

The project was a medium size, and the cycle was painful to watch.

As I was watching this happen for a period of days, I compared to another current project where I would make changes and hit reload in the browser to see test.

I am saving time consistently, on a minute by minute basis.

On the other hand, it is quite infrequent for me to sit there and wish that I had “insert name of fancy refactoring”. When this happens it can be a pain, that is for sure. I hope that tools get better and better for non Java/static languages.

However, the more I thought about the comparison, the more I realised the the theoretical in me “man i love those refactorings. I just need them!” was dwarfed by the pragmatic reality of “man it is so painful to work in that stack, having these slowdowns CONSTANTLY”.

Nov 02

Good night, and good luck

Politics, TV / Movie 53 Comments »

I was fortunate to go to a screening of “Good night, and good luck”, hosted by the Capital Times, a fantastic local paper in Madison.

The crowd was full of people who were around in the McCarthy era, and their faces were stern, and you heard “deja vu” a few times.

The fear of that time was quite palpable.

It is no mistake that this film is out at this time. The parallels to the current fear-mongering, and lies, are quite apparant.

John Nichols, who writes for the Cap Times as well as The Nation, started off the evening talking about why we were all there.

I hope we can turn things around. I am hopeful, especially as friends from the right are acknowledging that the current regime does not match them as republicans, and more and more lies come out.

Nov 01

Sam Ruby and Beyond Java

Tech 32 Comments »

I agree with Bruce, and dissagree with Sam. I think that JRuby is crucial. It isn’t crucial from “does ODEO care”, but it is from “will it be adopted widely”.

Java is not a language, it is a platform. Many companies have a lot of investment in this platform. The JVM itself is pretty darn good. It can out perform a lot of C++ code (as it isn’t compiled for the lowest common denominator), and there are lot of great tools for debugging, scaling, yada yada.

So many of our clients are invested in this platform that it is hard to move over to Ruby. This is why Groovy got a lot of buzz at the beginning. It allowed you to get to a dynamic through the back door. It was the glue for the VM.

Many people would much prefer Ruby thought of course, and having JRuby is a huge asset. Now I can use it at large clients, it can work with the Java world in a nicer way than web services or Web services. rabble from Odeo complained about the libraries. Now you have a choice of easily using a Java library that may be superior for what you are doing, and mixing and matching.

If I want to write a quick script I would much prefer writing on a Java class and start using it compared to using darn web services :)

If Ruby is then on the JVM it will grow at a different level. It will also give you a lot more choices for libraries.

The best of all worlds. VMs are a nice abstraction. While we are talking about YARV and such we have a great VM on the other side of the fence. Rubys runtime is so darn poor, that running a top the well engineered JVMs from Sun, IBM, JRockit, …. is a great thing. It doesn’t mean that it is the only choice for Ruby, but its a great tool in the box.

I hope it can happen. JRuby is around 90% of the way, and needs a bit of help.

Sam, can’t you get IBM to help out? :)

Nov 01

Migrating data centers with zero downtime

Tech 202 Comments »

Last week I had to move a live heavily trafficed web application from a data center in south america, to a top class offering at Contegix.

I can’t say enough for the guys at Contegix. Normally I never look forward to talking to “the hosting guys”, but ever since we have worked with Mathew Porter and his team, it has become a pleasure. They really go above and beyond.

Anyway, back to the move. Some of you will probably think that this is a no-brainer, but a few too many people were talking about sitting there doing the DNS switch and “waiting for people to migrate over” so I thought I would put up our process for moving. Note that we will use the term “old server” to mean the soon-to-be old server, and the “new server” to be the soon to be live and current server :)

High level process

  • Get the app running on the old and new boxes
  • Configure the old server for redirect mode
  • Configure the new server for live mode
  • Do the switch
  • Watch and see

Get it running

Of course, the first step was to have migrated the application to the new machine, and run the battery of tests to make sure all is well.

Part of this will involve a test plan of things to check off.

You will also probably want to setup your /etc/hosts file to point to the new machine to do some testing to make sure that there are not any URL gliches “it worked as but not when it was live at!”.

Redirect Mode

The magic is simple. At some point you need to tell DNS that your server is actually at a new location. We all know that it can take roughly up to 24 hours or so for this to propogate through the internet. There are some tricks such as taking the TTL down a notch (and making sure that gets through a cycle), but it will still take time for the full propogation.

You only want people going to ONE machine though, and as soon as you flip the switch, you want it to be the new machine.

For this to happen, you simply make sure that anyone finding their way to the old machine gets proxied through to the correct place. You can’t just tell them “hey go here” unless you want to redirect them to an IP address, and that would only work if your server as the only guy on that IP, and that nothing had a domain in it, and that it didn’t have a domain piece.

In our case, our app relies a lot on the URL coming in to do different things (respond with different applications).

Anyway, to do the proxy thing, we simply go through each virtual host in our httpd.conf, put an entry in the /etc/hosts of the old machine to point to the new machine, and proxy over with mod_proxy:

RewriteEngine on
RewriteRule ^/(.*)$$1 [P]

We had to have in our /etc/hosts pointing to the new machine, as again, our app happened to care about the URL, so we had to proxy over to “”.

This form of proxy means that if someone connects to the old machine, that old machine connects to the new one and sends the data across. If you looked at the logs on your new machine you will see that each request is coming from the old machines ip address (so your web stats may be a bit weird for this day).

If you use mod_proxy to talk to a different domain, then you will want to make sure that content coming back is rewritten so it hits you on the backside.


RewriteRule ^/(.*)$$1 [P]
ProxyPassReverse /

Now html with can be rewritten as /test.html so it still works.

Configure the new server for live mode

Live mode is going to be the same as the httpd.conf on the old box, although you will probably have to do a s/oldip/newip/ on the config file(s).

Do the switch

When you are ready to do the switch (say, 3am on a Sunday?) you will want to have a process/script that is able to dump the latest data from the database on the old setup, migrate it to the new one, load in that data, and then restart httpd on the old box in redirect mode.

Now, you could bring down the old system for this to happen, and then you will get some downtime.

A set of our applications are very heavy read applications (typical in web apps), so rather than have downtime you can make the choice of:

  • Putting the app into a read only mode
  • Keep it running, and migrate the couple of records that you missed in the time that you restarted
  • Keep it running, and tough luck to the few people who may have added/modified/deleted records in the short window

Depending on the app, this is valid. If this is a community site, maybe we don’t care too much if Bob replied to a thread and he has to do it again later? :)

Now things are running fine on the new machine it is time to switch your DNS to point to the new boy.

Watch and see

Now the app is switched, you can have windows on both machines with tail -f on the logs.

Over time you will slowly see more different ip addresses on the new machine as the dns change propogates. You will also slowly see less and less traffic on the old machine. When you are done with traffic there you can take it out of commision.

If there are some major problems, you can quickly revert back to the old server :)