http://theabstracttruth.wordpress.com/2008/05/06/twitter-versus-the-stock-market/

What about moving Twitter and Google AppEngine ? ;)

Tug

I agree that you could probably make data-grids work, and they have the advantage of being more flexible than MoMs, but recognize, you’ll have to build MoM functionality on top of the grid. I mean, this seems like a slam dunk excuse to use Java to me, simply because tried and true, market proven solutions exist. Otherwise, you have to build everything yourself, which you can do if you’re Google (GFS, BigTable, Map/Reduce, etc), but not everyone has the resources or time to do what Google does.

Roy i believe that data-grid/distributed-caching technologies are better alternative then many of the pure messaging or database solution as it already deals with distributed data management however it is still lacking the messaging end and therefore is not enough.

Sorry clicked the submit button too soon – i meant Ray not Roy.

Twitter is, fundamentally, a messaging system. Twitter was not designed as a messaging system, however. For expediency’s sake, Twitter was built with technologies and practices that are more appropriate to a content management system.

This is really the main point i.e. event driven architecture lend itself for scaling which means that with the proper design we wouldn’t be talking about twitter scaling in the first place as i already wrote in recent post about twitter scaling: http://natishalom.typepad.com/nati_shaloms_blog/2008/05/twitter-as-an-e.html

Now saying that is too simplistic IMO as what we have here a combination of events (messaging) and complex matching (data) – if you choose to use message driven design you’ll probably end up with complexity of managing state i.e. messaging doesn’t provide means for dealing with concurrent update, consistency, affinity etc. If your going to build your system with database centric design your going to end up writing lots of twiking to deal with event correlation, subscription model etc.

I believe that the separation between messaging and data in the first place is one of the bigger fallacies in distributed architecture i.e. i believe that messaging and data are the same and are always tightly coupled. In many cases you trigger a message to update a state change and when you update something you need to trigger various actions as a result of that update. Dealing with it by writing messaging and database at the application level often lead to complex architecture and non scalable design . Just imagine what it takes to do a simple task like sending a message and updating two or more entities of that change – we need to send an event to messaging queue, read it update the database, to avoid paritial failure we need to use transaction between the messaging systems and databse, to ensure high availability we need to maintain two separate clusters, to disciminate that update to multiple subscribers we need to perform reverse matching (based on the subscription) and send event to the relevant subsribers, normally this involves another broker service which is yet another tier in our already complex environment. Nither messaging or database centric design address well this type of associated networking.

I faced this problem in the past when i was involved in building of a b2b exchange. It was that realization that led me to TupleSpaces and to build a solution around JavaSpaces – JavaSpaces provides a mean to deal with distributed state (data-grid/caching) and messaging (events..) in one simple technology using only 4 verbs (write,read,take and notify).
JavaSpaces enables you to share state, subscribe to changes in that state, query the data etc.

A Space Based Architecture is a design pattern that was built to provide a solution for scaling out of stateful event driven applications using a combination of pattern from Messaging, data-grid/caching and parallel processing worlds.

Roy i believe that data-grid/distributed-caching technologies are better alternative then many of the pure messaging or database solution as it already deals with distributed data management however it is still lacking the messaging end and therefore is not enough.

A good starting point on Space Based Architecture can be found on wikipedia:
http://en.wikipedia.org/wiki/Space-based_architecture

It’s not really about programming languages, yet at the same time it is. Twitter is broke because it had the wrong design. However, I do believe the choice of tool encourages the design.

People use Ruby on Rails because it makes writing database backed sites a joy. RoR, as a platform, therefore, tends to produce database centric designs.

Erlang, for example, is fine tuned for event driven messaging, and therefore, if someone builds something with Erlang, it’s highly likely to leverage Erlang’s strengths.

RoR brings the allure of quick-and-cheap development, but at the same time, it encourages hacking from the bottom up, rather than design from the top-down. Granted, there are many problems with top-down design as well, but there’s a middle ground. I

Theoretically, Twitter could keep RoR around for a lot of the Web oriented form stuff, and replace the underlying messaging and track stuff with a pub/sub system with distribution and push updates.

The only advantage to using Java is that there are tons of high performance, feature filled MoMs. Mature fail over, transience, persistence, distribution/federation, management, etc. Plus, mature APIs for designing and deploying MoM components, you’ve got JMS, Message Driven Beans, and ESB.

For that reason, if I were CTO of Twitter, and looking to fix things in a hurry, I might buy either a top JMS MoM, or Coherence, and combine it with Java or Erlang. There is likely to be a push for Memcached usage, but Memcached doesn’t come close to Coherence in capability. Hell, IIRC, Coherence allows you to run complex cache queries and filters across the cache servers without pulling down the data.

What I find confusing is how many people think Twitter is doing something unprecedented, as an excuse for the downtime. Anyone who has worked with real time stock feeds knows the message rate dwarfs Twitter easily, and the queries/processing that people run against the data are also more complex than Twitter. As you point out, the way it’s handled is by replication.

One of the most charming things about email, SMS, and IM is the asynchronous nature of it. You usually don’t know how long it took for the message to get to you (unless you look at header time stamps), and you don’t care, unless it arrives spectacularly late. Asynchronous expectations are Twitter’s friend for scaling here.