I’ve always wanted to do benchmarking as Ilya suggests. Real production traffic is the best way to test. Good stuff.

I’m tempted by the beanstalkd use case as well – he uses his proxy to detect and route certain requests to an archiving mysql instead of to his beanstalkd instance. I’m leary of maintainability issues however – I’ve generally found indirection, particularly at wire protocol level, can quickly lead to hard to find bugs.

Something to experiment with at some point.

If you have enough traffic, at some point the cost of servers outweighs the cost of programmers

Absolutely true, which is why places like Yahoo and Google are among the last bastions of very skilled C/C++ programmers.

Of course I should mention: you are not at that point. You really aren’t. So for now ignore this quote.

A couple of benefits that I knew of but didn’t really appreciate until Alex started using beanstalkd in his application:

- Having a worker-pulls-jobs-from-queue model provides near optimal use of the machine and prevents overload and thrashing. Setup as many simultaneous workers as your resources can handle and let them go. You have a controlled number of workers, preventing thrashing, and your workers work continuously. You don’t have to worry about a load distribution strategy – workers pull jobs as fast as they can.

- Provisioning new workers into the system becomes trivial. Want to add another box into the mix? Just set it up and have the workers start pulling jobs. You don’t have to worry about registering the new box and getting it into the load distribution system – so long as it knows how to connect to the queue it can grab jobs. Alex commented that scaling his system is as easy as bringing up another virtual machine – as soon as it’s up it starts pulling jobs.

These are both very important operational benefits that I had largely ignored.

Interestingly no mention of an underlying distributed file system.

Via High Scalability.

Drizzle: A High-Performance Microkernel DBMS for Scale-Out Applications
Drizzle is a community-driven project based on the popular MySQL DBMS that is focused on MySQL’s original goals of ease-of-use, reliability and performance.

Headed up by Brian Aker, Director of Architecture at MySQL AB. Take a look at the MySQL Differences page and you’ll mostly see features removed and cleaned up, which is great. Designed for high levels of concurrency, targeted to “cloud” applications. Monty and Brian’s posts offer motivation for the project.

Something to keep an eye on.

So I’m wondering what the downside to aggressively going virtual is – why not make all servers virtual?

The main issue that comes to mind is performance, or the loss thereof. Presumably the performance of a virtual server is less than that of the same server running directly on the native OS.

Just how much of a performance difference is there, say in terms of per request latency and capacity, for a web server, a database server, and a cpu-bound heavy computation server, for any of the common virtualization systems (Xen, VMWare, etc)? I haven’t seen any good materials on this, so if you have knowledge or pointers please let me know.