A Guy Walks Into a NoSQL Bar [Director’s Cut]
or DataStax, Aerospike, Couchbase and Redis: Serving 1M Writes/Sec From Google Compute Engine
A couple of months ago I published a post over at the Google Cloud Platform blog called “A guy walks into a NoSQL bar and asks: how many servers to get 1 million ops. a second?” That particular post had gone through several editing cycles and I’m very happy with the final result. When I wrote that particular post, I abbreviated my cheeky story, but given Couchbase’s recent announcement about getting to 1M writes/sec with only 50 servers (!), I thought it would be relevant to share the full glory of my analogy here.
Note: the joke is still not very funny – suggestions for improvement are welcome!
Preface
Comparisons are as hard to do right as they are easy to do wrong and when the subjects of comparison are different the problem is only compounded. Therefore, when practicing comparative “advertising” I often feel that the results begotten should be interpreted in spirit rather be presented as numbers to quibble about. With that in mind, let me share a brief related story…
Prologue
A guy walks into a NoSQL bar looking quite upset. “Can I have a glass of red wine and a glass of white wine?” he asks the bartender. “Right away sir,” says the bartender, “but may I ask why the long face?” “I was in that other NoSQL bar further down the road, the Spiky Aerrow,” answers the patron, “and asked for that exact same order. The server there is known for his speed (some people swear he can do several things at once) as well as the amazing way he prepares drinks, so I was kind of expecting a show.”
The bartender, whose bar has been open for quite a few years, nods sympathetically and says, “I’m guessing he didn’t quite dazzle you?” “On the contrary,” replies the customer, it was an awesome show. To prepare my glass of red, that spectacular master of liquids uncorked no less than 30 bottles of wine in a fraction of a second! He then continued by whirlwinding between all these bottles and pouring a little wine from each into my glass. A lot of wine splashed around and spilled, given the speeds involved, but it was a heck of a show. And then, he did the same thing for the white… but with 50 bottles!”
Introduction
Some time ago, we at Redis set out to answer the question: “How many Google Compute Engine nodes do you need to serve 1 million write operations per second?” The question, besides sounding like a variation of the lightbulb joke, is quite open-ended in terms of defining the actual nature of the write operation based on previous work on the subject. Given our experience with cloud platforms in general, and particularly with Google’s, we were fairly confident that we’d need only a few so we set out to find out.
Benchmark Setup
To run the benchmark, we chose the biggest Google Compute Engine nodes currently available in us-central – the n1-highcpu-16 type – and we ended up using just two such servers to run our Redis Enterprise Cluster software (freely downloadable from here). We used another pair of servers – this time n1-highcpu-8 – to generate the load with memtier_benchmark using the following command line arguments:
memtier_benchmark -s 10.0.0.1 -p 6379 --ratio=1:1 --test-time=120 -d 100 -t 2 -c 50 --pipeline=50 --ratio=1:0
Benchmark Results
We ran the benchmark three times, first serving only reds reads, then serving only whites writes, and finally both in equal mix. Here are the results from those tests:
- For read-only operations, our Redis database provided throughput of 1.29M read operations per second at an average latency of 0.15 millisecond per operation.
- With the write-only load, the cluster’s measured throughput was at 1.14M operations per second at 0.36sec average latency per request.
- An equal mix of read and write operations gave a throughput of 1.16M operations per second at an average latency of 0.17msec per operation.
Conclusion
Redis needed only two Google Cloud Engine servers in the cluster to cross the 1M op/sec threshold.
Epilogue
Before the man can finish his story, the bartender had finished preparing the order (using just one bottle of white and one bottle of red, mind you). He placed the drinks alongside the bill in front of the customer, and said “sounds like an amazing show indeed, but I still don’t understand why you’re so unhappy.” “I’ll tell you why,” answered the man, “guess who had to pay for all 80 bottles of spilled wine?”
The barman smiled and said, “You should consider yourself lucky then, sir. I heard of a place further uptown called Dat Stacks where for every glass of white wine they charge for 300 bottles.”
Ok, so maybe this joke isn’t really that funny, but neither is paying too much for less-than-top-notch performance 🙂 With Redis, reaching and exceeding the 1 million operations per second mark doesn’t require a truckload of cloud servers – just one or two will do nicely.