Comments on: Tokyo Cabinet Observations

By: Brian

Brian — Wed, 04 Nov 2009 14:10:05 +0000

Is RAM usage a function of the bnum parameter?

By: TokyoCabinet HDB slowdown :: Kelvin Tan - Lucene Solr Nutch Consultant

TokyoCabinet HDB slowdown :: Kelvin Tan - Lucene Solr Nutch Consultant — Sat, 10 Oct 2009 15:53:00 +0000

[...] experienced a similar phenomenon, and just stumbled upon http://parand.com/say/index.php/2009/04/09/tokyo-cabinet-observations/ , where I realized my problem was with bnum being too small (default of [...]

By: Kelvin Tan

Kelvin Tan — Sat, 10 Oct 2009 15:48:38 +0000

Thanks for the post and for everyone’s comments. I too was running into the slowdown with large hdbs, and I didn’t realize I needed to up bnum.

By: Fz

Fz — Thu, 25 Jun 2009 19:42:01 +0000

You can also try pyrant if you need to do network operations. Great library and clear.

http://code.google.com/p/pyrant/

By: Reid Lynch

Reid Lynch — Thu, 21 May 2009 14:31:28 +0000

I’ve only just started experimenting with Tokyo Cabinet and Tyrant, but it’s pretty clear you’ve got to up your hash bucket number (bnum) based on the number of records you anticipate. TC recommends a bnum of .5 – 4x the number of records. With the optimize function, this can be altered on an existing db.

I would also be very interested to see a comprehensive post somewhere detailing the all of the tuning and optimization parameters than can be done on Tokyo Cabinet databases.

By: Parand

Parand — Mon, 11 May 2009 20:04:18 +0000

Robin, I didn’t change any of the parameters (including bnum). If you learn of interesting parameters or different setups, if you could leave a comment or post something about it I’d appreciate it.

By: Robin Luckey

Robin Luckey — Mon, 11 May 2009 19:11:01 +0000

Did you try increasing the bnum parameter to increase the bucket size? I suspect that by default, the entire 2^64 key space is not available, and that you are in fact encountering key collisions.

Thanks for posting — I’m preparing to do some large TC experiments myself, and I’m curious about your results.

By: Parand

Parand — Fri, 01 May 2009 16:01:11 +0000

Thanks Didier. I’ve largely given up on the many-tc-instances approach, maybe I can revisit based the tuning parameters you describe.

By: Didier Spezia

Didier Spezia — Fri, 01 May 2009 15:38:15 +0000

If you partition your data, you may want to play with the tchdbsetxmsiz function (C API) to limit the amount of memory TC mmaps per database. The default is 64 Mb. You may want to reduce it in order to get more partitions. You can also have a look at the ulimit -a output and check the virtual memory, max memory size, and data seg size limits.

You may also want to review the tuning parameters to be set by tchdbtune (C API), and especially the bnum parameter. It should be set to 2 or 4 times the expected number of items to get good performance.

By: Parand

Parand — Wed, 15 Apr 2009 03:56:03 +0000

I have ~2.5 million distinct keys. (2^64)/2500000 = 7.37e12 . Not exactly running out of room for keys here. Math is our friend.

I’m not asking for extra features. I just want the data store to provide reasonable performance for larger data sizes. I don’t think handling large data sizes would be feature bloat or turn TC into postgres jr.

I’m not trying to dump on TC here – in fact I’m trying hard to use it in my project and have high hopes for it. I’m just curious as to the performance characteristics on large data – given the relative abundance of hash like data stores, I’d have imagined the newer ones would take care of scaling out of the box. My question was mainly to find out if there’s something obvious I’m doing wrong in my setup – which I still suspect there is.