Comments on: Tokyo Cabinet Observations

By: al

al — Mon, 27 Dec 2010 16:27:20 +0000

could anyone give an example of how to change bnum parameter?

By: Brian

Brian — Wed, 04 Nov 2009 14:10:05 +0000

Is RAM usage a function of the bnum parameter?

By: TokyoCabinet HDB slowdown :: Kelvin Tan - Lucene Solr Nutch Consultant

TokyoCabinet HDB slowdown :: Kelvin Tan - Lucene Solr Nutch Consultant — Sat, 10 Oct 2009 15:53:00 +0000

[...] experienced a similar phenomenon, and just stumbled upon http://parand.com/say/index.php/2009/04/09/tokyo-cabinet-observations/ , where I realized my problem was with bnum being too small (default of [...]

By: Kelvin Tan

Kelvin Tan — Sat, 10 Oct 2009 15:48:38 +0000

Thanks for the post and for everyone’s comments. I too was running into the slowdown with large hdbs, and I didn’t realize I needed to up bnum.

By: Fz

Fz — Thu, 25 Jun 2009 19:42:01 +0000

You can also try pyrant if you need to do network operations. Great library and clear.

http://code.google.com/p/pyrant/

By: Reid Lynch

Reid Lynch — Thu, 21 May 2009 14:31:28 +0000

I’ve only just started experimenting with Tokyo Cabinet and Tyrant, but it’s pretty clear you’ve got to up your hash bucket number (bnum) based on the number of records you anticipate. TC recommends a bnum of .5 – 4x the number of records. With the optimize function, this can be altered on an existing db.

I would also be very interested to see a comprehensive post somewhere detailing the all of the tuning and optimization parameters than can be done on Tokyo Cabinet databases.

By: Parand

Parand — Mon, 11 May 2009 20:04:18 +0000

Robin, I didn’t change any of the parameters (including bnum). If you learn of interesting parameters or different setups, if you could leave a comment or post something about it I’d appreciate it.

By: Robin Luckey

Robin Luckey — Mon, 11 May 2009 19:11:01 +0000

Did you try increasing the bnum parameter to increase the bucket size? I suspect that by default, the entire 2^64 key space is not available, and that you are in fact encountering key collisions.

Thanks for posting — I’m preparing to do some large TC experiments myself, and I’m curious about your results.

By: Parand

Parand — Fri, 01 May 2009 16:01:11 +0000

Thanks Didier. I’ve largely given up on the many-tc-instances approach, maybe I can revisit based the tuning parameters you describe.

By: Didier Spezia

Didier Spezia — Fri, 01 May 2009 15:38:15 +0000

If you partition your data, you may want to play with the tchdbsetxmsiz function (C API) to limit the amount of memory TC mmaps per database. The default is 64 Mb. You may want to reduce it in order to get more partitions. You can also have a look at the ulimit -a output and check the virtual memory, max memory size, and data seg size limits.

You may also want to review the tuning parameters to be set by tchdbtune (C API), and especially the bnum parameter. It should be set to 2 or 4 times the expected number of items to get good performance.