Manage your expenses via Email, SMS, iPhone, Twitter, Voice (Call and say your expense), IM (Yahoo, AIM, MSN), or Web.
Beanstalkd / Python Basic Tutorial
Comments(15)
(First install beanstalkd and pybeanstalk)
Beanstalkd is an in-memory queuing system. It supports named queues (called ‘tubes’), priorities, and delayed delivery of messages.
Terminology: a message is called a job, and queues are called tubes.
Let’s look at an example scenario. Say you want to create 2 tubes, one called “orders” and another called “emails”, place orders into the first tube (or queue) and emails into the second, and have different processes handle orders and emails.
You can create queues by simply naming and putting messages into them. On the producer side:
from beanstalk import serverconn
c = serverconn.ServerConn('localhost', 99988)
# put a message (or job) into the default queue:
c.put('first message, into default tube')
# now start using a named tube:
c.use('orders')
c.put('second message, into orders tube')
Now on the consumer:
from beanstalk import serverconn
c = serverconn.ServerConn('localhost', 99988)
# by default your connection will be listening on the 'default' tube.
# switch it to use the 'orders' tube.
# This should return the 'orders' message and ignore the 'default' message:
c.watchlist = ['orders']
j = c.reserve()
print j
# {'data': 'second message, into orders tube', 'jid': 39, 'bytes': 32, 'state': 'ok'}
You can similarly setup another consumer to listen on only the ‘emails’ tube, or both, or any other scenario you want.
Beanstalkd also supports priorities, with 0 being highest priority and higher numbers meaning lower priority. You define message priority with:
c.put('low priority message', pri=999 )
c.put('high priority message', pri=0 )
j = c.reserve()
print j
# {'data': 'high priority message', 'jid': 41, 'bytes': 21, 'state': 'ok'}
# the high priority message was delivered before the low priority message, even
# though the low priority message was first into the queue
The beanstalkd consumption model is to “reserve” a message (or job), process it, and then tell beanstalkd you’ve successfully dealt with the message so it can be thrown away. When you first get the job via c.reserve() you haven’t actually fully consumed it; you’ve just reserved it for processing.
What does this mean? Imagine a scenario where you reserve a message but your process dies before you have a chance to fully process it. Beanstalkd holds your message in reserve for a period of time, but since it hasn’t heard from you confirming you’ve successfully dealt with the message, it eventually removes the reservation and makes the message available once again for the next consumer to grab. This is a basic handshake between the consumer and the server to allow for some level of resiliency.
So once you’re finished dealing with the message you’ve reserved, be sure to “delete” it, letting the beanstalkd server know it can throw that message away:
j = c.reserve()
# do some processing with j
c.delete(j['jid'])
So there you have the basics. Let me know if you’re interested and I can cover a few more topics.
same question
：Adnan on November 7th, 2010
：I find no details how to backup and restore data from queue in case it fails
：pls help
This was a very helpful article. It wasn’t readily apparent how to use tubes from the pybeanstalk source example. And I didn’t even know about priorities. I added your link to my beanstalk post: http://www.saltycrane.com/blog/2008/10/installing-beanstalkd-and-pybeanstalk-ubuntu/.
Nice blog. You’ve written about some interesting topics.
Funny, I found your blog posts just a few minutes ago and subscribed, I wish I’d found it before I wrote mine
There’s also delayed message delivery which I haven’t covered here, maybe I’ll do a followup post.
I just published a preliminary version of a twisted client if you want to do something a bit more concurrently:
http://github.com/dustin/beanstalk-client-twisted
It ends up being core to many of my apps.
[...] Beanstalkd / Python Basic Tutorial – Standard Deviations (tags: beanstalkd python tutorial *important) [...]
can you use objects besides strings? ie will it automatically pickle and unpickle objects?
Hi,
Does this python script do automatic polling for new jobs?
Jason, the model is not exactly polling. The c.reserve() call blocks, waiting for a message to become available. So you don’t poll, you just reserve.
[...] There’s a good tutorial here: http://parand.com/say/index.php/2008/10/12/beanstalkd-python-basic-tut orial/ [...]
Hey all, no offense intended to the pybeanstalk maintainer, but it seems to me that beanstalkc is a better-maintained python beanstalk client these days. Check out this tutorial:
http://github.com/earl/beanstalkc/blob/master/TUTORIAL
And you can see the guts of the server protocol here for more context, if interested:
http://github.com/kr/beanstalkd/blob/master/doc/protocol.txt
I find no details how to backup and restore data from queue in case it fails
pls help
ah exactly the kind of example I was looking for. Although I was hoping we could see a complete working example with mail send. Most of the posts I found only mention the word email but do not apply any real world scenario.
Thank you for sharing the simple example.
Beanstalk versus Mecache…
Yes its the title but this isnt a smackdown. The fact is beanstalkd was inspired from memcache. However memcahe is considerably old and not very useful in many real world scenarios. That is why we have Queues. While memcahe could be made to behave as s…
oh and I must add your blog is beautiful. Dark tones and good colors. I am going to steal this theme but I dont like WP much
Hope Icanslap it into serendipity
When you start the beanstalkd process you can pass the -b flag along with a path for the log file. Beanstalkd will write the jobs in the queue into that log and you can resume from that if there’s a problem.