"Python Generator Expressions"

Thu 04 September 2008

I wasn't aware of generator expressions:

wwwlog     = open("access-log")
bytecolumn = (line.rsplit(None, 1)[1] for line in wwwlog)
bytes      = (int(x) for x in bytecolumn if x != '-')
print "Total", sum(bytes)

Similar to list comprehensions, but evaluated lazily. Voidspace describes it:

None of the generators are consumed until the final call to sum. As it iterates a line at a time (not keeping the log file in memory) it can handle huge log files - and as a bonus it runs faster than a typical solution with loops!

Neat.