The Catch

Python: Tips, Tricks and Idioms

by Michael

My programming language of preference is python for the simple reason that I feel I write better code faster with it then I do with other languages. However also has a lot of nice tricks and idioms to do things well. And partly as a reminder to myself to use them, and partly because I thought this might be of general interest I have put together this collection of some of my favourite idioms. I am also putting this on gist.github.com so that anyone that wants to contribute there own things can, and I will try and keep this post up to date.

enumerate

A fairly common thing to do is loop over a list while also keeping track of what index we are up to. Now we could use a count variable, but python gives us a nicer syntax for this with the enumerate() function.

set

set is a useful little data structure, it is kind of like a list but each value in it is unique. There are some useful operations, besides creating a list of unique items, that we can do with it. For example let try some different ways of validating lists of input.

Control statements

with

The with statement is a useful when accessing anything that supports the context management protocol. Which means open() for example. It basically makes sure that any set up and clean up code, such as closing files, is run without having to worry about it. So for example to open a file:

for ... else

This is a interesting bit of syntax, it allows you to run some code if the loop never reached the break statement. It basically replaces the need to keep a tracking variable for if you broke or not. Just looking over my code, here is a pseudo version of something I was doing.

Conditional Expressions

Python allows for conditional expressions, so instead of writing an if .. else with just one variable assignment in each branch, you can do the following:

This one of the reasons I really like python. That above is very readable, compared to the teneray operator that looks like a ? b : c that exits in other languages. It always confuses me.

List Comprension

List comprehensions are supposed to replace building a list by looping and calling append. Compare the following.

We can also make this more complicated, by adding in filtering or putting a conditional assignment in:

Generator expressions

List comprehensions have one possible problem, and that is they build the list in memory right away. If your dealing with big data sets, that can be a big problem, but even with small lists, it is still extra overhead that might not be needed. If you are only going to loop over the results once it has no gain in building this list. So if you can give up being able to index into the result, and do other list operations, you can use a generator expression, which uses very similar syntax, but creates a lazy object, that computes nothing until you ask for a value.

Generators using yield

Generator expressions are great, but sometimes you want something, with similar properties, but not limited by the syntax that generators use. Enter the yield statement. So for example the below will create generator is an infinite series of random numbers. So as long as we continue to keep asking for another random number, it will happy supply one.

Dictionary Comprehensions

One use for generators can be to build a dictionary, like in the first example below. This proved itself to be common enough that now there is even a new dictionary comprehension syntax for it. Both of these examples swap the keys and values of the dictionary.

zip

If you thought that generating an infinite number of random int was not that useful, well here I want to use it to show of another function that I like to use zip(). zip() takes a number of iterables, and joins the nth item of each into a tuple. So for example:

So basically, it prints out all the names with a random number (from our random number generator from before) next to a name. Notice that zip() will stop as soon as it reaches the end of the shortest iterable. If that is not desired, the itertools module, has one that goes till the end of the longest.

We could also do something similar to get a dict of each name mapped to a random number like this.

itertools

I mentioned itertools before. If you have not looked at it before, it is worth reading thought, plus at the end there is a whole section of recipes on how to use the module to create even more interesting operators on iterables.

Collections

Python comes with a module that contains a number of container data types called Collections. Though I only want to look at two right, now it also has three more called namedtuple(), deque (a linked list like structure), and OrderedDict.

defaultdict

This is a data type that I use a fair bit. One useful case is when you are appending to lists inside a dictionary. If the your using a dict() you would need to check if the key exists before appending, but with defaultdict this is not needed. So for example.

We could also count them like this.

Counter

But the last example is kind of redundant, in that Collections already contains a class for doing this, called Counter. In this case, I need to first extract the second item from each tuple, for which I can use a generator expression.

Another, maybe better, example might be counting all the different lines that appear in a file. It becomes very simple.

If you enjoyed this post, or found it useful, please do leave a comment or share it on twitter etc. If people find this useful I will try and do some follow up posts explaining some things in more detail and with extra examples.

Edit: if you have found this useful, but want more there is a great book Python Cookbook, Third edition by O'Reilly Media that has a whole lot more. If you want something simpler try Learning Python, 5th Edition.

If you found this interesting you might want to