Side effects and generators in Python

Or How to Avoid Hard-to-Debug Problems

Take a look at this function:

def scale(matrix, factor):
    for row in matrix:
        for c in range(len(row)):
            row[c] = factor * row[c]
    return matrix

Now, do you see what's wrong with it? It is intended to return a matrix whose values are those in the input matrix multiplied by a factor, and it does that quite well. But it does something else that it should not do.

I've just spent way too much time hunting down a bug that was hiding in a function very much like this one. It was part of a class that generated binary files to be read and processed from a C program, and I was naturally inclined to search for the cause of the problems in the C code. Wrong.

Here' an example of the function in action:

scale([[1, 2],
       [3, 4]], 0.5)
0.5 1.0
1.5 2.0

So far so good.

Now imagine that somebody wants to run this function with a rather large 2D matrix. If this person is lazy enough he might think about generating the matrix as thus:

m = 4*[2*[2, 3]]
2 3 2 3
2 3 2 3
2 3 2 3
2 3 2 3

Now, what will happen when we apply the scale function to this matrix? Take a look:

scale(m, 0.5)

Most likely not the expected outcome. You can try to figure out why is it multiplying exactly four times each number and not 8, and if you do please do tell me1. But the root of the problem is clear: all rows are the same object. Modifying them in place is a very bad idea.

a = [[1, 2, 3, 4],
     [5, 6, 7, 8]]
b = scale(a, 0.5)



So we intended to scale a to produce b, and ended up modifying a along the way. It is a bad idea to write a function whose output depends on the way the matrix fed to it is constructed.

The proper way to do it

The first thing you'd do in order to avoid this problem might be something like this:

def scale(matrix, factor):
    out = []
    for row in matrix:
        r = []
        for c in range(len(row)):
            r.append(factor * row[c])
    return out

Sure enough this works,

m = 4*[2*[2, 3]]
scale(m, 0.5)

Using numpy

Another solution would be to use numpy, as in

import numpy as np
0.5 * np.array([[1, 2],
                [3, 4]])

It produces a copy on the output, it is safe, and it has the added benefit of being able to manage very big matrices.

Think generators

Generators make it much more difficult to produce ugly and wrong stuff like the above. This function, for example, will apply a function to anything that looks like a matrix of any number of dimensions, and will return a matrix with the same dimensions:

def amap(fn, ar):
    if getattr(ar[0], '__iter__', False):
        for v in ar:
            yield amap(fn, v)
        for v in ar:
            yield fn(v)
m = amap(lambda v: 0.3*v, [[1, 2],
                           [3, 4]])
2 3 2 3
2 3 2 3
2 3 2 3
2 3 2 3

It is more efficient if you need to deal with lots of data, as it will not do anything until you start requiring the output —you need to pull the data, as it were, when you need it. In this case, it did not do any multiplication until the assignment to m forced it.

Learning more

Getting it right

No book that I know of will teach you not to make the mistakes above, but there is one that might help: The Structure and Interpretation of Computer Programs2, by Harold Abelson and Gerald Sussman. This book is also available for free, and it is one of the few truly great programming books. The Practice of Programming, by Brian Kernigan and Rob Pike, is also an excellent book full of good advice.


There are plenty of good resources available online for free, so you probably don't need to buy anything. The one book I have found useful is the Python Essential Reference, by David M. Beazley: it is surprisingly easy to find in it what you are looking for.



I did get an explanation (thanks Doetoe!): 2*[2,3] does not give you two shallow copies of the single list [2,3], while 2 * [ [2,3] ] does. Actually the python documentation states: "s * n, n * s: n shallow copies of s concatenated", so it is clear that 2 * [ [2,3] ] gives you two shallow copies, but to know that 2*[2,3] does NOT, you have to know that concatenating references to the same list gives rise to elementwise copies.


Disclaimer: I do get a cut from your Amazon purchase. Thank you very much for your support.

Juan Reyero Barcelona, 2011-12-03

blog comments powered by Disqus