Side effects and generators in Python

Or How to Avoid Hard-to-Debug Problems

Take a look at this function:

def scale(matrix, factor):
    for row in matrix:
        for c in range(len(row)):
            row[c] = factor * row[c]
    return matrix

Now, do you see what's wrong with it? It is intended to return a matrix whose values are those in the input matrix multiplied by a factor, and it does that quite well. But it does something else that it should not do.

I've just spent way too much time hunting down a bug that was hiding in a function very much like this one. It was part of a class that generated binary files to be read and processed from a C program, and I was naturally inclined to search for the cause of the problems in the C code. Wrong.

Here' an example of the function in action:

scale([[1, 2],
       [3, 4]], 0.5)
0.5 1.0
1.5 2.0

So far so good.

Now imagine that somebody wants to run this function with a rather large 2D matrix. If this person is lazy enough he might think about generating the matrix as thus:

m = 4*[2*[2, 3]]
m
2 3 2 3
2 3 2 3
2 3 2 3
2 3 2 3

Now, what will happen when we apply the scale function to this matrix? Take a look:

scale(m, 0.5)
0.125 0.1875 0.125 0.1875
0.125 0.1875 0.125 0.1875
0.125 0.1875 0.125 0.1875
0.125 0.1875 0.125 0.1875

Most likely not the expected outcome. You can try to figure out why is it multiplying exactly four times each number and not 8, and if you do please do tell me1. But the root of the problem is clear: all rows are the same object. Modifying them in place is a very bad idea.

a = [[1, 2, 3, 4],
     [5, 6, 7, 8]]
b = scale(a, 0.5)
b
0.5 1.0 1.5 2.0
2.5 3.0 3.5 4.0

But…

a
0.5 1.0 1.5 2.0
2.5 3.0 3.5 4.0

So we intended to scale a to produce b, and ended up modifying a along the way. It is a bad idea to write a function whose output depends on the way the matrix fed to it is constructed.

The proper way to do it

The first thing you'd do in order to avoid this problem might be something like this:

def scale(matrix, factor):
    out = []
    for row in matrix:
        r = []
        for c in range(len(row)):
            r.append(factor * row[c])
        out.append(r)
    return out

Sure enough this works,

m = 4*[2*[2, 3]]
scale(m, 0.5)
1.0 1.5 1.0 1.5
1.0 1.5 1.0 1.5
1.0 1.5 1.0 1.5
1.0 1.5 1.0 1.5

but it's ugly and way too verbose. Much better —short and explicit— to use list comprehensions,

def scale(matrix, factor):
    return [[factor * v for v in row] for row in matrix]
m = 4*[2*[2, 3]]
scale(m, 0.5)
1.0 1.5 1.0 1.5
1.0 1.5 1.0 1.5
1.0 1.5 1.0 1.5
1.0 1.5 1.0 1.5

Using numpy

Another solution would be to use numpy, as in

import numpy as np
0.5 * np.array([[1, 2],
                [3, 4]])
0.5 1
1.5 2

It produces a copy on the output, it is safe, and it has the added benefit of being able to efficiently manage large matrices.

Learning more

Getting it right

No book that I know of will teach you not to make the mistakes above, but there is one that might help: The Structure and Interpretation of Computer Programs2, by Harold Abelson and Gerald Sussman. This book is also available for free, and it is one of the few truly great programming books. The Practice of Programming, by Brian Kernigan and Rob Pike, is also an excellent book full of good advice.

Python

There are plenty of good resources available online for free, so you probably don't need to buy anything. The one book I have found useful is the Python Essential Reference, by David M. Beazley: it is surprisingly easy to find in it what you are looking for.

Footnotes:

1

I did get an explanation (thanks Doetoe!): 2*[2,3] does not give you two shallow copies of the single list [2,3], while 2 * [ [2,3] ] does. Actually the python documentation states: "s * n, n * s: n shallow copies of s concatenated", so it is clear that 2 * [ [2,3] ] gives you two shallow copies, but to know that 2*[2,3] does NOT, you have to know that concatenating references to the same list gives rise to elementwise copies.

2

Disclaimer: I do get a cut from your Amazon purchase. Thank you very much for your support.

Juan Reyero Barcelona, 2011-12-03
 

blog comments powered by Disqus