Learning Python - Mark Lutz [274]
The chief code difference between generator and normal functions is that a generator yields a value, rather than returning one—the yield statement suspends the function and sends a value back to the caller, but retains enough state to enable the function to resume from where it left off. When resumed, the function continues execution immediately after the last yield run. From the function’s perspective, this allows its code to produce a series of values over time, rather than computing them all at once and sending them back in something like a list.
Iteration protocol integration
To truly understand generator functions, you need to know that they are closely bound up with the notion of the iteration protocol in Python. As we’ve seen, iterable objects define a __next__ method, which either returns the next item in the iteration, or raises the special StopIteration exception to end the iteration. An object’s iterator is fetched with the iter built-in function.
Python for loops, and all other iteration contexts, use this iteration protocol to step through a sequence or value generator, if the protocol is supported; if not, iteration falls back on repeatedly indexing sequences instead.
To support this protocol, functions containing a yield statement are compiled specially as generators. When called, they return a generator object that supports the iteration interface with an automatically created method named __next__ to resume execution. Generator functions may also have a return statement that, along with falling off the end of the def block, simply terminates the generation of values—technically, by raising a StopIteration exception after any normal function exit actions. From the caller’s perspective, the generator’s __next__ method resumes the function and runs until either the next yield result is returned or a StopIteration is raised.
The net effect is that generator functions, coded as def statements containing yield statements, are automatically made to support the iteration protocol and thus may be used in any iteration context to produce results over time and on demand.
* * *
Note
As noted in Chapter 14, in Python 2.6 and earlier, iterable objects define a method named next instead of __next__. This includes the generator objects we are using here. In 3.0 this method is renamed to __next__. The next built-in function is provided as a convenience and portability tool: next(I) is the same as I.__next__() in 3.0 and I.next() in 2.6. Prior to 2.6, programs simply call I.next() instead to iterate manually.
* * *
Generator functions in action
To illustrate generator basics, let’s turn to some code. The following code defines a generator function that can be used to generate the squares of a series of numbers over time:
>>> def gensquares(N):
... for i in range(N):
... yield i ** 2 # Resume here later
...
This function yields a value, and so returns to its caller, each time through the loop; when it is resumed, its prior state is restored and control picks up again immediately after the yield statement. For example, when it’s used in the body of a for loop, control returns to the function after its yield statement each time through the loop:
>>> for i in gensquares(5): # Resume the function
... print(i, end=' : ') # Print last yielded value
...
0 : 1 : 4 : 9 : 16 :
>>>
To end the generation of values, functions either use a return statement with no value or simply allow control to fall off the end of the function body.
If you want to see what is going on inside