Learning Python - Mark Lutz [276]
Note that while Python 3.0 provides a next(X) convenience built-in that calls the X.__next__() method of an object, other generator methods, like send, must be called as methods of generator objects directly (e.g., G.send(X)). This makes sense if you realize that these extra methods are implemented on built-in generator objects only, whereas the __next__ method applies to all iterable objects (both built-in types and user-defined classes).
Generator Expressions: Iterators Meet Comprehensions
In all recent versions of Python, the notions of iterators and list comprehensions are combined in a new feature of the language, generator expressions. Syntactically, generator expressions are just like normal list comprehensions, but they are enclosed in parentheses instead of square brackets:
>>> [x ** 2 for x in range(4)] # List comprehension: build a list
[0, 1, 4, 9]
>>> (x ** 2 for x in range(4)) # Generator expression: make an iterable
In fact, at least on a function basis, coding a list comprehension is essentially the same as wrapping a generator expression in a list built-in call to force it to produce all its results in a list at once: >>> list(x ** 2 for x in range(4)) # List comprehension equivalence [0, 1, 4, 9] Operationally, however, generator expressions are very different—instead of building the result list in memory, they return a generator object, which in turn supports the iteration protocol to yield one piece of the result list at a time in any iteration context: >>> G = (x ** 2 for x in range(4)) >>> next(G) 0 >>> next(G) 1 >>> next(G) 4 >>> next(G) 9 >>> next(G) Traceback (most recent call last): ...more text omitted... StopIteration We don’t typically see the next iterator machinery under the hood of a generator expression like this because for loops trigger it for us automatically: >>> for num in (x ** 2 for x in range(4)): ... print('%s, %s' % (num, num / 2.0)) ... 0, 0.0 1, 0.5 4, 2.0 9, 4.5 As we’ve already learned, every iteration context does this, including the sum, map, and sorted built-in functions; list comprehensions; and other iteration contexts we learned about in Chapter 14, such as the any, all, and list built-in functions. Notice that the parentheses are not required around a generator expression if they are the sole item enclosed in other parentheses, like those of a function call. Extra parentheses are required, however, in the second call to sorted: >>> sum(x ** 2 for x in range(4)) 14 >>> sorted(x ** 2 for x in range(4)) [0, 1, 4, 9] >>> sorted((x ** 2 for x in range(4)), reverse=True) [9, 4, 1, 0] >>> import math >>> list( map(math.sqrt, (x ** 2 for x in range(4))) ) [0.0, 1.0, 2.0, 3.0] Generator expressions are primarily a memory-space optimization—they do not require the entire result list to be constructed all at once, as the square-bracketed list comprehension does. They may also run slightly slower in practice, so they are probably best used only for very large result sets. A more authoritative statement about performance, though, will have to await the timing script we’ll code later in this chapter. Generator Functions Versus Generator Expressions Interestingly, the same iteration can often be coded with either a generator function or a generator expression. The following generator expression, for example, repeats each character in a string four times: >>> G = (c * 4 for c in 'SPAM') # Generator expression >>> list(G) # Force generator to produce all results ['SSSS', 'PPPP', 'AAAA', 'MMMM'] The equivalent generator function requires slightly more code, but as a multistatement function it will be able to code more logic and use more state information if needed: >>> def timesfour(S): # Generator function ... for c in S: ... yield c * 4 ... >>> G = timesfour('spam') >>> list(G) # Iterate automatically ['ssss', 'pppp', 'aaaa', 'mmmm'] Both expressions and functions support both automatic and manual iteration—the prior list call iterates