Generators, yield and Generator Expression in Python

In this Python advance article, we will study the generators in python along with python generator expression. How to use generator and yield keyword with Python list and other data types along with various examples in detail.

1. Why and When to use Generators in Python?

Generator functions allow us to declare a function that behaves like an Python Iterators, i.e. we can usee that function in a for loop.

The generator can be categorized into two parts: Generator functions and Generator objects.

Normally, it is easy to use Generators, but it’s a little tough to understand the generators at an instance.

As we discussed the iterator and generator are similar. Yet, generators are simple functions that return an iterable set of items, one at a time, in a special way.

In order to use the iterator method, we need to use the class with a method like  __next__() and __iter__(), which identifies internal state(a specific set of all its attributes’ value, the internal state is that a method applied to the object being in a defined) track, if there is not a single value which can be returned then the program raise StopIteration exception.

Seems too complex? Let’s talk simpler. A generator can be treated as a function type that returns an object in iteration rather than returning a single object value.

We use yield keyword instead of return keyword in the generator function.

Now the question is Why Generators? Think about a situation in which we have to process data that can not fit into the memory of our system, for example, Big Data or Infinite series. If we try to load this data at once and process it, it will fail because we do not have enough memory. Now to process this data we can use Generators so that we can divide the data into sensible and manageable chunks and process it step by step. That is one of the major use of Python Generators. If you have more ideas on where we can use Generators effectively, then comment below.

1.1. How to create a Generator in python?

The implementation and creation of a generator are too simple. It is the same as defining a normal function and replacing its return keyword with a yield keyword.

It is as easy as defining a normal function, but with a yield statement instead of a return statement.

To make a function into a generator function, a minimum of one yield keyword must be used.

Both yield and return performs a similar role of returning something from the function.

The major difference between the yield and return keywords is that function ends after encountering the return keyword whereas the function is paused after encountering the yield keyword, saves the current output, and continues from there when it is recalled.

1.2. Characteristics of a Generator

• The generator function contains yield keyword (one or more than one).
• The return keyword is not used, only yield keyword is used.
• After calling, an object(generally iterator) is returned however immediate execution is not initiated.
• Automatic initialization of methods like __iter__() and __next__() to iterate through values.
• After a successful yield, the function is paused and the system transfers the control to the caller.
• Local variables along with their current values are remembered between every successive call.
• Finally, when the function terminates, StopIteration is raised automatically on further calls.

Let’s take an example to understand the concept better.

# Generator function
def func():
  val = 5
  print('First value')
  yield val

  val -= 2
  print('Second value')
  yield val

  val -= 1
  print('Third value')
  yield val


x = func()
next(x)
next(x)
next(x)

Output
First value
Second value
Third value

The value of x is remembered between every call. The local variable is not erased after the yield of the function.

Furthermore, we can iterate the generator object only one time. For restarting the process, another generator object has to be created.

1.3. Benefits of Generators in Python

Some points are there which make generator essential to use. Let’s look at each point.

• Ease in use: If compared to the iterator, generators are easy and clear to implement in a precise manner.
• Efficiency of space: While using a normal function, in order to return the result, the entire task is executed in the memory which can exceed the size if too many and too large items are used. Implementing a generator for such sequences is preferred as it memory friendly.
• Infinite stream representation: Generators prove themselves as the best choice for representing an infinite data stream. As it is not possible to store infinite data streams into the memory, hence generators can be used to create a single item at an instance as it can represent an infinite data stream easily.
• Multiple generators can be used to pipeline a series of operations. 

2. How to use Loops with Generator in Python?

In the previous example, if we repeat the next(x) statement, it will continue and it will not stop even if  StopIteration is raised.

In order to make make sure that the generator ends in a terminating condition, we can use for loop.

#PREVIOUS EXAMPLE
def func():
  ...

x = func()    
next(x)
next(x)
next(x)
next(x)  # This line will raise error

Output
First value: 
Second value: 
Third value: 
Traceback (most recent call last):
  File "main.py", line 19, in 
    next(x)  # This line will raise error
StopIteration

Let’s use a for loop to check if it stops automatically or not.

# NEW METHOD
def string_reverse(input):
  for x in input.split(' '):
    yield x


print("The output from for loop: ")
for word in string_reverse("test generator with for loop"):
  print(word)

Output
The output from for loop: 
test
generator
with
for
loop

Let’s implement an example using the while loop to use the generator.

def double(val):
  for n in range(val):
    yield n+n


create = double(10)
while True:
  try:
    print("Value from next: ", next(create))
  except StopIteration:
    break

Output
Value from next:  0
Value from next:  2
Value from next:  4
Value from next:  6
Value from next:  8
Value from next:  10
Value from next:  12
Value from next:  14
Value from next:  16
Value from next:  18

3. What is Generator Expression?

Python allows users to create simple generators that can be created conveniently with the help of generator expressions. This brings ease to generator building.

Generator expressions can also prepare anonymous generator functions for the users.

A list and a generator expression have similar syntax. The list has square brackets and the generator expression has round parentheses.

A list pops out all the values in the list whereas a generator expression pops out a single value at a time after applying the operation as implemented.

value = [2, 4, 8, 16]

x = (i**3 for i in value)
print(next(x))
print(next(x))
print(next(x))
print(next(x))
print(next(x))

Output
8
64
512
4096
Traceback (most recent call last):
  File "main.py", line 9, in 
    print(next(x))
StopIteration

4. Conclusion

Finally, if we sum up, in this article we discussed the generator in python along with its different properties and their implementation, we have covered:

• Why and when to use generators and also how to create a Generator in python, its Characteristics and different uses.
• How to use Loops with Generator in python
• And lastly, What is Generator Expression?

Helpful Links

Please follow the Python tutorial series or the menu in the sidebar for the complete tutorial series.

Also for examples in Python and practice please refer to Python Examples.

Complete code samples are present on Github project.

Recommended Books

An investment in knowledge always pays the best interest. I hope you like the tutorial. Do come back for more because learning paves way for a better understanding

Do not forget to share and Subscribe.

Happy coding!! ?

Recommended -