itertools in Python: Memory-Efficient Iteration Without Manual Loops

Why itertools exists

Writing custom loops to chain sequences, repeat values, or generate every possible pair of elements is repetitive and error-prone. The itertools module, part of the Python standard library, provides composable building blocks for iterator algebra. Because every function returns a lazy iterator, memory usage stays constant even for very large inputs.

Function	Group	Short description
count(start, step)	Infinite	Counts up from start indefinitely
cycle(iterable)	Infinite	Repeats the iterable endlessly
repeat(obj, n)	Infinite	Yields obj n times (or forever)
chain(*iterables)	Chaining	Concatenates iterables end-to-end
islice(it, stop)	Chaining	Lazy slice of an iterator
takewhile(pred, it)	Chaining	Yields while predicate is true
dropwhile(pred, it)	Chaining	Skips while predicate is true
groupby(it, key)	Grouping	Groups consecutive equal elements
product(*its)	Combinatoric	Cartesian product
permutations(it, r)	Combinatoric	All ordered r-length arrangements
combinations(it, r)	Combinatoric	All unordered r-length subsets
combinations_with_replacement(it, r)	Combinatoric	Combinations allowing repeats

Infinite iterators: count, cycle, repeat

These functions never raise StopIteration on their own. Always pair them with islice, zip, or a break statement to avoid infinite loops.

import itertools

# count: evenly-spaced numbers starting at 10, stepping by 2
for n in itertools.islice(itertools.count(10, 2), 5):
    print(n)
# 10 12 14 16 18

# cycle: round-robin assignment
colours = itertools.cycle(["red", "green", "blue"])
labels = [f"item-{i}" for i in range(6)]
paired = list(zip(labels, colours))
# [('item-0','red'),('item-1','green'),('item-2','blue'),
#  ('item-3','red'),('item-4','green'),('item-5','blue')]

# repeat: fill a list with the same default value
defaults = list(itertools.repeat(0, 5))
# [0, 0, 0, 0, 0]

Chaining and slicing: chain, islice, takewhile, dropwhile

chain connects multiple iterables as if they were one. chain.from_iterable is useful when you have a single iterable of iterables (flattening one level):

from itertools import chain, islice, takewhile, dropwhile

# chain: concatenate without creating a new list
combined = list(chain([1, 2], [3, 4], [5]))
# [1, 2, 3, 4, 5]

# chain.from_iterable: flatten one level
nested = [[1, 2], [3, 4], [5]]
flat = list(chain.from_iterable(nested))
# [1, 2, 3, 4, 5]

# islice: lazy slice — works on any iterator, not just sequences
first_three = list(islice(iter(range(1000)), 3))
# [0, 1, 2]

# takewhile / dropwhile
data = [2, 4, 6, 7, 8, 10]
print(list(takewhile(lambda x: x % 2 == 0, data)))  # [2, 4, 6]
print(list(dropwhile(lambda x: x % 2 == 0, data)))  # [7, 8, 10]

Grouping: groupby

groupby groups consecutive elements that share the same key. If you want every element with the same key to end up in the same group regardless of position, sort the input first:

from itertools import groupby

data = [("fruit","apple"),("fruit","pear"),("veggie","carrot"),("fruit","mango")]

# Sort by category first
data.sort(key=lambda x: x[0])

for category, items in groupby(data, key=lambda x: x[0]):
    names = [i[1] for i in items]
    print(f"{category}: {names}")
# fruit: ['apple', 'mango', 'pear']
# veggie: ['carrot']

Each call to groupby yields (key, group_iterator) pairs. Consume the group iterator immediately or convert it to a list; once you advance to the next group the previous iterator is exhausted.

Combinatoric iterators: product, permutations, combinations

These replace nested loops for exhaustive search problems:

from itertools import product, permutations, combinations

# product: Cartesian product of two sequences
for suit, rank in product(["H","D"], [1, 2, 3]):
    print(suit, rank)
# H 1, H 2, H 3, D 1, D 2, D 3

# permutations: all ordered arrangements of length r
print(list(permutations("ABC", 2)))
# [('A','B'),('A','C'),('B','A'),('B','C'),('C','A'),('C','B')]

# combinations: unordered subsets of length r
print(list(combinations("ABCD", 2)))
# [('A','B'),('A','C'),('A','D'),('B','C'),('B','D'),('C','D')]

The number of results grows rapidly. product generates n^r items; permutations generates n!/(n-r)!; combinations generates n!/(r!(n-r)!). Use islice to take only what you need.

Common recipes: flatten, pairwise, sliding window

The Python documentation includes a set of recipes built from itertools primitives. Python 3.10 added itertools.pairwise directly:

from itertools import pairwise, islice

# pairwise (Python 3.10+): overlapping pairs
steps = list(pairwise([1, 2, 3, 4, 5]))
# [(1,2),(2,3),(3,4),(4,5)]

# sliding window of width n
from collections import deque
def sliding_window(iterable, n):
    it = iter(iterable)
    window = deque(islice(it, n), maxlen=n)
    if len(window) == n:
        yield tuple(window)
    for item in it:
        window.append(item)
        yield tuple(window)

print(list(sliding_window(range(6), 3)))
# [(0,1,2),(1,2,3),(2,3,4),(3,4,5)]

The lazy evaluation principle

Every itertools function returns an iterator object. No element is computed until it is requested. This means you can compose a pipeline of transformations and apply it to a multi-gigabyte data stream without holding more than one element in memory at a time.

import itertools

# This pipeline never materialises an intermediate list
stream = range(10_000_000)
result = next(
    itertools.dropwhile(
        lambda x: x < 999_990,
        itertools.islice(stream, 1_000_000)
    )
)
print(result)  # 999990 — found without building any list

The rule of thumb: call list(), tuple(), or set() on an itertools result only when you genuinely need random access or need to iterate more than once. Otherwise let the iterator stay lazy.