Python's ast Module: Parsing Code Into a Syntax Tree

What ast.parse() actually produces

Before CPython executes anything, it parses source text into an abstract syntax tree: a hierarchy of nodes representing the grammatical structure of the code, independent of exact spacing or line breaks. ast.parse() exposes this same tree to Python code, before bytecode compilation happens:

import ast

source = "x = 1 + 2 * 3"
tree = ast.parse(source)
print(ast.dump(tree, indent=2))

The output shows a Module node containing an Assign node, whose value is a BinOp node combining an Add operator with a Constant(1) on the left and another BinOp (the multiplication) on the right — precedence is already resolved into nesting, exactly as the language grammar defines it. Every syntactic construct in Python has a corresponding node type: FunctionDef for function definitions, Call for function calls, If for conditionals, and dozens more, all documented in the ast module reference.

Walking the tree with NodeVisitor

Writing a manual recursive function to walk every possible node type is tedious and error-prone. ast.NodeVisitor handles the traversal for you: subclass it, define visit_NodeTypeName methods for the node types you care about, and call .generic_visit(node) inside them to continue into child nodes:

import ast

class FunctionNameCollector(ast.NodeVisitor):
    def __init__(self):
        self.names = []

    def visit_FunctionDef(self, node):
        self.names.append(node.name)
        self.generic_visit(node)   # keep walking into nested functions

source = '''
def outer():
    def inner():
        pass
    return inner

def top_level():
    pass
'''

tree = ast.parse(source)
collector = FunctionNameCollector()
collector.visit(tree)
print(collector.names)   # ['outer', 'inner', 'top_level']

Forgetting generic_visit() is the most common mistake: without it, the visitor stops descending the moment it hits a node type it has a handler for, silently missing anything nested inside.

A practical example: finding every function call

A common real task is auditing a codebase for calls to a specific function, without executing any of the code being scanned:

import ast

class CallFinder(ast.NodeVisitor):
    def __init__(self, target_name):
        self.target_name = target_name
        self.locations = []

    def visit_Call(self, node):
        if isinstance(node.func, ast.Name) and node.func.id == self.target_name:
            self.locations.append(node.lineno)
        self.generic_visit(node)

source = '''
eval(user_input)
result = compute()
data = eval(other_input)
'''

tree = ast.parse(source)
finder = CallFinder("eval")
finder.visit(tree)
print(finder.locations)   # [2, 4] -- line numbers of every eval() call

Every node carries lineno and col_offset attributes pointing back to its position in the original source, which is how tools report exact line numbers for warnings without needing to re-scan the raw text separately.

NodeTransformer: rewriting code programmatically

ast.NodeTransformer extends the same pattern to modify the tree: visit_* methods return a replacement node (or the original, or None to delete it), and after transforming, ast.unparse() converts the tree back into source text:

import ast

class ConstantDoubler(ast.NodeTransformer):
    def visit_Constant(self, node):
        if isinstance(node.value, int):
            return ast.Constant(value=node.value * 2)
        return node

tree = ast.parse("x = 5 + 10")
tree = ConstantDoubler().visit(tree)
ast.fix_missing_locations(tree)   # required after manually building/replacing nodes
print(ast.unparse(tree))            # x = 10 + 20

ast.fix_missing_locations() is necessary whenever you construct or splice in new nodes by hand, since they don't automatically inherit line and column information from their surroundings, and later stages that expect it will error without it.

Where this shows up: linters, formatters, and static analysis

This module is the actual foundation under tools that feel like magic from the outside. flake8 and pylint walk the AST looking for suspicious patterns — unused imports, shadowed variables, bare except: clauses. black and other formatters parse source into a tree, then regenerate formatted text from the tree's structure rather than pattern-matching on the original text, which is why they can normalise spacing and quote style without ever misinterpreting a string that happens to contain code-like text. Coverage tools instrument branches by walking the same tree to find every conditional and loop that needs a counter. None of this requires executing the target code at all, which is exactly why static analysis tools can safely run against code they don't trust.