Python's pickle Module: Serializing Objects to Bytes

What pickle actually does

The pickle module converts a Python object into a stream of bytes, and converts that stream back into an equivalent object later — possibly in a different process, or after being written to disk:

import pickle

data = {"name": "Ada", "scores": [91, 88, 76], "active": True}

with open("state.pkl", "wb") as f:
    pickle.dump(data, f)

with open("state.pkl", "rb") as f:
    restored = pickle.load(f)

print(restored == data)   # True

dumps()/loads() do the same thing entirely in memory, returning and accepting bytes rather than writing to a file object. Unlike JSON, pickle isn't limited to a handful of primitive types: it can serialize custom class instances, nested objects, sets, tuples, and even objects that reference each other in cycles, because pickle tracks object identity internally rather than flattening everything to a tree of primitives.

Pickle format has several protocol versions; pickle.HIGHEST_PROTOCOL in current Python gives the most compact, fastest encoding, and pickle.dump(obj, f, protocol=pickle.HIGHEST_PROTOCOL) is worth specifying explicitly if file size or speed matters, since the default protocol is chosen for backward compatibility rather than efficiency.

What can't be pickled

Pickle serializes state, not behaviour tied to a live process. Open file handles, network sockets, database connections, and running generator objects all fail to pickle, because there's nothing meaningful to reconstruct on the other end — a file descriptor number from one process means nothing in another. Lambda functions fail too, since pickle stores functions by reference (module path and qualified name), and a lambda has no stable name to look up:

pickle.dumps(lambda x: x + 1)   # PicklingError

Regular module-level functions and classes pickle fine as references — only the name is stored, and unpickling re-imports the module and looks the name up. That means the class or function must still exist, under the same name, in the same module, at load time. Renaming a class after old pickle files exist in production is a real, easy-to-hit failure mode.

The security warning you shouldn't skip

Unpickling data is not equivalent to parsing JSON. A pickle stream can include instructions that call arbitrary callables during reconstruction — the __reduce__ protocol that customises pickling can just as easily be abused to run os.system() or open a reverse shell the moment pickle.load() touches the data. This isn't a theoretical edge case; it's the documented, intended behaviour of the format, which is why the official docs carry an explicit warning against loading pickle data from any source you don't fully trust.

Practically: pickle is fine for your own cache files, your own inter-process queues, or data your own service produced and stores privately. It's the wrong tool the moment the bytes might have come from a user upload, a public API response, or anything crossing a trust boundary.

Customising pickling with __getstate__

Sometimes an object holds a reference that genuinely shouldn't (or can't) be serialized — a database connection cached as an instance attribute, say. Implementing __getstate__ and __setstate__ lets a class control exactly what gets saved and how it's rebuilt:

class Connection:
    def __init__(self, dsn):
        self.dsn = dsn
        self.handle = connect(dsn)   # not picklable

    def __getstate__(self):
        return {"dsn": self.dsn}   # only the reconnection info

    def __setstate__(self, state):
        self.dsn = state["dsn"]
        self.handle = connect(self.dsn)   # reconnect on load

This pattern shows up constantly in multiprocessing code, since worker processes communicate by pickling arguments and results across the process boundary — anything passed to a Pool method has to survive a pickle round trip, which is the usual reason an otherwise ordinary object suddenly raises PicklingError deep inside multiprocessing internals.

pickle vs json: picking the right one

JSON is text, language-agnostic, human-readable, and safe to parse from untrusted input (it can only produce data, never instructions). Pickle is binary, Python-specific, and can represent far more object types, but comes with the code-execution caveat above. The practical rule: reach for json whenever the data might be read by something other than your own trusted Python code, or stored/transmitted anywhere a stranger's bytes might end up substituted in; reach for pickle when you need to persist rich Python objects between runs of code you control end to end, such as caching a trained model object or checkpointing complex application state.