Python's urllib.parse Module: Building and Breaking Down URLs

urlparse(): splitting a URL into its parts

urlparse() takes a full URL string and returns a named tuple with each logical component already separated out:

from urllib.parse import urlparse

result = urlparse("https://example.com/products?category=books&sort=price#reviews")
print(result.scheme)     # https
print(result.netloc)      # example.com
print(result.path)         # /products
print(result.query)        # category=books&sort=price
print(result.fragment)     # reviews

Because the result is a named tuple, each field is accessible by name (result.path) or by index, and the whole thing can be reassembled with urlunparse() after modifying one part — useful for taking an existing URL, swapping only the query string, and rebuilding it without touching the rest by hand. Trying to do this splitting with str.split() on "://" and "?" gets the common case right and then breaks the moment a URL has an unusual combination of components, such as a port number or embedded credentials in the netloc.

urlencode(): building a query string from a dict

Building a query string by hand with string concatenation is a common source of bugs the moment a value contains a space, an ampersand, or a non-ASCII character:

from urllib.parse import urlencode

params = {"q": "python tutorials", "page": 2, "sort": "date & relevance"}
print(urlencode(params))
# q=python+tutorials&page=2&sort=date+%26+relevance

urlencode() percent-encodes every value correctly, including turning the literal ampersand in the sort value into %26 so it isn't mistaken for a parameter separator. Passing doseq=True additionally allows a dict value to be a list, in which case the key is repeated once per item — the correct way to build a query string with a repeated parameter like tag=python&tag=web.

parse_qs() and parse_qsl(): reading query strings back

Going the other direction, parse_qs() turns a query string back into a dictionary — but every value comes back as a list, because HTTP query strings permit the same key to appear more than once:

from urllib.parse import parse_qs

parse_qs("q=python&page=2&tag=web&tag=tutorial")
# {'q': ['python'], 'page': ['2'], 'tag': ['web', 'tutorial']}

Code that expects a single value per key has to remember to index into the list (parsed["page"][0]), which trips people up the first time since most query strings in practice only ever have one value per key — right up until one doesn't. parse_qsl() returns the same data as a flat list of (key, value) tuples instead, which is more convenient when duplicate keys need to be preserved in order rather than grouped.

urljoin(): resolving relative URLs correctly

Combining a base URL with a relative link is not just string concatenation — the rules for ../, a leading /, and a relative path with no leading slash are all different, and browsers apply a well-defined resolution algorithm that urljoin() reproduces exactly:

from urllib.parse import urljoin

urljoin("https://example.com/docs/guide/", "../api/")
# 'https://example.com/docs/api/'

urljoin("https://example.com/docs/guide/", "/settings")
# 'https://example.com/settings'   -- leading slash means "from the root"

This is the correct tool any time code scrapes links out of an HTML page and needs to turn a relative href into an absolute URL — naive concatenation gets the common relative-path case right and then produces a subtly broken URL the first time a page uses a root-relative link or a ../ segment.

quote() and unquote(): percent-encoding

For encoding a single path segment or value rather than a whole query string, quote() and its inverse unquote() handle percent-encoding directly:

from urllib.parse import quote, unquote

quote("café & bar")     # 'caf%C3%A9%20%26%20bar'
unquote("caf%C3%A9%20%26%20bar")   # 'café & bar'

quote() defaults to leaving / unescaped, on the assumption that it's usually encoding a path rather than a single opaque segment; passing safe="" forces every reserved character, including slashes, to be percent-encoded, which is what's needed when a single path segment itself might legitimately contain a slash character that shouldn't be treated as a path separator.