Follow one request through your logs: structured records and a request ID in Python

Why plain sentences fail during an incident

Free-text log lines are written for a person reading one line at a time. During an incident you need the opposite: every line from one request, in order, without the lines from the hundred other requests running at the same moment. That needs two things the lines usually lack: a structure a program can parse and an identifier that ties the lines of one request together. The Python logging cookbook states the first need directly: some messages must be parsed by a program without complex regular expressions. The identifier is the second need, and it has to be set when the request enters the service and carried through everything the request does.

• Pick one real failing request and try to gather its lines by hand. What did you have to guess?

Emit one record per line

The cookbook shows one way to produce structured messages: a small object holds a message and keyword arguments, and its string form is the message followed by a delimiter and a JSON object, so the consumer splits on the delimiter and parses the JSON. A formatter that writes the whole record as one JSON object is another design you can choose. Either way, use the same field names everywhere, such as time, level, message, component, request identifier and error details, and write to standard output or one agreed destination. The cookbook also notes that the order of keys in the JSON may differ with the Python version, so a consumer must not rely on it.

• Write the field names down once and use them in every log call.
• Make error records carry the exception type and message as fields, not just inside a sentence.

Attach the request ID where the request starts

Filters in the logging package may modify the records passed to them, which lets one filter add the request identifier to every record without touching each log call. LoggerAdapter is another route that adds context to calls. For the identifier itself, the cookbook recommends context variables, which work with both threads and asyncio, over thread-local storage. Set the variable once when the request begins, read it in the filter and every line written while that request is handled carries it. The cookbook cautions against creating a logger for each connection, because loggers are not garbage collected and their number can grow without limit. Create the identifier at the edge of the service, and validate any identifier supplied by a caller before accepting it.

• Test with two requests running at the same time and check that no line carries the other request's identifier.
• Decide what happens to work the request hands to a background task, and test that too.

Where it breaks, and how the paid job is accepted

The cookbook says logging from several threads is fine but logging to one file from several processes is not supported, and recommends a queue handler and listener for handlers that can block, which matters if logs go to a network destination. Redaction needs the same attention: filter fields by name before they are written, and test it with invented secrets, because a log line is a copy of whatever it was given. The structured logging and alert job configures this in one place for one service, with the entry points you name, and accepts it on tests you can read: a captured sample of 100 consecutive lines must all parse, two concurrent requests must not share an identifier, invented secrets must not appear, and one agreed test failure must raise the alert. Send the language, where logs go and what you want alerted; never send log samples or code.