Edit File: architecture.rst
:mod:`email` Package Architecture ================================= Overview -------- The email package consists of three major components: Model An object structure that represents an email message, and provides an API for creating, querying, and modifying a message. Parser Takes a sequence of characters or bytes and produces a model of the email message represented by those characters or bytes. Generator Takes a model and turns it into a sequence of characters or bytes. The sequence can either be intended for human consumption (a printable unicode string) or bytes suitable for transmission over the wire. In the latter case all data is properly encoded using the content transfer encodings specified by the relevant RFCs. Conceptually the package is organized around the model. The model provides both "external" APIs intended for use by application programs using the library, and "internal" APIs intended for use by the Parser and Generator components. This division is intentionally a bit fuzzy; the API described by this documentation is all a public, stable API. This allows for an application with special needs to implement its own parser and/or generator. In addition to the three major functional components, there is a third key component to the architecture: Policy An object that specifies various behavioral settings and carries implementations of various behavior-controlling methods. The Policy framework provides a simple and convenient way to control the behavior of the library, making it possible for the library to be used in a very flexible fashion while leveraging the common code required to parse, represent, and generate message-like objects. For example, in addition to the default :rfc:`5322` email message policy, we also have a policy that manages HTTP headers in a fashion compliant with :rfc:`2616`. Individual policy controls, such as the maximum line length produced by the generator, can also be controlled individually to meet specialized application requirements. The Model --------- The message model is implemented by the :class:`~email.message.Message` class. The model divides a message into the two fundamental parts discussed by the RFC: the header section and the body. The `Message` object acts as a pseudo-dictionary of named headers. Its dictionary interface provides convenient access to individual headers by name. However, all headers are kept internally in an ordered list, so that the information about the order of the headers in the original message is preserved. The `Message` object also has a `payload` that holds the body. A `payload` can be one of two things: data, or a list of `Message` objects. The latter is used to represent a multipart MIME message. Lists can be nested arbitrarily deeply in order to represent the message, with all terminal leaves having non-list data payloads. Message Lifecycle ----------------- The general lifecycle of a message is: Creation A `Message` object can be created by a Parser, or it can be instantiated as an empty message by an application. Manipulation The application may examine one or more headers, and/or the payload, and it may modify one or more headers and/or the payload. This may be done on the top level `Message` object, or on any sub-object. Finalization The Model is converted into a unicode or binary stream, or the model is discarded. Header Policy Control During Lifecycle -------------------------------------- One of the major controls exerted by the Policy is the management of headers during the `Message` lifecycle. Most applications don't need to be aware of this. A header enters the model in one of two ways: via a Parser, or by being set to a specific value by an application program after the Model already exists. Similarly, a header exits the model in one of two ways: by being serialized by a Generator, or by being retrieved from a Model by an application program. The Policy object provides hooks for all four of these pathways. The model storage for headers is a list of (name, value) tuples. The Parser identifies headers during parsing, and passes them to the :meth:`~email.policy.Policy.header_source_parse` method of the Policy. The result of that method is the (name, value) tuple to be stored in the model. When an application program supplies a header value (for example, through the `Message` object `__setitem__` interface), the name and the value are passed to the :meth:`~email.policy.Policy.header_store_parse` method of the Policy, which returns the (name, value) tuple to be stored in the model. When an application program retrieves a header (through any of the dict or list interfaces of `Message`), the name and value are passed to the :meth:`~email.policy.Policy.header_fetch_parse` method of the Policy to obtain the value returned to the application. When a Generator requests a header during serialization, the name and value are passed to the :meth:`~email.policy.Policy.fold` method of the Policy, which returns a string containing line breaks in the appropriate places. The :meth:`~email.policy.Policy.cte_type` Policy control determines whether or not Content Transfer Encoding is performed on the data in the header. There is also a :meth:`~email.policy.Policy.binary_fold` method for use by generators that produce binary output, which returns the folded header as binary data, possibly folded at different places than the corresponding string would be. Handling Binary Data -------------------- In an ideal world all message data would conform to the RFCs, meaning that the parser could decode the message into the idealized unicode message that the sender originally wrote. In the real world, the email package must also be able to deal with badly formatted messages, including messages containing non-ASCII characters that either have no indicated character set or are not valid characters in the indicated character set. Since email messages are *primarily* text data, and operations on message data are primarily text operations (except for binary payloads of course), the model stores all text data as unicode strings. Un-decodable binary inside text data is handled by using the `surrogateescape` error handler of the ASCII codec. As with the binary filenames the error handler was introduced to handle, this allows the email package to "carry" the binary data received during parsing along until the output stage, at which time it is regenerated in its original form. This carried binary data is almost entirely an implementation detail. The one place where it is visible in the API is in the "internal" API. A Parser must do the `surrogateescape` encoding of binary input data, and pass that data to the appropriate Policy method. The "internal" interface used by the Generator to access header values preserves the `surrogateescaped` bytes. All other interfaces convert the binary data either back into bytes or into a safe form (losing information in some cases). Backward Compatibility ---------------------- The :class:`~email.policy.Policy.Compat32` Policy provides backward compatibility with version 5.1 of the email package. It does this via the following implementation of the four+1 Policy methods described above: header_source_parse Splits the first line on the colon to obtain the name, discards any spaces after the colon, and joins the remainder of the line with all of the remaining lines, preserving the linesep characters to obtain the value. Trailing carriage return and/or linefeed characters are stripped from the resulting value string. header_store_parse Returns the name and value exactly as received from the application. header_fetch_parse If the value contains any `surrogateescaped` binary data, return the value as a :class:`~email.header.Header` object, using the character set `unknown-8bit`. Otherwise just returns the value. fold Uses :class:`~email.header.Header`'s folding to fold headers in the same way the email5.1 generator did. binary_fold Same as fold, but encodes to 'ascii'. New Algorithm ------------- header_source_parse Same as legacy behavior. header_store_parse Same as legacy behavior. header_fetch_parse If the value is already a header object, returns it. Otherwise, parses the value using the new parser, and returns the resulting object as the value. `surrogateescaped` bytes get turned into unicode unknown character code points. fold Uses the new header folding algorithm, respecting the policy settings. surrogateescaped bytes are encoded using the ``unknown-8bit`` charset for ``cte_type=7bit`` or ``8bit``. Returns a string. At some point there will also be a ``cte_type=unicode``, and for that policy fold will serialize the idealized unicode message with RFC-like folding, converting any surrogateescaped bytes into the unicode unknown character glyph. binary_fold Uses the new header folding algorithm, respecting the policy settings. surrogateescaped bytes are encoded using the `unknown-8bit` charset for ``cte_type=7bit``, and get turned back into bytes for ``cte_type=8bit``. Returns bytes. At some point there will also be a ``cte_type=unicode``, and for that policy binary_fold will serialize the message according to :rfc:``5335``.