Mailman - The GNU Mailing List Management System¶

This is GNU Mailman, a mailing list management system distributed under the terms of the GNU General Public License (GPL) version 3 or later. The name of this software is spelled “Mailman” with a leading capital ‘M’ but with a lower case second ‘m’. Any other spelling is incorrect.

Mailman is written in Python which is available for all platforms that Mailman is supported on, including GNU/Linux and most other Unix-like operating systems (e.g. Solaris, *BSD, MacOSX, etc.). Mailman is not supported on Windows, although web and mail clients on any platform should be able to interact with Mailman just fine.

The Mailman home page is:

http://www.list.org

and there is a community driven wiki at

http://wiki.list.org

For more information on Mailman, see the above web sites, or the documentation provided with this software.

Table of Contents¶

Mailman - The GNU Mailing List Management System¶

This is GNU Mailman, a mailing list management system distributed under the terms of the GNU General Public License (GPL) version 3 or later.

Mailman is written in Python, a free object-oriented programming language. Python is available for all platforms that Mailman is supported on, which includes GNU/Linux and most other Unix-like operating systems (e.g. Solaris, *BSD, MacOSX, etc.). Mailman is not supported on Windows, although web and mail clients on any platform should be able to interact with Mailman just fine.

Learn more about GNU Mailman in the Getting Started documentation.

Copyright¶

This file is part of GNU Mailman.

GNU Mailman is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

GNU Mailman is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with GNU Mailman. If not, see <http://www.gnu.org/licenses/>.

Spelling¶

The name of this software is spelled Mailman with a leading capital M but with a lower case second m. Any other spelling is incorrect. Its full name is GNU Mailman but is often referred colloquially as Mailman.

History¶

Mailman was originally developed by John Viega. Subsequent development (through version 1.0b3) was by Ken Manheimer. Further work towards the 1.0 final release was a group effort, with the core contributors being: Barry Warsaw, Ken Manheimer, Scott Cotton, Harald Meland, and John Viega. Version 1.0 and beyond have been primarily maintained by Barry Warsaw with contributions from many; see the ACKNOWLEDGMENTS file for details. Jeremy Hylton helped considerably with the Pipermail code in Mailman 2.0. Mailman 2.1 is primarily maintained by Mark Sapiro, with previous help by Tokio Kikuchi. Barry Warsaw is the lead developer on Mailman 3.

Help¶

The Mailman home page is:

http://www.list.org

with mirrors at:

http://www.gnu.org/software/mailman

http://mailman.sf.net

The community driven wiki (including the FAQ) is at:

http://wiki.list.org

Other help resources, such as on-line documentation, links to the mailing lists and archives, etc., are available at:

http://www.list.org/help.html

Bits and pieces¶

Mailman 3 is really a suite of 5 projects:

Core - the core message processing and delivery system, exposing a REST API for administrative control. Requires Python 3.4 or newer.

Postorius - the new web user interfaces built on Django.

HyperKitty - the new archiver, also built on Django.

mailman.client - a Python binding to the core’s REST API. Compatible with both Python 2 and Python 3.

Bundler - a convenient installer.

Getting started with GNU Mailman¶

Contact Us¶

Contributions of code, problem reports, and feature requests are welcome. Please submit bug reports on the Mailman bug tracker at https://gitlab.com/mailman/mailman/issues (you need to have a login on Gitlab to do so). You can also send email to the mailman-developers@python.org mailing list, or ask on IRC channel #mailman on Freenode.

Requirements¶

For the Core, Python 3.4 or newer is required. It can either be the default ‘python3’ on your $PATH or it can be accessible via the python3.4 binary. If your operating system does not include Python, see http://www.python.org for information about downloading installers (where available) and installing it from source (when necessary or preferred). Python 2 is not supported.

You may need some additional dependencies, which are either available from your OS vendor, or can be downloaded automatically from the Python Cheeseshop.

Documentation¶

The documentation for Mailman 3 is distributed throughout the sources. The core documentation (such as this file) is found in the src/mailman/docs directory, but much of the documentation is in module-specific places. A prebuilt HTML version of Mailman 3 documentation is available at pythonhosted.org, as is Postorius documentation. HyperKitty documentation is available at ReadTheDocs.

The Development Setup Guide is a recent step-by-step explanation of how to set up a complete Mailman 3 system including the Mailman 3 core and basic client API, Postorius, and HyperKitty.

Testing Mailman 3¶

To run the Mailman test suite, just use the tox command:

$ tox

tox creates a virtual environment (virtualenv) for you, installs all the dependencies into that virtualenv, and runs the test suite from that virtualenv. By default it does not use the –system-site-packages so it downloads everything from the Cheeseshop.

You do have access to the virtualenv, and you can use this to run individual tests, e.g.:

$ .tox/py34/bin/python -m nose2 -vv -P user

Use .tox/py34/bin/python -m nose2 –help for more options.

If you want to run the full test suite against the PostgreSQL database, set the database up as described in Setting up your database, then create a postgres.cfg file any where you want. This postgres.cfg file will contain the [database] section for PostgreSQL, e.g.:

[database]
class: mailman.database.postgresql.PostgreSQLDatabase
url: postgres://myuser:mypassword@mypghost/mailman

Then run the test suite like so:

$ MAILMAN_EXTRA_TESTING_CFG=/path/to/postgres.cfg tox -e pg

If you want to run an individual test against PostgreSQL, you would do it like so:

$ MAILMAN_EXTRA_TESTING_CFG=/path/to/postgres.cfg .tox/pg/bin/python -m nose2 -vv -P user

Building for development¶

To build Mailman for development purposes, you can create a virtual environment outside of tox. You need to have the virtualenv program installed, or you can use Python 3’s built-in pyvenv command.

First, create a virtual environment (venv). The directory you install the venv into is up to you, but for purposes of this document, we’ll install it into /tmp/mm3:

% pyvenv /tmp/mm3

Now, activate the virtual environment and set it up for development:

% source /tmp/mm3/bin/activate
% python setup.py develop

Sit back and have some Kombucha while you wait for everything to download and install.

Build the online docs by running:

% python setup.py build_sphinx

If setup.py fails to recognize the build_sphinx command, then just install Sphinx in your virtualenv:

% pip install sphinx

This will automatically add the build_sphinx command to setup.py, so just re-run the command.

Then visit:

build/sphinx/html/index.html

in your browser to start reading the documentation. Or you can just read the doctests by looking in all the ‘doc’ directories under the ‘mailman’ package. Doctests are documentation first, so they should give you a pretty good idea how various components of Mailman 3 work.

Once everything is downloaded and installed, you can initialize Mailman and get a display of the basic configuration settings by running:

$ mailman info -v

Running Mailman 3¶

What, you actually want to run Mailman 3? Oh well, if you insist. You will need to set up a configuration file to override the defaults and set things up for your environment. Mailman is configured using an “ini”-style configuration system.

src/mailman/config/schema.cfg defines the ini-file schema and contains documentation for every section and configuration variable. Sections that end in .template or .master are templates that must be overridden in actual configuration files. There is a default configuration file that defines these basic overrides in src/mailman/config/mailman.cfg. Your own configuration file will override those.

By default, all runtime files are put under a var directory in the current working directory.

Mailman searches for its configuration file using the following search path. The first existing file found wins.

-C config command line option

$MAILMAN_CONFIG_FILE environment variable

./mailman.cfg

~/.mailman.cfg

/etc/mailman.cfg

argv[0]/../../etc/mailman.cfg

Run the mailman info command to see which configuration file Mailman will use, and where it will put its database file. The first time you run this, Mailman will also create any necessary run-time directories and log files.

Try mailman --help for more details. You can use the commands mailman start to start the runner subprocess daemons, and of course mailman stop to stop them.

Postorius, a web UI for administration and subscriber settings, is being developed as a separate, Django-based project. For now, the most flexible means of configuration is via the command line and REST API.

Mailman Web UI¶

The Mailman 3 web UI, called Postorius, interfaces to core Mailman engine via the REST client API. It is expected that this architecture will make it possible for users with other needs to adapt the web UI, or even replace it entirely, with a reasonable amount of effort. However, as a core feature of Mailman, the web UI will emphasize usability over modularity at first, so most users should use the web UI described here.

The Archiver¶

In Mailman 3, the archivers are decoupled from the core engine. Instead, Mailman 3 provides a simple, standard interface for third-party archiving tools and services. For this reason, Mailman 3 defines a formal interface to insert messages into any of a number of configured archivers, using whatever protocol is appropriate for that archiver. Summary, search, and retrieval of archived posts are handled by a separate application.

A new archive UI called Hyperkitty, based on the notmuch mail indexer and Django, was prototyped at the PyCon 2012 sprint by Toshio Kuratomi. The Hyperkitty archiver is very loosely coupled to Mailman 3 core. In fact, any email application that speaks LMTP or SMTP will be able to use Hyperkitty.

Release notes¶

Mailman 3 is a fully rewritten code base. The developers believe it has sufficient functionality to provide full mailing list services. It should be ready for production use by experienced system developers, but it may not be easy to install or run by novices.

We expect it to be possible to migrate Mailman 2.1 mailing lists to Mailman 3, but sufficient caution, backups, and testing should be performed.

We expect it to be possible to run Mailman 3 and Mailman 2.1 together on the same systems, but you may need to be quite experienced with configuring your mail server and web infrastructure.

Mailman 3 may have bugs.

Mailman 3 is not yet feature complete with Mailman 2.1.

The documentation here describes the Mailman Core in great detail. Postorius, Hyperkitty, mailman.client, and the bundler are described and developed elsewhere.

More release notes are maintained on the Mailman wiki.

Setting up your database¶

Mailman uses the SQLAlchemy ORM to provide persistence of data in a relational database. By default, Mailman uses Python’s built-in SQLite3 database, however, SQLAlchemy is compatible with PostgreSQL and MySQL, among possibly others.

Currently, Mailman is known to work with either the default SQLite3 database, or PostgreSQL. (Volunteers to port it to other databases are welcome!). If you want to use SQLite3, you generally don’t need to change anything, but if you want Mailman to use PostgreSQL, you’ll need to set that up first, and then change a configuration variable in your /etc/mailman.cfg file.

Two configuration variables control which database Mailman uses. The first names the class implementing the database interface. The second names the URL for connecting to the database. Both variables live in the [database] section of the configuration file.

SQLite3¶

As mentioned, if you want to use SQLite3 in the default configuration, you generally don’t need to change anything. However, if you want to change where the SQLite3 database is stored, you can change the url variable in the [database] section. By default, the database is stored in the data directory in the mailman.db file. Here’s how you’d force Mailman to store its database in /var/lib/mailman/sqlite.db file:

[database]
url: sqlite:////var/lib/mailman/sqlite.db

PostgreSQL¶

First, you need to configure PostgreSQL itself. This Ubuntu article may help. Let’s say you create the mailman database in PostgreSQL via:

$ sudo -u postgres createdb -O $USER mailman

You would then need to set both the class and url variables in mailman.cfg like so:

[database]
class: mailman.database.postgresql.PostgreSQLDatabase
url: postgres://myuser:mypassword@mypghost/mailman

That should be it.

If you have any problems, you may need to delete the database and re-create it:

$ sudo -u postgres dropdb mailman
$ sudo -u postgres createdb -O myuser mailman

My thanks to Stephen A. Goss for his contribution of PostgreSQL support.

Database Migrations¶

Mailman uses Alembic to manage database migrations. Let’s say you change something in the models, what steps are needed to reflect that change in the database schema? You need to create and enter a virtual environment, install Mailman into that, and then run the alembic command. For example:

$ virtualenv -p python3 /tmp/mm3
$ source /tmp/mm3/bin/activate
$ python setup.py develop
$ alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m
  "<migration_name>"

This would create a new migration which would automatically be migrated to the database on the next run of Mailman. Note that the database needs to be in the older state so that Alembic can track the changes in the schema and autogenerate a migration. If you don’t have the database in the older state you can remove the –autogenerate flag in the above command. It would then create a new empty revision which you can edit manually to reflect your changes in the database schema.

People upgrading Mailman from previous versions need not do anything manually, as soon as a new migration is added in the sources, it will be automatically reflected in the schema on first-run post-update.

Note: When auto-generating migrations using Alembic, be sure to check the created migration before adding it to the version control. You will have to manually change some of the special data types defined in mailman.database.types. For example, mailman.database.types.Enum() needs to be changed to sa.Integer(), as the Enum type stores just the integer in the database. A more complex migration would be needed for UUID depending upon the database layer to be used.

Hooking up your mail server¶

Mailman needs to communicate with your MTA (mail transport agent or mail server, the software which handles sending mail across the Internet), both to accept incoming mail and to deliver outgoing mail. Mailman itself never delivers messages to the end user. It sends them to its immediate upstream MTA, which delivers them. In the same way, Mailman never receives mail directly. Mail from outside always comes via the MTA.

Mailman accepts incoming messages from the MTA using the Local Mail Transfer Protocol (LMTP) interface. Mailman can use other incoming transports, but LMTP is much more efficient than spawning a process just to do the delivery. Most open source MTAs support LMTP for local delivery. If yours doesn’t, and you need to use a different interface, please ask on the mailing list or on IRC.

Mailman passes all outgoing messages to the MTA using the Simple Mail Transfer Protocol (SMTP).

Cooperation between Mailman and the MTA requires some configuration of both. MTA configuration differs for each of the available MTAs, and there is a section for each one. Instructions for Postfix and Exim (v4) are given below. We would really appreciate a contribution of a configuration for Sendmail, and welcome information about other popular open source mail servers.

Configuring Mailman to communicate with the MTA is straightforward, and basically the same for all MTAs. In your mailman.cfg file, add (or edit) a section like the following:

[mta]
incoming: mailman.mta.postfix.LMTP
outgoing: mailman.mta.deliver.deliver
lmtp_host: 127.0.0.1
lmtp_port: 8024
smtp_host: localhost
smtp_port: 25
configuration: python:mailman.config.postfix

This configuration is for a system where Mailman and the MTA are on the same host.

Note that the modules that configure the communication protocol (especially incoming) are full-fledged Python modules, and may use these configuration parameters to automatically configure the MTA to recognize the list addresses and other attributes of the communication channel. This is why some constraints on the format of attributes arise (e.g., lmtp_host), even though Mailman itself has no problem with them.

The incoming and outgoing parameters identify the Python objects used to communicate with the MTA. The python: scheme indicates that the paths should be a dotted Python module specification. The deliver module used in outgoing should be satisfactory for most MTAs. The postfix module in incoming is specific to the Postfix MTA. See the section for your MTA below for details on these parameters.

lmtp_host and lmtp_port are parameters which are used by Mailman, but also will be passed to the MTA to identify the Mailman host. The “same host” case is special; some MTAs (including Postfix) do not recognize “localhost”, and need the numerical IP address. If they are on different hosts, lmtp_host should be set to the domain name or IP address of the Mailman host. lmtp_port is fairly arbitrary (there is no standard port for LMTP). Use any port convenient for your site. “8024” is as good as any, unless another service is using it.

smtp_host and smtp_port are parameters used to identify the MTA to Mailman. If the MTA and Mailman are on separate hosts, smtp_host should be set to the domain name or IP address of the MTA host. smtp_port will almost always be 25, which is the standard port for SMTP. (Some special site configurations set it to a different port. If you need this, you probably already know that, know why, and what to do, too!)

Mailman also provides many other configuration variables that you can use to tweak performance for your operating environment. See the src/mailman/config/schema.cfg file for details.

Postfix¶

Postfix is an open source mail server by Wietse Venema.

Mailman settings¶

You need to tell Mailman that you are using the Postfix mail server. In your mailman.cfg file, add the following section:

[mta]
incoming: mailman.mta.postfix.LMTP
outgoing: mailman.mta.deliver.deliver
lmtp_host: mail.example.com
lmtp_port: 8024
smtp_host: mail.example.com
smtp_port: 25

Some of these settings are already the default, so take a look at Mailman’s src/mailman/config/schema.cfg file for details. You’ll need to change the lmtp_host and smtp_host to the appropriate host names of course. Generally, Postfix will listen for incoming SMTP connections on port 25. Postfix will deliver via LMTP over port 24 by default, however if you are not running Mailman as root, you’ll need to change this to a higher port number, as shown above.

Basic Postfix connections¶

There are several ways to hook Postfix up to Mailman, so here are the simplest instructions. The following settings should be added to Postfix’s main.cf file.

Mailman supports a technique called Variable Envelope Return Path (VERP) to disambiguate and accurately record bounces. By default Mailman’s VERP delimiter is the + sign, so adding this setting allows Postfix to properly handle Mailman’s VERP’d messages:

# Support the default VERP delimiter.
recipient_delimiter = +

In older versions of Postfix, unknown local recipients generated a temporary failure. It’s much better (and the default in newer Postfix releases) to treat them as permanent failures. You can add this to your main.cf file if needed (use the postconf command to check the defaults):

unknown_local_recipient_reject_code = 550

While generally not necessary if you set recipient_delimiter as described above, it’s better for Postfix to not treat owner- and -request addresses specially:

owner_request_special = no

Transport maps¶

By default, Mailman works well with Postfix transport maps as a way to deliver incoming messages to Mailman’s LMTP server. Mailman will automatically write the correct transport map when its mailman aliases command is run, or whenever a mailing list is created or removed via other commands. To connect Postfix to Mailman’s LMTP server, add the following to Postfix’s main.cf file:

transport_maps =
    hash:/path-to-mailman/var/data/postfix_lmtp
local_recipient_maps =
    hash:/path-to-mailman/var/data/postfix_lmtp
relay_domains =
    hash:/path-to-mailman/var/data/postfix_domains

where path-to-mailman is replaced with the actual path that you’re running Mailman from. Setting local_recipient_maps as well as transport_maps allows Postfix to properly reject all messages destined for non-existent local users. Setting relay_domains means Postfix will start to accept mail for newly added domains even if they are not part of mydestination.

Note that if you are not using virtual domains, then relay_domains isn’t strictly needed (but it is harmless). All you need to do in this scenario is to make sure that Postfix accepts mail for your one domain, normally by including it in mydestination.

Postfix documentation¶

For more information regarding how to configure Postfix, please see the Postfix documentation at:

Exim¶

Exim 4 is an MTA maintained by the University of Cambridge and distributed by most open source OS distributions.

Mailman settings¶

Add or edit a stanza like this in mailman.cfg:

[mta]
# For all Exim4 installations.
incoming: mailman.mta.exim4.LMTP
outgoing: mailman.mta.deliver.deliver
# Typical single host with MTA and Mailman configuration.
# Adjust to your system's configuration.
# Exim happily works with the "localhost" alias rather than IP address.
lmtp_host: localhost
smtp_host: localhost
# Mailman should not be run as root.
# Use any convenient port > 1024.  8024 is a convention, but can be
# changed if there is a conflict with other software using that port.
lmtp_port: 8024
# smtp_port rarely needs to be set.
smtp_port: 25
# Exim4-specific configuration parameter defaults.  Currently empty.
configuration: python:mailman.config.exim4

For further information about these settings, see mailman/config/schema.cfg.

Exim4 configuration¶

The configuration presented below is mostly boilerplate that allows Exim to automatically discover your list addresses, and route both posts and administrative messages to the right Mailman services. For this reason, the mailman.mta.exim4 module ends up with all methods being no-ops.

This configuration is field-tested in a Debian “conf.d”-style Exim installation, with multiple configuration files that are assembled by a Debian-specific script. If your Exim v4 installation is structured differently, ignore the comments indicating location in the Debian installation.

# /etc/exim4/conf.d/main/25_mm3_macros
# The colon-separated list of domains served by Mailman.
domainlist mm_domains=list.example.net

MM3_LMTP_PORT=8024

# Assuming a typical source installation in /usr/local, with
# links to the Mailman bin directory and so on from MM3_HOME.
MM3_HOME=/usr/local/var/mailman
MM3_UID=list
MM3_GID=list

################################################################
# The configuration below is boilerplate:
# you should not need to change it.

# The path to the list receipt (used as the required file when
# matching list addresses)
MM3_LISTCHK=MM3_HOME/lists/${local_part}@${domain}

# /etc/exim4/conf.d/router/455_mm3_router
mailman3_router:
  driver = accept
  domains = +mm_domains
  require_files = MM3_LISTCHK
  local_part_suffix_optional
  local_part_suffix = -admin : \
     -bounces   : -bounces+* : \
     -confirm   : -confirm+* : \
     -join      : -leave     : \
     -owner     : -request   : \
     -subscribe : -unsubscribe
  transport = mailman3_transport

# /etc/exim4/conf.d/transport/55_mm3_transport
mailman3_transport:
  driver = smtp
  protocol = lmtp
  allow_localhost
  hosts = localhost
  port = MM3_LMTP_PORT

Troubleshooting¶

The most likely causes of failure to deliver to Mailman are typos in the configuration, and errors in the MM3_HOME macro or the mm_domains list. Mismatches in the LMTP port could be a cause. Finally, Exim’s router configuration is order-sensitive. Especially if you are being tricky and supporting Mailman 2 and Mailman 3 at the same time, you could have one shadow the other.

Exim 4 documentation¶

There is copious documentation for Exim. The parts most relevant to configuring communication with Mailman 3 are the chapters on the accept router and the LMTP transport. Unless you are already familiar with Exim configuration, you probably want to start with the chapter on how Exim receives and delivers mail.

Sendmail¶

The core Mailman developers generally do not use Sendmail, so experience is limited. Any and all contributions are welcome! The follow information from a post by Gary Algier <gaa@ulticom.com> may be useful as a starting point, although it describes Mailman 2:

I have it working fine. I recently replaced a very old implementation of sendmail and Mailman 2 on Solaris with a new one on CentOS 6. When I did so, I used the POSTFIX_ALIAS_CMD mechanism to automatically process the aliases. See:
https://mail.python.org/pipermail/mailman-users/2004-June/037518.html
In mm_cfg.py:
MTA='Postfix'
POSTFIX_ALIAS_CMD = '/usr/bin/sudo /etc/mail/import-mailman-aliases'
/etc/mail/import-mailman-aliases contains:
#! /bin/sh
/bin/cp /etc/mailman/aliases /etc/mail/mailman.aliases
/usr/bin/newaliases
In /etc/sudoers.d/mailman:
Cmnd_Alias IMPORT_MAILMAN_ALIASES = /etc/mail/import-mailman-aliases
apache ALL= NOPASSWD: IMPORT_MAILMAN_ALIASES
mailman ALL= NOPASSWD: IMPORT_MAILMAN_ALIASES
Defaults!IMPORT_MAILMAN_ALIASES !requiretty
In the sendmail.mc file I changed:
define(`ALIAS_FILE', `/etc/aliases')dnl
to:
define(`ALIAS_FILE', `/etc/aliases,/etc/mail/mailman.aliases')dnl
so that the Mailman aliases would be in a separate file.

The main issue here is that Mailman 2 expects to receive messages from the MTA via pipes, whereas Mailman 3 uses LMTP exclusively. Recent Sendmail does support LMTP, so it’s a matter of configuring a stock Sendmail. But rather than using aliases, it needs to be configured to relay to the LMTP port of Mailman.

Notes from the PyCon 2012 Mailman Sprint¶

These are notes from the Mailman sprint at PyCon 2012. They are not terribly well organized, nor fully fleshed out. Please edit and push branches to `Gitlab`_ or post patches to the Mailman bug tracker at <https://gitlab.com/mailman/mailman/issues>.

The intent of this document is to provide a view of Mailman 3’s workflow and structures from “eight miles high”.

Basic Messaging Handling Workflow¶

Mailman accepts a message via the LMTP protocol (RFC 2033). It implements a simple LMTP server internally based on the LMTP server provided in the Python standard library. The LMTP server’s responsibility is to parse the message into a tuple (mlist, msg, msgdata). If the parse fails (including messages which Mailman considers to be invalid due to lack of Message-Id as strongly recommended by RFC 2822 and RFC 5322), the message will be rejected, otherwise the parsed message and metadata dictionary are pickled, and the resulting message pickle added to one of the in, command, or bounce processing queues.

$digraph msgflow { rankdir = LR; node [shape=box, color=lightblue, style=filled]; msg [shape=ellipse, color=black, fillcolor=white]; lmtpd [label="LMTP\nSERVER"]; rts [label="Return\nto Sender"]; msg -> MTA [label="SMTP"]; MTA -> lmtpd [label="LMTP"]; lmtpd -> MTA [label="reject"]; lmtpd -> IN -> PIPELINE [label=".pck"]; IN -> rts; lmtpd -> BOUNCES [label=".pck"]; lmtpd -> COMMAND [label=".pck"]; }$

The in queue is processed by filter chains (explained below) to determine whether the post (or administrative request) will be processed. If not allowed, the message pickle is discarded, rejected (returned to sender), or held (saved for moderator approval – not shown). Otherwise the message is added to the pipeline (i.e. posting) queue. (Note that rejecting at this stage is not equivalent to rejecting during LMTP processing. This issue is currently unresolved.)

Each of the command, bounce, and pipeline queues is processed by a pipeline of handlers as in Mailman 2’s pipeline. (Some functions such as spam detection that were handled in the Mailman 2 pipeline are now in the filter chains.)

Handlers may copy messages to other queues (e.g., archive), and eventually posted messages for distribution to the list membership end up in the out queue for injection into the MTA.

The virgin queue (not depicted above) is a special queue for messages created by Mailman.

$digraph pipeline { node [shape=box, style=rounded, group=0] { "MIME\ndelete" -> "cleanse headers" -> "add headers" -> "calculate\nrecipients" -> "to digest" -> "to archive" -> "to outgoing" } node [shape=box, color=lightblue, style=filled, group=1] { rank=same; PIPELINE -> "MIME\ndelete" } { rank=same; "to digest" -> DIGEST } { rank=same; "to archive" -> ARCHIVE } { rank=same; "to outgoing" -> OUT } }$

Message Filtering¶

Once a message has been classified as a post or administrivia, rules are applied to determine whether the message should be distributed or acted on. Rules include things like “if the message’s sender is a non-member, hold it for moderation”, or “if the message contains an Approved header with a valid password, allow it to be posted”. A rule may also make no decision, in which case message processing is passed on to the next rule in the filter chain. The default set of rules looks something like this:

Configuration¶

Mailman 3 uses lazr.config, essentially an “ini”-style configuration format.

Each Runner’s configuration object knows whether it should be started when the Mailman daemon starts, and what queue the Runner manages.

Shell Commands¶

mailman: This is an ubercommand, with subcommands for all the various things admins might want to do, similar to Mailman 2’s mailmanctl, but with more functionality.

bin/master: The runner manager: starts, watches, stops the runner daemons.

bin/runner: Individual runner daemons. Each instance is configured with arguments specified on the command line.

User Model¶

A user represents a person. A user has an id and a display name, and optionally a list of linked addresses.

Each address is a separate object, linked to no more than one user.

A list member associates an address with a mailing list. Each list member has a id, a mailing list name, an address (which may be None, representing the user’s preferred address), a list of preferences, and a role such as “owner” or “moderator”. Roles are used to determine what kinds of mail the user receives via that membership. Owners will receive mail to list-owner, but not posts and moderation traffic, for example. A user with multiple roles on a single list will therefore have multiple memberships in that list, one for each role.

Roles are implemented by “magical, invisible” rosters which are objects representing queries on the membership database.

List Styles¶

Each list style is a named object. Its attributes are functions used to apply the relevant style settings to the mailing list at creation time. Since these are functions, they can be composed in various ways, to create substyles, etc.

Set up Postorius in five minutes¶

This is a quick guide for setting up a development environment to work on Mailman 3’s web UI, called Postorius. If all goes as planned, you should be done within 5 minutes. This has been tested on Ubuntu 11.04.

In order to download the components necessary you need to have the Git version control system installed on your system. Mailman requires Python 3.4, while mailman.client needs at least Python version 2.6.

It’s probably a good idea to set up a virtual Python environment using virtualenv. Here is a brief HOWTO. You would need two separate virtual environment one using Python version 2.6 or 2.7 (for Postorius and mailman.client) and other using Python version 3.4 (for Mailman core).

GNU Mailman 3¶

First download the latest revision of Mailman 3 from Gitlab.

$(py3) git clone git@gitlab.com:mailman/mailman.git

Install the Core:

$(py3) cd mailman
$(py3) python setup.py develop

If you get no errors you can now start Mailman:

$(py3) mailman start
$(py3) cd ..

At this point Mailman will not send nor receive any real emails. But that’s fine as long as you only want to work on the components related to the REST client or the web ui.

mailman.client (the Python bindings for Mailman’s REST API)¶

Now you should switch to the virtual environment running Python version 2.6 or 2.7. Download the client from Gitlab:

$(py2) git clone git@gitlab.com:mailman/mailmanclient.git

Install in development mode to be able to change the code without working directly on the PYTHONPATH.

$(py2) cd mailman.client
$(py2) python setup.py develop
$(py2) cd ..

Postorius¶

$(py2) git clone git@gitlab.com:mailman/postorius.git
$(py2) cd postorius
$(py2) python setup.py develop

Start the development server¶

Postorius is a Django app which can be used with any Django project. We have a project already developed which you can set up like this:

$(py2) git clone git@github.com:mailman/postorius_standalone.git
$(py2) cd postorius_standalone
$(py2) python manage.py syncdb
$(py2) python manage.py runserver

The last command will start the dev server on http://localhost:8000.

A note for MacOS X users (and possibly others running python 2.7)¶

Note: These paragraphs are struck-through on the Mailman wiki.

On an OS X 10.7 (Lion) system, some of these steps needed to be modified to use python2.6 instead of python. (In particular, bzr is known to behave badly when used python2.7 on OS X 10.7 at the moment – hopefully this will be fixed and no longer an issue soon.)

You will need to install the latest version of XCode on MacOS 10.7, which is available for free from the App Store. If you had a previous version of XCode installed when you upgraded to 10.7, it will no longer work and will not have automatically been upgraded, so be prepared to install again. Once you have it installed from the App Store, you will still need to go run the installer from /Applications to complete the installation.

Set up the archive ui in five minutes¶

Note

This document is way out of date. If you have access to the Web, the most recent explanation of setting up a full Mailman 3 system, including Postorius and HyperKitty, is the Development Setup Guide at FedoraHosted. If you must work offline, this document may be of some use, but be careful.

The hyperkitty application aims at providing an interface to visualize and explore Mailman archives.

This is a Django project.

Requirements¶

A mail archive in maildir format (no, you don’t need a running Mailman 3!) Eventually hyperkitty will support mbox format for backward compatibility with Pipermail, and zipped maildirs seem like a good idea to save space. Beware: Although you’d think that we would be able to manipulate the venerable mbox format safely and efficiently, that doesn’t seem to be the case. Maildir archives are strongly preferred, because they are more robust to program bugs (whether in Mailman, hyperkitty, or in the originating MUA!)
Django is the web framework that supports the UI.
bunch DOES WHAT?
The notmuch mail indexer is used to generate indexes (and requires Xapian).
hyperkitty itself, which is a UI, and not responsible for maintaining the message archive itself. (Since the archive is in maildir format, any modern MTA or MDA can build one for you.)

Get it running (under virtualenv):¶

It is generally a good idea to use virtualenv to create a stable environment for your Python applications.

Create the virtualenv:
```
% virtualenv mailman3
```
Activate the virtualenv:
```
% cd mailman3
% source bin/activate
```

You don’t have to use virtualenv, though, and if you don’t want to, just omit the preceding steps. Continue with these steps.

Install Django and dependencies:

% easy_install django
% easy_install bunch

Install notmuch – these are bindings that come with the notmuch C library. The easiest way is probably to install them for your OS vendor and then symlink them into the virtualenv similar to this:
```
% yum install -y python-notmuch
```
Note: on a multiarch system like Fedora, the directories may be lib64 rather than lib on 64 bit systems. Next:
```
% cd lib/python2.7/site-packages
% ln -s /usr/lib/python2.7/site-packages/notmuch .
```

Note: this is the version of notmuch I tested with; others may work:

% ln -s /usr/lib/python2.7/site-packages/notmuch-0.11-py2.7.egg-info .

Install the hyperkitty sources:

% git clone https://github.com/hyperkitty/kittystore.git
% git clone https://github.com/hyperkitty/hyperkitty.git
% git clone https://github.com/hyperkitty/hyperkitty_standalone.git

Running hyperkitty¶

Start it:
```
% cd hyperkitty
```
Put the static content where it should be:
```
% python manage.py collectstatic
```
Run the Django server:
```
% python manage.py runserver
```

Developing Mailman¶

The following documentation is generated from the internal developer documentation. This documentation is also used by the test suite. Another good source of architectural information is available in the chapter written by Barry Warsaw for the Architecture of Open Source Applications.

For now, this will have to suffice as an overview of the Mailman system.

Object model¶

Every major component of the system is defined in an interface. Look through src/mailman/interfaces for an understanding of the system components. Mailman objects which are stored in the database, are defined by model classes. Objects such as mailing lists, users, members, and addresses are primary objects within the system.

Obviously, the mailing list is the central object which holds all the configuration settings for a particular mailing list. A mailing list is associated with a domain, and all mailing lists are managed by the mailing list manager.

Users represent people, and have a user id and a display name. Users are linked to addresses which represent a single email address. One user can be linked to many addresses, but an address is only linked to one user. Addresses can be verified or not verified. Mailman will deliver email only to verified addresses.

Users and addresses are managed by the user manager.

A member is created when a user subscribes to a mailing list, either via their preferred address or explicitly via one of their linked and verified addresses. Members link an address to a mailing list, and represent regular members, digest members, list owners, and list moderators. Members can even represent non-members (i.e. people not yet subscribed to the mailing list) for various moderation purposes.

Process model¶

Messages move around inside the Mailman system by way of queue directories managed by the switchboard. For example, when a message is first received by Mailman, it is moved to the in (for “incoming”) queue. During the processing of this message, it – or copies of it – may be moved to other queues such as the out queue (for outgoing email), the archive queue (for sending to the archivers), the digest queue (for composing digests), etc.

A message in a queue is represented by a single file, a .pck file. This file contains two objects, serialized as Python pickles. The first object is the message being processed, already parsed into a more efficient internal representation. The second object is a metadata dictionary that records additional information about the message as it is being processed.

.pck files only exist for messages moving between different system queues. There is no .pck file for messages while they are actively being processed.

Each queue directory is associated with a runner process which wakes up every so often. When the runner wakes up, it examines all the .pck files in FIFO order, deserializing the message and metadata objects, processing them. If the message needs further processing in a different queue, it will be re-serialized back into a .pck file. If not (e.g. because processing of the message is complete), then no .pck file is written.

The Mailman system uses a few other runners which don’t process messages in a queue. You can think of these as fairly typical server process, and examples include the LMTP server, and the HTTP server for processing REST commands.

All of the runners are managed by a master watcher process. When you type mailman start you are actually starting the master. Based on configuration options, the master will start the appropriate runners as subprocesses, and it will watch for the clean exiting of these subprocesses when mailman stop is called.

Rules and chains¶

When a message is first received for posting to a mailing list, Mailman processes the message to determine whether the message is appropriate for the mailing list. If so, it approves the message and it gets posted. Mailman can also discard the message so that no further processing occurs. Mailman can also reject the message, bouncing it back to the original sender, usual with some indication of why the message was rejected. Mailman can also hold the message for moderator approval.

Moderation is the phase of processing that determines which of the above four dispositions will occur for the newly posted message. Moderation does not generally change the message, but it may record information in the metadata dictionary. Moderation is performed by the in queue runner.

Each step in the moderation phase is performed by applying a rule to the message and asking whether the rule hits or misses. Each rule is linked to an action which is taken if the rule hits (i.e. matches). If the rule misses (i.e. doesn’t match), then the next rule is tried. All of the rule/action links are strung together sequentially into a chain, and every mailing list has a start chain where rule processing begins.

Actually, every mailing list has two start chains, one for regular postings to the mailing list, and another for posting to the owners of the mailing list.

To recap: when a message comes into Mailman for posting to a mailing list, the incoming runner finds the destination mailing list, determines whether the message is for the entire list membership, or the list owners, and retrieves the appropriate start chain. The message is then passed to the chain, where each link in the chain first checks to see if its rule matches, and if so, it executes the linked action. This action is usually one of the typical accept, reject, discard, and hold, but other actions are possible.

As you might imagine, you can write new rules, compose them into new chains, and configure a mailing list to use your custom chain when processing the message during the moderation phase.

Pipeline of handlers¶

Once a message is accepted for posting to the mailing list, the message is usually modified in a number of different ways. For example, some message headers may be added or removed, some MIME parts might be scrubbed, added, or rearranged, and various informative headers and footers may be added to the message.

The process of preparing the message for the list membership (as well as the digests, archivers, and NNTP) falls to the pipeline of handlers managed by the pipeline queue.

The pipeline of handlers is similar to the processing chain, except here, a handler can make any modifications to the message it wants, and there is no rule decision or action. The message and metadata simply flow through a sequence of handlers arranged in a named pipeline. Some of the handlers modify the message in ways described above, and others copy the message to the outgoing, NNTP, archiver, or digester queues.

As with chains, each mailing list has two pipelines, one for posting to the list membership, and the other for posting to the list’s owners.

Of course, you can define new handlers, compose them into new pipelines, and change a mailing list’s pipelines.

Other bits and pieces¶

There are lots of other pieces to the Mailman puzzle, such as the REST API, the set of core functionality (logging, initialization, event handling, etc.), mailing list styles, the API for integrating external archivers and mail servers. The database layer is an critical piece, and Mailman has an extensive set of command line commands, and email commands.

Mailman modules¶

These documents are generated from the doctest suite.

Email addresses¶

Addresses represent a text email address, along with some meta data about those addresses, such as their registration date, and whether and when they’ve been validated. Addresses may be linked to the users that Mailman knows about. Addresses are subscribed to mailing lists though members.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)

Creating addresses¶

Addresses are created directly through the user manager, which starts out with no addresses.

>>> dump_list(address.email for address in user_manager.addresses)
*Empty*

Creating an unlinked email address is straightforward.

>>> address_1 = user_manager.create_address('aperson@example.com')
>>> dump_list(address.email for address in user_manager.addresses)
aperson@example.com

However, such addresses have no real name.

>>> print(address_1.display_name)
<BLANKLINE>

You can also create an email address object with a real name.

>>> address_2 = user_manager.create_address(
...     'bperson@example.com', 'Ben Person')
>>> dump_list(address.email for address in user_manager.addresses)
aperson@example.com
bperson@example.com
>>> dump_list(address.display_name for address in user_manager.addresses)
<BLANKLINE>
Ben Person

The str() of the address is the RFC 2822 preferred originator format, while the repr() carries more information.

>>> print(str(address_2))
Ben Person <bperson@example.com>
>>> print(repr(address_2))
<Address: Ben Person <bperson@example.com> [not verified] at 0x...>

You can assign real names to existing addresses.

>>> address_1.display_name = 'Anne Person'
>>> dump_list(address.display_name for address in user_manager.addresses)
Anne Person
Ben Person

These addresses are not linked to users, and can be seen by searching the user manager for an associated user.

>>> print(user_manager.get_user('aperson@example.com'))
None
>>> print(user_manager.get_user('bperson@example.com'))
None

You can create email addresses that are linked to users by using a different interface.

>>> user_1 = user_manager.create_user(
...     'cperson@example.com', u'Claire Person')
>>> dump_list(address.email for address in user_1.addresses)
cperson@example.com
>>> dump_list(address.email for address in user_manager.addresses)
aperson@example.com
bperson@example.com
cperson@example.com
>>> dump_list(address.display_name for address in user_manager.addresses)
Anne Person
Ben Person
Claire Person

And now you can find the associated user.

>>> print(user_manager.get_user('aperson@example.com'))
None
>>> print(user_manager.get_user('bperson@example.com'))
None
>>> user_manager.get_user('cperson@example.com')
<User "Claire Person" (...) at ...>

Deleting addresses¶

You can remove an unlinked address from the user manager.

>>> user_manager.delete_address(address_1)
>>> dump_list(address.email for address in user_manager.addresses)
bperson@example.com
cperson@example.com
>>> dump_list(address.display_name for address in user_manager.addresses)
Ben Person
Claire Person

Deleting a linked address does not delete the user, but it does unlink the address from the user.

>>> dump_list(address.email for address in user_1.addresses)
cperson@example.com
>>> user_1.controls('cperson@example.com')
True
>>> address_3 = list(user_1.addresses)[0]
>>> user_manager.delete_address(address_3)
>>> dump_list(address.email for address in user_1.addresses)
*Empty*
>>> user_1.controls('cperson@example.com')
False
>>> dump_list(address.email for address in user_manager.addresses)
bperson@example.com

Registration and verification¶

Addresses have two dates, the date the address was registered on and the date the address was validated on. The former is set when the address is created, but the latter must be set explicitly.

>>> address_4 = user_manager.create_address(
...     'dperson@example.com', 'Dan Person')
>>> print(address_4.registered_on)
2005-08-01 07:49:23
>>> print(address_4.verified_on)
None

The verification date records when the user has completed a mail-back verification procedure. It takes a datetime object.

>>> from mailman.utilities.datetime import now
>>> address_4.verified_on = now()
>>> print(address_4.verified_on)
2005-08-01 07:49:23

The address shows the verified status in its representation.

>>> address_4
<Address: Dan Person <dperson@example.com> [verified] at ...>

An event is triggered when the address gets verified.

>>> saved_event = None
>>> address_5 = user_manager.create_address(
...     'eperson@example.com', 'Elle Person')
>>> def save_event(event):
...     global saved_event
...     saved_event = event
>>> from mailman.testing.helpers import event_subscribers
>>> with event_subscribers(save_event):
...     address_5.verified_on = now()
>>> print(saved_event)
<AddressVerificationEvent eperson@example.com 2005-08-01 07:49:23>

An event is also triggered when the address is unverified. In this case, check the event’s address’s verified_on attribute; if this is None, then the address is being unverified.

>>> with event_subscribers(save_event):
...     address_5.verified_on = None
>>> print(saved_event)
<AddressVerificationEvent eperson@example.com unverified>
>>> print(saved_event.address.verified_on)
None

Case-preserved addresses¶

Technically speaking, email addresses are case sensitive in the local part. Mailman preserves the case of addresses and uses the case preserved version when sending the user a message, but it treats addresses that are different in case equivalently in all other situations.

>>> address_6 = user_manager.create_address(
...     'FPERSON@example.com', 'Frank Person')

The str() of such an address prints the RFC 2822 preferred originator format with the original case-preserved address. The repr() contains all the gory details.

>>> print(str(address_6))
Frank Person <FPERSON@example.com>
>>> print(repr(address_6))
<Address: Frank Person <FPERSON@example.com> [not verified]
          key: fperson@example.com at 0x...>

Both the case-insensitive version of the address and the original case-preserved version are available on attributes of the IAddress object.

>>> print(address_6.email)
fperson@example.com
>>> print(address_6.original_email)
FPERSON@example.com

Because addresses are case-insensitive for all other purposes, you cannot create an address that differs only in case. You can get the address using either the lower cased version or case-preserved version. In fact, searching for an address is case insensitive.

>>> print(user_manager.get_address('fperson@example.com').email)
fperson@example.com
>>> print(user_manager.get_address('FPERSON@example.com').email)
fperson@example.com

Automatic responder¶

In various situations, Mailman will send an automatic response to the author of an email message. For example, if someone sends a command to the -request address, Mailman will send a response, but to cut down on third party spam, the sender will only get a certain number of responses per day.

First, given a mailing list you need to adapt it to an IAutoResponseSet.

>>> mlist = create_list('test@example.com')
>>> from mailman.interfaces.autorespond import IAutoResponseSet
>>> response_set = IAutoResponseSet(mlist)

>>> from zope.interface.verify import verifyObject
>>> verifyObject(IAutoResponseSet, response_set)
True

You can’t adapt other objects to an IAutoResponseSet.

>>> IAutoResponseSet(object())
Traceback (most recent call last):
...
TypeError: ('Could not adapt', ...

There are various kinds of response types. For example, Mailman will send an automatic response when messages are held for approval, or when it receives an email command. You can find out how many responses for a particular address have already been sent today.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> address = getUtility(IUserManager).create_address(
...     'aperson@example.com')

>>> from mailman.interfaces.autorespond import Response
>>> print(response_set.todays_count(address, Response.hold))
0
>>> print(response_set.todays_count(address, Response.command))
0

Using the response set, we can record that a hold response is sent to the address.

>>> response_set.response_sent(address, Response.hold)
>>> print(response_set.todays_count(address, Response.hold))
1
>>> print(response_set.todays_count(address, Response.command))
0

We can also record that a command response was sent.

>>> response_set.response_sent(address, Response.command)
>>> print(response_set.todays_count(address, Response.hold))
1
>>> print(response_set.todays_count(address, Response.command))
1

Let’s send one more.

>>> response_set.response_sent(address, Response.command)
>>> print(response_set.todays_count(address, Response.hold))
1
>>> print(response_set.todays_count(address, Response.command))
2

Now the day flips over and all the counts reset.

>>> from mailman.utilities.datetime import factory
>>> factory.fast_forward()

>>> print(response_set.todays_count(address, Response.hold))
0
>>> print(response_set.todays_count(address, Response.command))
0

Response dates¶

You can also use the response set to get the date of the last response sent.

>>> response = response_set.last_response(address, Response.hold)
>>> response.mailing_list
<mailing list "test@example.com" at ...>
>>> response.address
<Address: aperson@example.com [not verified] at ...>
>>> response.response_type
<Response.hold: 1>
>>> response.date_sent
datetime.date(2005, 8, 1)

When another response is sent today, that becomes the last one sent.

>>> response_set.response_sent(address, Response.command)
>>> response_set.last_response(address, Response.command).date_sent
datetime.date(2005, 8, 2)

>>> factory.fast_forward(days=3)
>>> response_set.response_sent(address, Response.command)
>>> response_set.last_response(address, Response.command).date_sent
datetime.date(2005, 8, 5)

If there’s been no response sent to a particular address, None is returned.

>>> address = getUtility(IUserManager).create_address(
...     'bperson@example.com')
>>> print(response_set.todays_count(address, Response.command))
0
>>> print(response_set.last_response(address, Response.command))
None

Bounces¶

When a message to an email address bounces, Mailman’s bounce runner will register a bounce event. This registration is done through a utility.

>>> from zope.component import getUtility
>>> from zope.interface.verify import verifyObject
>>> from mailman.interfaces.bounce import IBounceProcessor
>>> processor = getUtility(IBounceProcessor)
>>> verifyObject(IBounceProcessor, processor)
True

Registration¶

When a bounce occurs, it’s always within the context of a specific mailing list.

>>> mlist = create_list('test@example.com')

The bouncing email contains useful information that will be registered as well. In particular, the Message-ID is a key piece of data that needs to be recorded.

>>> msg = message_from_string("""\
... From: mail-daemon@example.org
... To: test-bounces@example.com
... Message-ID: <first>
...
... """)

There is a suite of bounce detectors that are used to heuristically extract the bouncing email addresses. Various techniques are employed including VERP, DSN, and magic. It is the bounce runner’s responsibility to extract the set of bouncing email addresses. These are passed one-by-one to the registration interface.

>>> event = processor.register(mlist, 'anne@example.com', msg)
>>> print(event.list_id)
test.example.com
>>> print(event.email)
anne@example.com
>>> print(event.message_id)
<first>

Bounce events have a timestamp.

>>> print(event.timestamp)
2005-08-01 07:49:23

Bounce events have a flag indicating whether they’ve been processed or not.

>>> event.processed
False

When a bounce is registered, you can indicate the bounce context.

>>> msg = message_from_string("""\
... From: mail-daemon@example.org
... To: test-bounces@example.com
... Message-ID: <second>
...
... """)

If no context is given, then a default one is used.

>>> event = processor.register(mlist, 'bart@example.com', msg)
>>> print(event.message_id)
<second>
>>> print(event.context)
BounceContext.normal

A probe bounce carries more weight than just a normal bounce.

>>> from mailman.interfaces.bounce import BounceContext
>>> event = processor.register(
...     mlist, 'bart@example.com', msg, BounceContext.probe)
>>> print(event.message_id)
<second>
>>> print(event.context)
BounceContext.probe

Domains¶

Domains are how Mailman interacts with email host names and web host names.

>>> from operator import attrgetter
>>> def show_domains(*, with_owners=False):
...     if len(manager) == 0:
...         print('no domains')
...         return
...     for domain in sorted(manager, key=attrgetter('mail_host')):
...         print(domain)
...     owners = sorted(owner.addresses[0].email
...                     for owner in domain.owners)
...     for owner in owners:
...         print('- owner:', owner)

>>> show_domains()
no domains

Adding a domain requires some basic information, of which the email host name is the only required piece. The other parts are inferred from that.

>>> manager.add('example.org')
<Domain example.org, base_url: http://example.org>
>>> show_domains()
<Domain example.org, base_url: http://example.org>

We can remove domains too.

>>> manager.remove('example.org')
<Domain example.org, base_url: http://example.org>
>>> show_domains()
no domains

Sometimes the email host name is different than the base url for hitting the web interface for the domain.

>>> manager.add('example.com', base_url='https://mail.example.com')
<Domain example.com, base_url: https://mail.example.com>
>>> show_domains()
<Domain example.com, base_url: https://mail.example.com>

Domains can have explicit descriptions, and can be created with one or more owners.

>>> manager.add(
...     'example.net',
...     base_url='http://lists.example.net',
...     description='The example domain',
...     owners=['anne@example.com'])
<Domain example.net, The example domain,
        base_url: http://lists.example.net>

>>> show_domains(with_owners=True)
<Domain example.com, base_url: https://mail.example.com>
<Domain example.net, The example domain,
        base_url: http://lists.example.net>
- owner: anne@example.com

Domains can have multiple owners, ideally one of the owners should have a verified preferred address. However this is not checked right now and the configuration’s default contact address may be used as a fallback.

>>> net_domain = manager['example.net']
>>> net_domain.add_owner('bart@example.org')
>>> show_domains(with_owners=True)
<Domain example.com, base_url: https://mail.example.com>
<Domain example.net, The example domain, base_url: http://lists.example.net>
- owner: anne@example.com
- owner: bart@example.org

Domains can list all associated mailing lists with the mailing_lists property.

>>> def show_lists(domain):
...     mlists = list(domain.mailing_lists)
...     for mlist in mlists:
...         print(mlist)
...     if len(mlists) == 0:
...         print('no lists')

>>> net_domain = manager['example.net']
>>> com_domain = manager['example.com']
>>> show_lists(net_domain)
no lists

>>> create_list('test@example.net')
<mailing list "test@example.net" at ...>
>>> transaction.commit()
>>> show_lists(net_domain)
<mailing list "test@example.net" at ...>

>>> show_lists(com_domain)
no lists

In the global domain manager, domains are indexed by their email host name.

>>> for domain in sorted(manager, key=attrgetter('mail_host')):
...     print(domain.mail_host)
example.com
example.net

>>> print(manager['example.net'])
<Domain example.net, The example domain,
        base_url: http://lists.example.net>

As with dictionaries, you can also get the domain. If the domain does not exist, None or a default is returned.

>>> print(manager.get('example.net'))
<Domain example.net, The example domain,
        base_url: http://lists.example.net>

>>> print(manager.get('doesnotexist.com'))
None

>>> print(manager.get('doesnotexist.com', 'blahdeblah'))
blahdeblah

Confirmation tokens¶

Confirmation tokens can be added to the domain’s url to generate the URL to a page users can use to confirm their subscriptions.

>>> domain = manager['example.net']
>>> print(domain.confirm_url('abc'))
http://lists.example.net/confirm/abc

Languages¶

Mailman is multilingual. A language manager handles the known set of languages at run time, as well as enabling those languages for use in a running Mailman instance.

>>> from mailman.interfaces.languages import ILanguageManager
>>> from zope.component import getUtility
>>> from zope.interface.verify import verifyObject

>>> mgr = getUtility(ILanguageManager)
>>> verifyObject(ILanguageManager, mgr)
True

# Make a copy of the language manager's dictionary, so we can restore it
# after the test.  Currently the test layer doesn't manage this.
>>> saved = mgr._languages.copy()

# The language manager component comes pre-populated; clear it out.
>>> mgr.clear()

A language manager keeps track of the languages it knows about.

>>> list(mgr.codes)
[]
>>> list(mgr.languages)
[]

Adding languages¶

Adding a new language requires three pieces of information, the 2-character language code, the English description of the language, and the character set used by the language. The language object is returned.

>>> mgr.add('en', 'us-ascii', 'English')
<Language [en] English>
>>> mgr.add('it', 'iso-8859-1', 'Italian')
<Language [it] Italian>

And you can get information for all known languages.

>>> print(mgr['en'].description)
English
>>> print(mgr['en'].charset)
us-ascii
>>> print(mgr['it'].description)
Italian
>>> print(mgr['it'].charset)
iso-8859-1

Other iterations¶

You can iterate over all the known language codes.

>>> mgr.add('pl', 'iso-8859-2', 'Polish')
<Language [pl] Polish>
>>> sorted(mgr.codes)
['en', 'it', 'pl']

You can iterate over all the known languages.

>>> from operator import attrgetter
>>> languages = sorted((language for language in mgr.languages),
...                    key=attrgetter('code'))
>>> for language in languages:
...     print(language.code, language.charset, language.description)
en us-ascii English
it iso-8859-1 Italian
pl iso-8859-2 Polish

You can ask whether a particular language code is known.

>>> 'it' in mgr
True
>>> 'xx' in mgr
False

You can get a particular language by its code.

>>> print(mgr['it'].description)
Italian
>>> print(mgr['xx'].code)
Traceback (most recent call last):
...
KeyError: 'xx'
>>> print(mgr.get('it').description)
Italian
>>> print(mgr.get('xx'))
None
>>> print(mgr.get('xx', 'missing'))
missing

Clearing the known languages¶

The language manager can forget about all the language codes it knows about.

>>> 'en' in mgr
True

>>> mgr.clear()
>>> 'en' in mgr
False

# Restore the data.
>>> mgr._languages = saved

The mailing list manager¶

The IListManager is how you create, delete, and retrieve mailing list objects.

>>> from mailman.interfaces.listmanager import IListManager
>>> from zope.component import getUtility
>>> list_manager = getUtility(IListManager)

Creating a mailing list¶

Creating the list returns the newly created IMailList object.

>>> from mailman.interfaces.mailinglist import IMailingList
>>> mlist = list_manager.create('test@example.com')
>>> IMailingList.providedBy(mlist)
True

All lists with identities have a short name, a host name, a fully qualified listname, and an RFC 2369 list id. This latter will not change even if the mailing list moves to a different host, so it is what uniquely distinguishes the mailing list to the system.

>>> print(mlist.list_name)
test
>>> print(mlist.mail_host)
example.com
>>> print(mlist.fqdn_listname)
test@example.com
>>> print(mlist.list_id)
test.example.com

Deleting a mailing list¶

Use the list manager to delete a mailing list.

>>> list_manager.delete(mlist)
>>> sorted(list_manager.names)
[]

After deleting the list, you can create it again.

>>> mlist = list_manager.create('test@example.com')
>>> print(mlist.fqdn_listname)
test@example.com

Retrieving a mailing list¶

When a mailing list exists, you can ask the list manager for it and you will always get the same object back.

>>> mlist_2 = list_manager.get('test@example.com')
>>> mlist_2 is mlist
True

You can also get a mailing list by it’s list id.

>>> mlist_2 = list_manager.get_by_list_id('test.example.com')
>>> mlist_2 is mlist
True

If you try to get a list that doesn’t existing yet, you get None.

>>> print(list_manager.get('test_2@example.com'))
None
>>> print(list_manager.get_by_list_id('test_2.example.com'))
None

You also get None if the list name is invalid.

>>> print(list_manager.get('foo'))
None

Iterating over all mailing lists¶

Once you’ve created a bunch of mailing lists, you can use the list manager to iterate over the mailing list objects, the list posting addresses, or the list address components.

>>> mlist_3 = list_manager.create('test_3@example.com')
>>> mlist_4 = list_manager.create('test_4@example.com')

>>> for name in sorted(list_manager.names):
...     print(name)
test@example.com
test_3@example.com
test_4@example.com

>>> for list_id in sorted(list_manager.list_ids):
...     print(list_id)
test.example.com
test_3.example.com
test_4.example.com

>>> for fqdn_listname in sorted(m.fqdn_listname
...                             for m in list_manager.mailing_lists):
...     print(fqdn_listname)
test@example.com
test_3@example.com
test_4@example.com

>>> for list_name, mail_host in sorted(list_manager.name_components):
...     print(list_name, '@', mail_host)
test   @ example.com
test_3 @ example.com
test_4 @ example.com

Mailing lists¶

The mailing list is a core object in Mailman. It is uniquely identified in the system by its list-id which is derived from its posting address, i.e. the email address you would send a message to in order to post a message to the mailing list. The list id is defined in RFC 2369.

>>> mlist = create_list('aardvark@example.com')
>>> print(mlist.list_id)
aardvark.example.com
>>> print(mlist.fqdn_listname)
aardvark@example.com

The mailing list also has convenient attributes for accessing the list’s short name (i.e. local part) and host name.

>>> print(mlist.list_name)
aardvark
>>> print(mlist.mail_host)
example.com

Rosters¶

Mailing list membership is represented by rosters. Each mailing list has several rosters of members, representing the subscribers to the mailing list, the owners, the moderators, and so on. The rosters are defined by a membership role.

Addresses can be explicitly subscribed to a mailing list. By default, a subscription puts the address in the member role, meaning that address will receive a copy of any message sent to the mailing list.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)

>>> aperson = user_manager.create_address('aperson@example.com')
>>> bperson = user_manager.create_address('bperson@example.com')
>>> mlist.subscribe(aperson)
<Member: aperson@example.com on aardvark@example.com as MemberRole.member>
>>> mlist.subscribe(bperson)
<Member: bperson@example.com on aardvark@example.com as MemberRole.member>

Both addresses appear on the roster of members.

>>> from operator import attrgetter
>>> sort_key = attrgetter('address.email')

>>> for member in sorted(mlist.members.members, key=sort_key):
...     print(member)
<Member: aperson@example.com on aardvark@example.com as MemberRole.member>
<Member: bperson@example.com on aardvark@example.com as MemberRole.member>

By explicitly specifying the role of the subscription, an address can be added to the owner and moderator rosters.

>>> from mailman.interfaces.member import MemberRole
>>> mlist.subscribe(aperson, MemberRole.owner)
<Member: aperson@example.com on aardvark@example.com as MemberRole.owner>
>>> cperson = user_manager.create_address('cperson@example.com')
>>> mlist.subscribe(cperson, MemberRole.owner)
<Member: cperson@example.com on aardvark@example.com as MemberRole.owner>
>>> mlist.subscribe(cperson, MemberRole.moderator)
<Member: cperson@example.com on aardvark@example.com
         as MemberRole.moderator>

A Person is now both a member and an owner of the mailing list. C Person is an owner and a moderator.

>>> for member in sorted(mlist.owners.members, key=sort_key):
...     print(member)
<Member: aperson@example.com on aardvark@example.com as MemberRole.owner>
<Member: cperson@example.com on aardvark@example.com as MemberRole.owner>

>>> for member in mlist.moderators.members:
...     print(member)
<Member: cperson@example.com on aardvark@example.com
         as MemberRole.moderator>

All rosters can also be accessed indirectly.

>>> roster = mlist.get_roster(MemberRole.member)
>>> for member in sorted(roster.members, key=sort_key):
...     print(member)
<Member: aperson@example.com on aardvark@example.com as MemberRole.member>
<Member: bperson@example.com on aardvark@example.com as MemberRole.member>

>>> roster = mlist.get_roster(MemberRole.owner)
>>> for member in sorted(roster.members, key=sort_key):
...     print(member)
<Member: aperson@example.com on aardvark@example.com as MemberRole.owner>
<Member: cperson@example.com on aardvark@example.com as MemberRole.owner>

>>> roster = mlist.get_roster(MemberRole.moderator)
>>> for member in roster.members:
...     print(member)
<Member: cperson@example.com on aardvark@example.com
         as MemberRole.moderator>

Subscribing users¶

An alternative way of subscribing to a mailing list is as a user with a preferred address. This way the user can change their subscription address just by changing their preferred address.

The user must have a preferred address.

>>> from mailman.utilities.datetime import now
>>> user = user_manager.create_user('dperson@example.com', 'Dave Person')
>>> address = list(user.addresses)[0]
>>> address.verified_on = now()
>>> user.preferred_address = address

The preferred address is used in the subscription.

>>> mlist.subscribe(user)
<Member: Dave Person <dperson@example.com> on aardvark@example.com
         as MemberRole.member>
>>> for member in sorted(mlist.members.members, key=sort_key):
...     print(member)
<Member: aperson@example.com on aardvark@example.com as MemberRole.member>
<Member: bperson@example.com on aardvark@example.com as MemberRole.member>
<Member: Dave Person <dperson@example.com> on aardvark@example.com
         as MemberRole.member>

If the user’s preferred address changes, their subscribed email address also changes automatically.

>>> new_address = user.register('dave.person@example.com')
>>> new_address.verified_on = now()
>>> user.preferred_address = new_address

>>> for member in sorted(mlist.members.members, key=sort_key):
...     print(member)
<Member: aperson@example.com on aardvark@example.com as MemberRole.member>
<Member: bperson@example.com on aardvark@example.com as MemberRole.member>
<Member: dave.person@example.com on aardvark@example.com
         as MemberRole.member>

A user is allowed to explicitly subscribe again with a specific address, even if this address is their preferred address.

>>> mlist.subscribe(user.preferred_address)
<Member: dave.person@example.com
         on aardvark@example.com as MemberRole.member>

List memberships¶

Users represent people in Mailman, members represent subscriptions. Users control email addresses, and rosters are collections of members. A member ties a subscribed email address to a role, such as member, administrator, or moderator. Even non-members are represented by a roster.

Roster sets are collections of rosters and a mailing list has a single roster set that contains all its members, regardless of that member’s role.

Mailing lists and roster sets have an indirect relationship, through the roster set’s name. Roster also have names, but are related to roster sets by a more direct containment relationship. This is because it is possible to store mailing list data in a different database than user data.

When we create a mailing list, it starts out with no members, owners, moderators, administrators, or nonmembers.

>>> mlist = create_list('ant@example.com')
>>> dump_list(mlist.members.members)
*Empty*
>>> dump_list(mlist.owners.members)
*Empty*
>>> dump_list(mlist.moderators.members)
*Empty*
>>> dump_list(mlist.administrators.members)
*Empty*
>>> dump_list(mlist.nonmembers.members)
*Empty*

Administrators¶

A mailing list’s administrators are defined as union of the list’s owners and moderators. We can add new owners or moderators to this list by assigning roles to users. First we have to create the user, because there are no users in the user database yet.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)
>>> user_1 = user_manager.create_user('aperson@example.com', 'Anne Person')
>>> print(user_1)
<User "Anne Person" (...) at ...>

We can add Anne as an owner of the mailing list, by creating a member role for her.

>>> from mailman.interfaces.member import MemberRole
>>> address_1 = list(user_1.addresses)[0]
>>> mlist.subscribe(address_1, MemberRole.owner)
<Member: Anne Person <aperson@example.com> on
         ant@example.com as MemberRole.owner>
>>> dump_list(member.address for member in mlist.owners.members)
Anne Person <aperson@example.com>

Adding Anne as a list owner also makes her an administrator, but does not make her a moderator. Nor does it make her a member of the list.

>>> dump_list(member.address for member in mlist.administrators.members)
Anne Person <aperson@example.com>
>>> dump_list(member.address for member in mlist.moderators.members)
*Empty*
>>> dump_list(member.address for member in mlist.members.members)
*Empty*

Bart becomes a moderator of the list.

>>> user_2 = user_manager.create_user('bperson@example.com', 'Bart Person')
>>> print(user_2)
<User "Bart Person" (...) at ...>
>>> address_2 = list(user_2.addresses)[0]
>>> mlist.subscribe(address_2, MemberRole.moderator)
<Member: Bart Person <bperson@example.com>
         on ant@example.com as MemberRole.moderator>
>>> dump_list(member.address for member in mlist.moderators.members)
Bart Person <bperson@example.com>

Now, both Anne and Bart are list administrators.

>>> from operator import attrgetter
>>> def dump_members(roster):
...     all_addresses = list(member.address for member in roster)
...     sorted_addresses = sorted(all_addresses, key=attrgetter('email'))
...     dump_list(sorted_addresses)

>>> dump_members(mlist.administrators.members)
Anne Person <aperson@example.com>
Bart Person <bperson@example.com>

Members¶

Similarly, list members are born of users being subscribed with the proper role.

>>> user_3 = user_manager.create_user(
...     'cperson@example.com', 'Cris Person')
>>> address_3 = list(user_3.addresses)[0]
>>> member = mlist.subscribe(address_3, MemberRole.member)
>>> member
<Member: Cris Person <cperson@example.com>
         on ant@example.com as MemberRole.member>

Cris’s user record can also be retrieved from her member record.

>>> member.user
<User "Cris Person" (3) at ...>

Cris will be a regular delivery member but not a digest member.

>>> dump_members(mlist.members.members)
Cris Person <cperson@example.com>
>>> dump_members(mlist.regular_members.members)
Cris Person <cperson@example.com>
>>> dump_members(mlist.digest_members.addresses)
*Empty*

It’s easy to make the list administrators members of the mailing list too.

>>> members = []
>>> for address in mlist.administrators.addresses:
...     member = mlist.subscribe(address, MemberRole.member)
...     members.append(member)
>>> dump_list(members, key=attrgetter('address.email'))
<Member: Anne Person <aperson@example.com> on
         ant@example.com as MemberRole.member>
<Member: Bart Person <bperson@example.com> on
         ant@example.com as MemberRole.member>
>>> dump_members(mlist.members.members)
Anne Person <aperson@example.com>
Bart Person <bperson@example.com>
Cris Person <cperson@example.com>
>>> dump_members(mlist.regular_members.members)
Anne Person <aperson@example.com>
Bart Person <bperson@example.com>
Cris Person <cperson@example.com>
>>> dump_members(mlist.digest_members.members)
*Empty*

Nonmembers¶

Nonmembers are used to represent people who have posted to the mailing list but are not subscribed to the mailing list. These may be legitimate users who have found the mailing list and wish to interact without a direct subscription, or they may be spammers who should never be allowed to contact the mailing list. Because all the same moderation rules can be applied to nonmembers, we represent them as the same type of object but with a different role.

>>> user_6 = user_manager.create_user('fperson@example.com', 'Fred Person')
>>> address_6 = list(user_6.addresses)[0]
>>> member_6 = mlist.subscribe(address_6, MemberRole.nonmember)
>>> member_6
<Member: Fred Person <fperson@example.com> on ant@example.com
         as MemberRole.nonmember>
>>> dump_members(mlist.nonmembers.members)
Fred Person <fperson@example.com>

Nonmembers do not get delivery of any messages.

>>> dump_members(mlist.members.members)
Anne Person <aperson@example.com>
Bart Person <bperson@example.com>
Cris Person <cperson@example.com>
>>> dump_members(mlist.regular_members.members)
Anne Person <aperson@example.com>
Bart Person <bperson@example.com>
Cris Person <cperson@example.com>
>>> dump_members(mlist.digest_members.members)
*Empty*

Finding members¶

You can find the IMember object that is a member of a roster for a given text email address by using the IRoster.get_member() method.

>>> mlist.owners.get_member('aperson@example.com')
<Member: Anne Person <aperson@example.com> on
         ant@example.com as MemberRole.owner>
>>> mlist.administrators.get_member('aperson@example.com')
<Member: Anne Person <aperson@example.com> on
         ant@example.com as MemberRole.owner>
>>> mlist.members.get_member('aperson@example.com')
<Member: Anne Person <aperson@example.com> on
         ant@example.com as MemberRole.member>
>>> mlist.nonmembers.get_member('fperson@example.com')
<Member: Fred Person <fperson@example.com> on
         ant@example.com as MemberRole.nonmember>

However, if the address is not subscribed with the appropriate role, then None is returned.

>>> print(mlist.administrators.get_member('zperson@example.com'))
None
>>> print(mlist.moderators.get_member('aperson@example.com'))
None
>>> print(mlist.members.get_member('zperson@example.com'))
None
>>> print(mlist.nonmembers.get_member('aperson@example.com'))
None

All subscribers¶

There is also a roster containing all the subscribers of a mailing list, regardless of their role.

>>> def sortkey(member):
...     return (member.address.email, member.role.value)
>>> for member in sorted(mlist.subscribers.members, key=sortkey):
...     print(member.address.email, member.role)
aperson@example.com MemberRole.member
aperson@example.com MemberRole.owner
bperson@example.com MemberRole.member
bperson@example.com MemberRole.moderator
cperson@example.com MemberRole.member
fperson@example.com MemberRole.nonmember

Subscriber type¶

Members can be subscribed to a mailing list either via an explicit address, or indirectly through a user’s preferred address. Sometimes you want to know which one it is.

Herb subscribes to the mailing list via an explicit address.

>>> herb = user_manager.create_address(
...     'hperson@example.com', 'Herb Person')
>>> herb_member = mlist.subscribe(herb)

Iris subscribes to the mailing list via her preferred address.

>>> iris = user_manager.make_user(
...     'iperson@example.com', 'Iris Person')
>>> preferred = list(iris.addresses)[0]
>>> from mailman.utilities.datetime import now
>>> preferred.verified_on = now()
>>> iris.preferred_address = preferred
>>> iris_member = mlist.subscribe(iris)

When we need to know which way a member is subscribed, we can look at the this attribute.

>>> herb_member.subscriber
<Address: Herb Person <hperson@example.com> [not verified] at ...>
>>> iris_member.subscriber
<User "Iris Person" (5) at ...>

Moderation actions¶

All members of any role have a moderation action which specifies how postings from that member are handled. By default, owners and moderators are automatically accepted for posting to the mailing list.

>>> for member in sorted(mlist.administrators.members,
...                      key=attrgetter('address.email')):
...     print(member.address.email, member.role, member.moderation_action)
aperson@example.com MemberRole.owner     Action.accept
bperson@example.com MemberRole.moderator Action.accept

By default, members have a deferred action which specifies that the posting should go through the normal moderation checks.

>>> for member in sorted(mlist.members.members,
...                      key=attrgetter('address.email')):
...     print(member.address.email, member.role, member.moderation_action)
aperson@example.com MemberRole.member Action.defer
bperson@example.com MemberRole.member Action.defer
cperson@example.com MemberRole.member Action.defer
hperson@example.com MemberRole.member Action.defer
iperson@example.com MemberRole.member Action.defer

Postings by nonmembers are held for moderator approval by default.

>>> for member in mlist.nonmembers.members:
...     print(member.address.email, member.role, member.moderation_action)
fperson@example.com MemberRole.nonmember Action.hold

Changing subscriptions¶

When a user is subscribed to a mailing list via a specific address they control (as opposed to being subscribed with their preferred address), they can change their delivery address by setting the appropriate parameter. Note though that the address they’re changing to must be verified.

>>> bee = create_list('bee@example.com')
>>> gwen = user_manager.create_user('gwen@example.com')
>>> gwen_address = list(gwen.addresses)[0]
>>> gwen_member = bee.subscribe(gwen_address)
>>> for m in bee.members.members:
...     print(m.member_id.int, m.mailing_list.list_id, m.address.email)
9 bee.example.com gwen@example.com

Gwen gets a email address.

>>> new_address = gwen.register('gperson@example.com')

Gwen verifies her email address, and updates her membership.

>>> from mailman.utilities.datetime import now
>>> new_address.verified_on = now()
>>> gwen_member.address = new_address

Now her membership reflects the new address.

>>> for m in bee.members.members:
...     print(m.member_id.int, m.mailing_list.list_id, m.address.email)
9 bee.example.com gperson@example.com

Events¶

An event is triggered when a new member is subscribed to a mailing list.

>>> from mailman.testing.helpers import event_subscribers
>>> def handle_event(event):
...     print(event)

>>> cat = create_list('cat@example.com')
>>> herb = user_manager.create_address('herb@example.com')
>>> with event_subscribers(handle_event):
...     member = cat.subscribe(herb)
herb@example.com joined cat.example.com

An event is triggered when a member is unsubscribed from a mailing list.

>>> with event_subscribers(handle_event):
...     member.unsubscribe()
herb@example.com left cat.example.com

The message store¶

The message store is a collection of messages keyed off of Message-ID and X-Message-ID-Hash headers. Either of these values can be combined with the message’s List-Archive header to create a globally unique URI to the message object in the internet facing interface of the message store. The X-Message-ID-Hash is the base-32 SHA1 hash of the Message-ID.

>>> from mailman.interfaces.messages import IMessageStore
>>> from zope.component import getUtility
>>> message_store = getUtility(IMessageStore)

A message with a Message-ID header can be stored.

>>> msg = message_from_string("""\
... Subject: An important message
... Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
...
... This message is very important.
... """)
>>> x_message_id_hash = message_store.add(msg)
>>> print(x_message_id_hash)
AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
>>> print(msg.as_string())
Subject: An important message
Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
X-Message-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
<BLANKLINE>
This message is very important.
<BLANKLINE>

Finding messages¶

There are several ways to find a message given either the Message-ID or X-Message-ID-Hash headers. In either case, if no matching message is found, None is returned.

>>> print(message_store.get_message_by_id('nothing'))
None
>>> print(message_store.get_message_by_hash('nothing'))
None

Given an existing Message-ID, the message can be found.

>>> message = message_store.get_message_by_id(msg['message-id'])
>>> print(message.as_string())
Subject: An important message
Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
X-Message-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
<BLANKLINE>
This message is very important.
<BLANKLINE>

Similarly, we can find messages by the X-Message-ID-Hash:

>>> message = message_store.get_message_by_hash(msg['x-message-id-hash'])
>>> print(message.as_string())
Subject: An important message
Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
X-Message-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
<BLANKLINE>
This message is very important.
<BLANKLINE>

Iterating over all messages¶

The message store provides a means to iterate over all the messages it contains.

>>> messages = list(message_store.messages)
>>> len(messages)
1
>>> print(messages[0].as_string())
Subject: An important message
Message-ID: <87myycy5eh.fsf@uwakimon.sk.tsukuba.ac.jp>
X-Message-ID-Hash: AGDWSNXXKCWEILKKNYTBOHRDQGOX3Y35
<BLANKLINE>
This message is very important.
<BLANKLINE>

Deleting messages from the store¶

You delete a message from the storage service by providing the Message-ID for the message you want to delete.

>>> message_id = message['message-id']
>>> message_store.delete_message(message_id)
>>> list(message_store.messages)
[]
>>> print(message_store.get_message_by_id(message_id))
None
>>> print(message_store.get_message_by_hash(message['x-message-id-hash']))
None

Mailing list addresses¶

Every mailing list has a number of addresses which are publicly available. These are defined in the IMailingListAddresses interface.

>>> mlist = create_list('_xtest@example.com')

The posting address is where people send messages to be posted to the mailing list. This is exactly the same as the fully qualified list name.

>>> print(mlist.fqdn_listname)
_xtest@example.com
>>> print(mlist.posting_address)
_xtest@example.com

Messages to the mailing list’s no reply address always get discarded without prejudice.

>>> print(mlist.no_reply_address)
noreply@example.com

The mailing list’s owner address reaches the human moderators.

>>> print(mlist.owner_address)
_xtest-owner@example.com

The request address goes to the list’s email command robot.

>>> print(mlist.request_address)
_xtest-request@example.com

The bounces address accepts and processes all potential bounces.

>>> print(mlist.bounces_address)
_xtest-bounces@example.com

The join (a.k.a. subscribe) address is where someone can email to get added to the mailing list. The subscribe alias is a synonym for join, but it’s deprecated.

>>> print(mlist.join_address)
_xtest-join@example.com
>>> print(mlist.subscribe_address)
_xtest-subscribe@example.com

The leave (a.k.a. unsubscribe) address is where someone can email to get added to the mailing list. The unsubscribe alias is a synonym for leave, but it’s deprecated.

>>> print(mlist.leave_address)
_xtest-leave@example.com
>>> print(mlist.unsubscribe_address)
_xtest-unsubscribe@example.com

Email confirmations¶

Email confirmation messages are sent when actions such as subscriptions need to be confirmed. It requires that a cookie be provided, which will be included in the local part of the email address. The exact format of this is dependent on the verp_confirm_format configuration variable.

>>> print(mlist.confirm_address('cookie'))
_xtest-confirm+cookie@example.com
>>> print(mlist.confirm_address('wookie'))
_xtest-confirm+wookie@example.com

>>> config.push('test config', """
... [mta]
... verp_confirm_format: $address---$cookie
... """)
>>> print(mlist.confirm_address('cookie'))
_xtest-confirm---cookie@example.com
>>> config.pop('test config')

The pending database¶

The pending database is where various types of events which need confirmation are stored. These can include email address registration events, held messages (but only for user confirmation), auto-approvals, and probe bounces. This is not where messages held for administrator approval are kept.

In order to pend an event, you first need a pending database.

>>> from mailman.interfaces.pending import IPendings
>>> from zope.component import getUtility
>>> pendingdb = getUtility(IPendings)

There are nothing in the pendings database.

>>> pendingdb.count
0

The pending database can add any IPendable to the database, returning a token that can be used in urls and such.

>>> from zope.interface import implementer
>>> from mailman.interfaces.pending import IPendable
>>> @implementer(IPendable)
... class SimplePendable(dict):
...     pass

>>> subscription = SimplePendable(
...     type='subscription',
...     address='aperson@example.com',
...     display_name='Anne Person',
...     language='en',
...     password='xyz')
>>> token = pendingdb.add(subscription)
>>> len(token)
40

There’s exactly one entry in the pendings database now.

>>> pendingdb.count
1

You can confirm the pending, which means returning the IPendable structure (as a dictionary) from the database that matches the token. If the token isn’t in the database, None is returned.

>>> pendable = pendingdb.confirm(b'missing')
>>> print(pendable)
None
>>> pendable = pendingdb.confirm(token)
>>> dump_msgdata(pendable)
address     : aperson@example.com
display_name: Anne Person
language    : en
password    : xyz
type        : subscription

After confirmation, the token is no longer in the database.

>>> print(pendingdb.confirm(token))
None

There are a few other things you can do with the pending database. When you confirm a token, you can leave it in the database, or in other words, not expunge it.

>>> event_1 = SimplePendable(type='one')
>>> token_1 = pendingdb.add(event_1)
>>> event_2 = SimplePendable(type='two')
>>> token_2 = pendingdb.add(event_2)
>>> event_3 = SimplePendable(type='three')
>>> token_3 = pendingdb.add(event_3)
>>> pendable = pendingdb.confirm(token_1, expunge=False)
>>> dump_msgdata(pendable)
type: one
>>> pendable = pendingdb.confirm(token_1, expunge=True)
>>> dump_msgdata(pendable)
type: one
>>> print(pendingdb.confirm(token_1))
None

You can iterate over all the pendings in the database.

>>> pendables = list(pendingdb)
>>> def sort_key(item):
...     token, pendable = item
...     return pendable['type']
>>> sorted_pendables = sorted(pendables, key=sort_key)
>>> for token, pendable in sorted_pendables:
...     print(pendable['type'])
three
two

An event can be given a lifetime when it is pended, otherwise it just uses a default lifetime.

>>> from datetime import timedelta
>>> yesterday = timedelta(days=-1)
>>> event_4 = SimplePendable(type='four')
>>> token_4 = pendingdb.add(event_4, lifetime=yesterday)

Every once in a while the pending database is cleared of old records.

>>> pendingdb.evict()
>>> print(pendingdb.confirm(token_4))
None
>>> pendable = pendingdb.confirm(token_2)
>>> dump_msgdata(pendable)
type: two

Registration¶

When a user wants to join a mailing list, they must register and verify their email address. Then depending on how the mailing list is configured, they may need to confirm their subscription and have it approved by the list moderator. The IRegistrar interface manages this work flow.

>>> from mailman.interfaces.registrar import IRegistrar

Registrars adapt mailing lists.

>>> from mailman.interfaces.mailinglist import SubscriptionPolicy
>>> mlist = create_list('ant@example.com')
>>> mlist.send_welcome_message = False
>>> mlist.subscription_policy = SubscriptionPolicy.open
>>> registrar = IRegistrar(mlist)

Usually, addresses are registered, but users with preferred addresses can be registered too.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> anne = getUtility(IUserManager).create_address(
...     'anne@example.com', 'Anne Person')

Register an email address¶

When the registration steps involve confirmation or moderator approval, the process will pause until these steps are completed. A unique token is created which represents this work flow.

Anne attempts to join the mailing list.

>>> token, token_owner, member = registrar.register(anne)

Because her email address has not yet been verified, she has not yet become a member of the mailing list.

>>> print(member)
None
>>> print(mlist.members.get_member('anne@example.com'))
None

Once she verifies her email address, she will become a member of the mailing list. In this case, verifying implies that she also confirms her wish to join the mailing list.

>>> token, token_owner, member = registrar.confirm(token)
>>> member
<Member: Anne Person <anne@example.com> on ant@example.com
    as MemberRole.member>
>>> mlist.members.get_member('anne@example.com')
<Member: Anne Person <anne@example.com> on ant@example.com
    as MemberRole.member>

Register a user¶

Users can also register, but they must have a preferred address. The mailing list will deliver messages to this preferred address.

>>> bart = getUtility(IUserManager).make_user(
...     'bart@example.com', 'Bart Person')

Bart verifies his address and makes it his preferred address.

>>> from mailman.utilities.datetime import now
>>> preferred = list(bart.addresses)[0]
>>> preferred.verified_on = now()
>>> bart.preferred_address = preferred

The mailing list’s subscription policy does not require Bart to confirm his subscription, but the moderate does want to approve all subscriptions.

>>> mlist.subscription_policy = SubscriptionPolicy.moderate

Now when Bart registers as a user for the mailing list, a token will still be generated, but this is only used by the moderator. At first, Bart is not subscribed to the mailing list.

>>> token, token_owner, member = registrar.register(bart)
>>> print(member)
None
>>> print(mlist.members.get_member('bart@example.com'))
None

When the moderator confirms Bart’s subscription, he joins the mailing list.

>>> token, token_owner, member = registrar.confirm(token)
>>> member
<Member: Bart Person <bart@example.com> on ant@example.com
    as MemberRole.member>
>>> mlist.members.get_member('bart@example.com')
<Member: Bart Person <bart@example.com> on ant@example.com
    as MemberRole.member>

Moderator requests¶

Various actions will be held for moderator approval, such as subscriptions to closed lists, or postings by non-members. The requests database is the low level interface to these actions requiring approval.

An application level interface for holding messages and membership changes is also available.

Mailing list-centric¶

A set of requests are always related to a particular mailing list. Adapt the mailing list to get its requests.

>>> from mailman.interfaces.requests import IListRequests
>>> from zope.interface.verify import verifyObject

>>> mlist = create_list('test@example.com')
>>> requests = IListRequests(mlist)
>>> verifyObject(IListRequests, requests)
True
>>> requests.mailing_list
<mailing list "test@example.com" at ...>

Holding requests¶

The list’s requests database starts out empty.

>>> print(requests.count)
0
>>> dump_list(requests.held_requests)
*Empty*

At the lowest level, the requests database is very simple. Holding a request requires a request type (as an enum value), a key, and an optional dictionary of associated data. The request database assigns no semantics to the held data, except for the request type.

>>> from mailman.interfaces.requests import RequestType

We can hold messages for moderator approval.

>>> requests.hold_request(RequestType.held_message, 'hold_1')
1

We can hold subscription requests for moderator approval.

>>> requests.hold_request(RequestType.subscription, 'hold_2')
2

We can hold unsubscription requests for moderator approval.

>>> requests.hold_request(RequestType.unsubscription, 'hold_3')
3

Getting requests¶

We can see the total number of requests being held.

>>> print(requests.count)
3

We can also see the number of requests being held by request type.

>>> print(requests.count_of(RequestType.subscription))
1
>>> print(requests.count_of(RequestType.unsubscription))
1

We can also see when there are multiple held requests of a particular type.

>>> print(requests.hold_request(RequestType.held_message, 'hold_4'))
4
>>> print(requests.count_of(RequestType.held_message))
2

We can ask the requests database for a specific request, by providing the id of the request data we want. This returns a 2-tuple of the key and data we originally held.

>>> key, data = requests.get_request(2)
>>> print(key)
hold_2

There was no additional data associated with request 2.

>>> print(data)
None

If we ask for a request that is not in the database, we get None back.

>>> print(requests.get_request(801))
None

Additional data¶

When a request is held, additional data can be associated with it, in the form of a dictionary with string values.

>>> data = dict(foo='yes', bar='no')
>>> requests.hold_request(RequestType.held_message, 'hold_5', data)
5

The data is returned when the request is retrieved. The dictionary will have an additional key which holds the name of the request type.

>>> key, data = requests.get_request(5)
>>> print(key)
hold_5
>>> dump_msgdata(data)
_request_type: held_message
bar          : no
foo          : yes

Iterating over requests¶

To make it easier to find specific requests, the list requests can be iterated over by type.

>>> print(requests.count_of(RequestType.held_message))
3
>>> for request in requests.of_type(RequestType.held_message):
...     key, data = requests.get_request(request.id)
...     print(request.id, request.request_type, key)
...     if data is not None:
...         for key in sorted(data):
...             print('    {0}: {1}'.format(key, data[key]))
1 RequestType.held_message hold_1
4 RequestType.held_message hold_4
5 RequestType.held_message hold_5
    _request_type: held_message
    bar: no
    foo: yes

Deleting requests¶

Once a specific request has been handled, it can be deleted from the requests database.

>>> print(requests.count)
5
>>> requests.delete_request(2)
>>> print(requests.count)
4

Request 2 is no longer in the database.

>>> print(requests.get_request(2))
None

>>> for request in requests.held_requests:
...     requests.delete_request(request.id)
>>> print(requests.count)
0

The user manager¶

The IUserManager is how you create, delete, and manage users.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)

Creating users¶

There are several ways you can create a user object. The simplest is to create a blank user by not providing an address or real name at creation time. This user will have an empty string as their real name, but will not have a password.

>>> from mailman.interfaces.user import IUser
>>> from zope.interface.verify import verifyObject
>>> user = user_manager.create_user()
>>> verifyObject(IUser, user)
True

>>> dump_list(address.email for address in user.addresses)
*Empty*
>>> print(user.display_name)

>>> print(user.password)
None

The user has preferences, but none of them will be specified.

>>> print(user.preferences)
<Preferences ...>

A user can be assigned a real name.

>>> user.display_name = 'Anne Person'
>>> dump_list(user.display_name for user in user_manager.users)
Anne Person

A user can be assigned a password.

>>> user.password = 'secret'
>>> dump_list(user.password for user in user_manager.users)
secret

You can also create a user with an address to start out with.

>>> user_2 = user_manager.create_user('bperson@example.com')
>>> verifyObject(IUser, user_2)
True
>>> dump_list(address.email for address in user_2.addresses)
bperson@example.com
>>> dump_list(user.display_name for user in user_manager.users)
<BLANKLINE>
Anne Person

As above, you can assign a real name to such users.

>>> user_2.display_name = 'Ben Person'
>>> dump_list(user.display_name for user in user_manager.users)
Anne Person
Ben Person

You can also create a user with just a real name.

>>> user_3 = user_manager.create_user(display_name='Claire Person')
>>> verifyObject(IUser, user_3)
True
>>> dump_list(address.email for address in user.addresses)
*Empty*
>>> dump_list(user.display_name for user in user_manager.users)
Anne Person
Ben Person
Claire Person

Finally, you can create a user with both an address and a real name.

>>> user_4 = user_manager.create_user('dperson@example.com', 'Dan Person')
>>> verifyObject(IUser, user_3)
True
>>> dump_list(address.email for address in user_4.addresses)
dperson@example.com
>>> dump_list(address.display_name for address in user_4.addresses)
Dan Person
>>> dump_list(user.display_name for user in user_manager.users)
Anne Person
Ben Person
Claire Person
Dan Person

Deleting users¶

You delete users by going through the user manager. The deleted user is no longer available through the user manager iterator.

>>> user_manager.delete_user(user)
>>> dump_list(user.display_name for user in user_manager.users)
Ben Person
Claire Person
Dan Person

Finding users¶

You can ask the user manager to find the IUser that controls a particular email address. You’ll get back the original user object if it’s found. Note that the .get_user() method takes a string email address, not an IAddress object.

>>> address = list(user_4.addresses)[0]
>>> found_user = user_manager.get_user(address.email)
>>> found_user
<User "Dan Person" (...) at ...>
>>> found_user is user_4
True

If the address is not in the user database or does not have a user associated with it, you will get None back.

>>> print(user_manager.get_user('zperson@example.com'))
None
>>> user_4.unlink(address)
>>> print(user_manager.get_user(address.email))
None

Users can also be found by their unique user id.

>>> found_user = user_manager.get_user_by_id(user_4.user_id)
>>> user_4
<User "Dan Person" (...) at ...>
>>> found_user
<User "Dan Person" (...) at ...>
>>> user_4.user_id == found_user.user_id
True

If a non-existent user id is given, None is returned.

>>> from uuid import UUID
>>> print(user_manager.get_user_by_id(UUID(int=801)))
None

Finding all members¶

The user manager can return all the members known to the system.

>>> mlist = create_list('test@example.com')
>>> mlist.subscribe(list(user_2.addresses)[0])
<Member: bperson@example.com on test@example.com as MemberRole.member>
>>> mlist.subscribe(user_manager.create_address('eperson@example.com'))
<Member: eperson@example.com on test@example.com as MemberRole.member>
>>> mlist.subscribe(user_manager.create_address('fperson@example.com'))
<Member: fperson@example.com on test@example.com as MemberRole.member>

Bart is also the owner of the mailing list.

>>> from mailman.interfaces.member import MemberRole
>>> mlist.subscribe(list(user_2.addresses)[0], MemberRole.owner)
<Member: bperson@example.com on test@example.com as MemberRole.owner>

There are now four members in the system. Sort them by address then role.

>>> def sort_key(member):
...     return (member.address.email, member.role.name)
>>> members = sorted(user_manager.members, key=sort_key)
>>> for member in members:
...     print(member.mailing_list.list_id, member.address.email,
...           member.role)
test.example.com bperson@example.com MemberRole.member
test.example.com bperson@example.com MemberRole.owner
test.example.com eperson@example.com MemberRole.member
test.example.com fperson@example.com MemberRole.member

Creating a new user¶

A common situation (especially during the subscription life cycle) is to create a user linked to an address, with a preferred address. Say for example, we are asked to subscribe a new address we have never seen before.

>>> cris = user_manager.make_user('cris@example.com', 'Cris Person')

Since we’ve never seen cris@example.com before, this call creates a new user with the given email and display name.

>>> cris
<User "Cris Person" (5) at ...>

The user has a single unverified address object.

>>> for address in cris.addresses:
...     print(repr(address))
<Address: Cris Person <cris@example.com> [not verified] at ...>

Users¶

Users are entities that represent people. A user has a real name and a optional encoded password. A user may also have an optional preferences and a set of addresses they control. They can even have a preferred address, i.e. one that they use by default.

See usermanager.txt for examples of how to create, delete, and find users.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)

User data¶

Users may have a real name and a password.

>>> user_1 = user_manager.create_user()
>>> user_1.password = 'my password'
>>> user_1.display_name = 'Zoe Person'
>>> dump_list(user.display_name for user in user_manager.users)
Zoe Person
>>> dump_list(user.password for user in user_manager.users)
my password

The password and real name can be changed at any time.

>>> user_1.display_name = 'Zoe X. Person'
>>> user_1.password = 'another password'
>>> dump_list(user.display_name for user in user_manager.users)
Zoe X. Person
>>> dump_list(user.password for user in user_manager.users)
another password

When the user’s password is changed, an event is triggered.

>>> saved_event = None
>>> def save_event(event):
...     global saved_event
...     saved_event = event
>>> from mailman.testing.helpers import event_subscribers
>>> with event_subscribers(save_event):
...     user_1.password = 'changed again'
>>> print(saved_event)
<PasswordChangeEvent Zoe X. Person>

The event holds a reference to the IUser that changed their password.

>>> print(saved_event.user.display_name)
Zoe X. Person
>>> print(saved_event.user.password)
changed again

Basic user identification¶

Although rarely visible to users, every user has a unique immutable ID. This ID is generated randomly at the time the user is created, and is represented by a UUID.

>>> print(user_1.user_id)
00000000-0000-0000-0000-000000000001

User records also have a date on which they where created.

# The test suite uses a predictable timestamp. >>> print(user_1.created_on) 2005-08-01 07:49:23

Users addresses¶

One of the pieces of information that a user links to is a set of email addresses they control, in the form of IAddress objects. A user can control many addresses, but addresses may be linked to only one user.

The easiest way to link a user to an address is to just register the new address on a user object.

>>> user_1.register('zperson@example.com', 'Zoe Person')
<Address: Zoe Person <zperson@example.com> [not verified] at 0x...>
>>> user_1.register('zperson@example.org')
<Address: zperson@example.org [not verified] at 0x...>
>>> dump_list(address.email for address in user_1.addresses)
zperson@example.com
zperson@example.org
>>> dump_list(address.display_name for address in user_1.addresses)
<BLANKLINE>
Zoe Person

You can also create the address separately and then link it to the user.

>>> address_1 = user_manager.create_address('zperson@example.net')
>>> user_1.link(address_1)
>>> dump_list(address.email for address in user_1.addresses)
zperson@example.com
zperson@example.net
zperson@example.org
>>> dump_list(address.display_name for address in user_1.addresses)
<BLANKLINE>
<BLANKLINE>
Zoe Person

You can also ask whether a given user controls a given address.

>>> user_1.controls(address_1.email)
True
>>> user_1.controls('bperson@example.com')
False

Given a text email address, the user manager can find the user that controls that address.

>>> user_manager.get_user('zperson@example.com') is user_1
True
>>> user_manager.get_user('zperson@example.net') is user_1
True
>>> user_manager.get_user('zperson@example.org') is user_1
True
>>> print(user_manager.get_user('bperson@example.com'))
None

Addresses can also be unlinked from a user.

>>> user_1.unlink(address_1)
>>> user_1.controls('zperson@example.net')
False
>>> print(user_manager.get_user('aperson@example.net'))
None

Preferred address¶

Users can register a preferred address. When subscribing to a mailing list, unless some other address is explicitly specified, the user will be subscribed with their preferred address. This allows them to change their preferred address once, and have all their subscriptions automatically track this change.

By default, a user has no preferred address.

>>> user_2 = user_manager.create_user()
>>> print(user_2.preferred_address)
None

Even when a user registers an address, this address will not be set as the preferred address.

>>> anne = user_2.register('anne@example.com', 'Anne Person')
>>> print(user_2.preferred_address)
None

Once the address has been verified, it can be set as the preferred address, but only if the address is either controlled by the user or uncontrolled. In the latter case, setting it as the preferred address makes it controlled by the user.

>>> from mailman.utilities.datetime import now
>>> anne.verified_on = now()
>>> anne
<Address: Anne Person <anne@example.com> [verified] at ...>
>>> user_2.controls(anne.email)
True
>>> user_2.preferred_address = anne
>>> user_2.preferred_address
<Address: Anne Person <anne@example.com> [verified] at ...>

>>> aperson = user_manager.create_address('aperson@example.com')
>>> user_2.controls(aperson.email)
False
>>> aperson.verified_on = now()
>>> user_2.preferred_address = aperson
>>> user_2.controls(aperson.email)
True

A user can disavow their preferred address.

>>> user_2.preferred_address
<Address: aperson@example.com [verified] at ...>
>>> del user_2.preferred_address
>>> print(user_2.preferred_address)
None

The preferred address always shows up in the set of addresses controlled by this user.

>>> from operator import attrgetter
>>> for address in sorted(user_2.addresses, key=attrgetter('email')):
...     print(address.email)
anne@example.com
aperson@example.com

Users and preferences¶

This is a helper function for the following section.

>>> def show_prefs(prefs):
...     print('acknowledge_posts    :', prefs.acknowledge_posts)
...     print('preferred_language   :', prefs.preferred_language)
...     print('receive_list_copy    :', prefs.receive_list_copy)
...     print('receive_own_postings :', prefs.receive_own_postings)
...     print('delivery_mode        :', prefs.delivery_mode)

Users have preferences, but these preferences have no default settings.

>>> from mailman.interfaces.preferences import IPreferences
>>> show_prefs(user_1.preferences)
acknowledge_posts    : None
preferred_language   : None
receive_list_copy    : None
receive_own_postings : None
delivery_mode        : None

Some of these preferences are booleans and they can be set to True or False.

>>> from mailman.interfaces.languages import ILanguageManager
>>> getUtility(ILanguageManager).add('it', 'iso-8859-1', 'Italian')
<Language [it] Italian>

>>> from mailman.core.constants import DeliveryMode
>>> prefs = user_1.preferences
>>> prefs.acknowledge_posts = True
>>> prefs.preferred_language = 'it'
>>> prefs.receive_list_copy = False
>>> prefs.receive_own_postings = False
>>> prefs.delivery_mode = DeliveryMode.regular
>>> show_prefs(user_1.preferences)
acknowledge_posts    : True
preferred_language   : <Language [it] Italian>
receive_list_copy    : False
receive_own_postings : False
delivery_mode        : DeliveryMode.regular

Subscriptions¶

Users know which mailing lists they are subscribed to, regardless of membership role.

>>> user_1.link(address_1)
>>> dump_list(address.email for address in user_1.addresses)
zperson@example.com
zperson@example.net
zperson@example.org
>>> com = user_manager.get_address('zperson@example.com')
>>> org = user_manager.get_address('zperson@example.org')
>>> net = user_manager.get_address('zperson@example.net')

>>> mlist_1 = create_list('xtest_1@example.com')
>>> mlist_2 = create_list('xtest_2@example.com')
>>> mlist_3 = create_list('xtest_3@example.com')
>>> from mailman.interfaces.member import MemberRole

>>> mlist_1.subscribe(com, MemberRole.member)
<Member: Zoe Person <zperson@example.com> on xtest_1@example.com as
    MemberRole.member>
>>> mlist_2.subscribe(org, MemberRole.member)
<Member: zperson@example.org on xtest_2@example.com as MemberRole.member>
>>> mlist_2.subscribe(org, MemberRole.owner)
<Member: zperson@example.org on xtest_2@example.com as MemberRole.owner>
>>> mlist_3.subscribe(net, MemberRole.moderator)
<Member: zperson@example.net on xtest_3@example.com as
    MemberRole.moderator>

>>> memberships = user_1.memberships
>>> from mailman.interfaces.roster import IRoster
>>> from zope.interface.verify import verifyObject
>>> verifyObject(IRoster, memberships)
True
>>> def sortkey(member):
...     return member.address.email, member.mailing_list, member.role.value
>>> members = sorted(memberships.members, key=sortkey)
>>> len(members)
4
>>> for member in sorted(members, key=sortkey):
...     print(member.address.email, member.mailing_list.list_id,
...           member.role)
zperson@example.com xtest_1.example.com MemberRole.member
zperson@example.net xtest_3.example.com MemberRole.moderator
zperson@example.org xtest_2.example.com MemberRole.member
zperson@example.org xtest_2.example.com MemberRole.owner

Server owners¶

Some users are server owners. Zoe is not yet a server owner.

>>> user_1.is_server_owner
False

So, let’s make her one.

>>> user_1.is_server_owner = True
>>> user_1.is_server_owner
True

Alias Overview¶

A typical Mailman list exposes nine aliases which point to seven different wrapped scripts. E.g. for a list named mylist, you’d have:

mylist-bounces -> bounces
mylist-confirm -> confirm
mylist-join    -> join    (-subscribe is an alias)
mylist-leave   -> leave   (-unsubscribe is an alias)
mylist-owner   -> owner
mylist         -> post
mylist-request -> request

-request, -join, and -leave are a robot addresses; their sole purpose is to process emailed commands, although the latter two are hardcoded to subscription and unsubscription requests. -bounces is the automated bounce processor, and all messages to list members have their return address set to -bounces. If the bounce processor fails to extract a bouncing member address, it can optionally forward the message on to the list owners.

-owner is for reaching a human operator with minimal list interaction (i.e. no bounce processing). -confirm is another robot address which processes replies to VERP-like confirmation notices.

So delivery flow of messages look like this:

joerandom ---> mylist ---> list members
   |                           |
   |                           |[bounces]
   |        mylist-bounces <---+ <-------------------------------+
   |              |                                              |
   |              +--->[internal bounce processing]              |
   |              ^                |                             |
   |              |                |    [bounce found]           |
   |         [bounces *]           +--->[register and discard]   |
   |              |                |                      |      |
   |              |                |                      |[*]   |
   |        [list owners]          |[no bounce found]     |      |
   |              ^                |                      |      |
   |              |                |                      |      |
   +-------> mylist-owner <--------+                      |      |
   |                                                      |      |
   |           data/owner-bounces.mbox <--[site list] <---+      |
   |                                                             |
   +-------> mylist-join--+                                      |
   |                      |                                      |
   +------> mylist-leave--+                                      |
   |                      |                                      |
   |                      v                                      |
   +-------> mylist-request                                      |
   |              |                                              |
   |              +---> [command processor]                      |
   |                            |                                |
   +-----> mylist-confirm ----> +---> joerandom                  |
                                          |                      |
                                          |[bounces]             |
                                          +----------------------+

A person can send an email to the list address (for posting), the -owner address (to reach the human operator), or the -confirm, -join, -leave, and -request mailbots. Message to the list address are then forwarded on to the list membership, with bounces directed to the -bounces address.

[*] Messages sent to the -owner address are forwarded on to the list owner/moderators. All -owner destined messages have their bounces directed to the site list -bounces address, regardless of whether a human sent the message or the message was crafted internally. The intention here is that the site owners want to be notified when one of their list owners’ addresses starts bouncing (yes, the will be automated in a future release).

Any messages to site owners has their bounces directed to a special loop killer address, which just dumps the message into data/owners-bounces.mbox.

Finally, message to any of the mailbots causes the requested action to be performed. Results notifications are sent to the author of the message, which all bounces pointing back to the -bounces address.

The command runner¶

This runner’s purpose is to process and respond to email commands. Commands are extensible using the Mailman plug-in system, but Mailman comes with a number of email commands out of the box. These are processed when a message is sent to the list’s -request address.

>>> mlist = create_list('test@example.com')
>>> mlist.send_welcome_messages = False

A command in the Subject¶

For example, the echo command simply echoes the original command back to the sender. The command can be in the Subject header.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test-request@example.com
... Subject: echo hello
... Message-ID: <aardvark>
...
... """)

>>> from mailman.app.inject import inject_message
>>> filebase = inject_message(mlist, msg, switchboard='command')
>>> from mailman.runners.command import CommandRunner
>>> from mailman.testing.helpers import make_testable_runner
>>> command = make_testable_runner(CommandRunner)
>>> command.run()

And now the response is in the virgin queue.

>>> from mailman.testing.helpers import get_queue_messages
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
Subject: The results of your email commands
From: test-bounces@example.com
To: aperson@example.com
...

The results of your email command are provided below.

- Original message details:
    From: aperson@example.com
    Subject: echo hello
    Date: ...
    Message-ID: <aardvark>

- Results:
echo hello

- Done.


>>> dump_msgdata(messages[0].msgdata)
_parsemsg           : False
listid              : test.example.com
nodecorate          : True
recipients          : {'aperson@example.com'}
reduced_list_headers: True
version             : ...

A command in the body¶

The command can also be found in the body of the message, as long as the message is plain text.

>>> msg = message_from_string("""\
... From: bperson@example.com
... To: test-request@example.com
... Message-ID: <bobcat>
...
... echo foo bar
... """)

>>> filebase = inject_message(mlist, msg, switchboard='command')
>>> command.run()
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
Subject: The results of your email commands
From: test-bounces@example.com
To: bperson@example.com
...
Precedence: bulk

The results of your email command are provided below.

- Original message details:
    From: bperson@example.com
    Subject: n/a
    Date: ...
    Message-ID: <bobcat>

- Results:
echo foo bar

- Done.

Implicit commands¶

For some commands, specifically for joining and leaving a mailing list, there are email aliases that act like commands, even when there’s nothing else in the Subject or body. For example, to join a mailing list, a user need only email the -join address or -subscribe address (the latter is deprecated).

Because Dirk has never registered with Mailman before, he gets two responses. The first is a confirmation message so that Dirk can validate his email address, and the other is the results of his email command.

>>> msg = message_from_string("""\
... From: Dirk Person <dperson@example.com>
... To: test-join@example.com
...
... """)

>>> filebase = inject_message(
...     mlist, msg, switchboard='command', subaddress='join')
>>> command.run()
>>> messages = get_queue_messages('virgin', sort_on='subject')
>>> len(messages)
2

>>> from mailman.interfaces.registrar import IRegistrar
>>> registrar = IRegistrar(mlist)
>>> for item in messages:
...     subject = item.msg['subject']
...     print('Subject:', subject)
...     if 'confirm' in str(subject):
...         token = str(subject).split()[1].strip()
...         new_token, token_owner, member = registrar.confirm(token)
...         assert new_token is None, 'Confirmation failed'
Subject: The results of your email commands
Subject: confirm ...

Similarly, to leave a mailing list, the user need only email the -leave or -unsubscribe address (the latter is deprecated).

>>> msg = message_from_string("""\
... From: dperson@example.com
... To: test-leave@example.com
...
... """)

>>> filebase = inject_message(
...     mlist, msg, switchboard='command', subaddress='leave')
>>> command.run()
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
Subject: The results of your email commands
From: test-bounces@example.com
To: dperson@example.com
...

The results of your email command are provided below.

- Original message details:
From: dperson@example.com
Subject: n/a
Date: ...
Message-ID: ...

- Results:
Dirk Person <dperson@example.com> left test@example.com

- Done.

The -confirm address is also available as an implicit command.

>>> msg = message_from_string("""\
... From: dperson@example.com
... To: test-confirm+123@example.com
...
... """)

>>> filebase = inject_message(
...     mlist, msg, switchboard='command', subaddress='confirm')
>>> command.run()
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
Subject: The results of your email commands
From: test-bounces@example.com
To: dperson@example.com
...

The results of your email command are provided below.

- Original message details:
From: dperson@example.com
Subject: n/a
Date: ...
Message-ID: ...

- Results:
Confirmation token did not match

- Done.

Stopping command processing¶

The end command stops email processing, so that nothing following is looked at by the command queue.

>>> msg = message_from_string("""\
... From: cperson@example.com
... To: test-request@example.com
... Message-ID: <caribou>
...
... echo foo bar
... end ignored
... echo baz qux
... """)

>>> filebase = inject_message(mlist, msg, switchboard='command')
>>> command.run()
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
Subject: The results of your email commands
...

- Results:
echo foo bar

- Unprocessed:
echo baz qux

- Done.

The stop command is an alias for end.

>>> msg = message_from_string("""\
... From: cperson@example.com
... To: test-request@example.com
... Message-ID: <caribou>
...
... echo foo bar
... stop ignored
... echo baz qux
... """)

>>> filebase = inject_message(mlist, msg, switchboard='command')
>>> command.run()
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
Subject: The results of your email commands
...

- Results:
echo foo bar

- Unprocessed:
echo baz qux

- Done.

Digesting¶

Mailman crafts and sends digests by a separate digest runner process. This starts by a number of messages being posted to the mailing list.

>>> mlist = create_list('test@example.com')
>>> mlist.digest_size_threshold = 0.6
>>> mlist.volume = 1
>>> mlist.next_digest_number = 1
>>> mlist.send_welcome_message = False

>>> from string import Template
>>> process = config.handlers['to-digest'].process

>>> def fill_digest():
...     size = 0
...     for i in range(1, 5):
...         text = Template("""\
... From: aperson@example.com
... To: xtest@example.com
... Subject: Test message $i
... List-Post: <test@example.com>
...
... Here is message $i
... """).substitute(i=i)
...         msg = message_from_string(text)
...         process(mlist, msg, {})
...         size += len(text)
...         if size >= mlist.digest_size_threshold * 1024:
...             break

>>> fill_digest()

The runner gets kicked off when a marker message gets dropped into the digest queue. The message metadata points to the mailbox file containing the messages to put in the digest.

>>> digestq = config.switchboards['digest']
>>> len(digestq.files)
1

>>> from mailman.testing.helpers import get_queue_messages
>>> entry = get_queue_messages('digest')[0]

The marker message is empty.

>>> print(entry.msg.as_string())

But the message metadata has a reference to the digest file.

>>> dump_msgdata(entry.msgdata)
_parsemsg    : False
digest_number: 1
digest_path  : .../lists/test@example.com/digest.1.1.mmdf
listid       : test.example.com
version      : 3
volume       : 1

There are 4 messages in the digest.

>>> from mailman.utilities.mailbox import Mailbox
>>> sum(1 for item in Mailbox(entry.msgdata['digest_path']))
4

When the runner runs, it processes the digest mailbox, crafting both the plain text (RFC 1153) digest and the MIME digest.

>>> from mailman.runners.digest import DigestRunner
>>> from mailman.testing.helpers import make_testable_runner
>>> runner = make_testable_runner(DigestRunner)
>>> runner.run()

If there are no members receiving digests, none are sent.

>>> messages = get_queue_messages('virgin')
>>> len(messages)
0

Once some users are subscribed and receiving digests, the digest runner places both digests into the virgin queue for final delivery.

>>> from mailman.testing.helpers import subscribe
>>> from mailman.interfaces.member import DeliveryMode

>>> anne = subscribe(mlist, 'Anne')
>>> anne.preferences.delivery_mode = DeliveryMode.mime_digests
>>> bart = subscribe(mlist, 'Bart')
>>> bart.preferences.delivery_mode = DeliveryMode.plaintext_digests

>>> fill_digest()
>>> runner.run()
>>> messages = get_queue_messages('virgin')
>>> len(messages)
2

Anne and Bart unsubscribe from the mailing list.

>>> anne.unsubscribe()
>>> bart.unsubscribe()

The MIME digest is a multipart, and the RFC 1153 digest is the other one.

>>> def mime_rfc1153(messages):
...     if messages[0].msg.is_multipart():
...         return messages[0], messages[1]
...     return messages[1], messages[0]

>>> mime, rfc1153 = mime_rfc1153(messages)

The MIME digest has lots of good stuff, all contained in the multipart.

>>> print(mime.msg.as_string())
Content-Type: multipart/mixed; boundary="===============...=="
MIME-Version: 1.0
From: test-request@example.com
Subject: Test Digest, Vol 1, Issue 2
To: test@example.com
Reply-To: test@example.com
Date: ...
Message-ID: ...
<BLANKLINE>
--===============...==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: Test Digest, Vol 1, Issue 2
<BLANKLINE>
Send Test mailing list submissions to
    test@example.com
<BLANKLINE>
To subscribe or unsubscribe via the World Wide Web, visit
    http://lists.example.com/listinfo/test@example.com
or, via email, send a message with subject or body 'help' to
    test-request@example.com
<BLANKLINE>
You can reach the person managing the list at
    test-owner@example.com
<BLANKLINE>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Test digest..."
<BLANKLINE>
--===============...==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: Today's Topics (4 messages)
<BLANKLINE>
Today's Topics:
<BLANKLINE>
   1. Test message 1 (aperson@example.com)
   2. Test message 2 (aperson@example.com)
   3. Test message 3 (aperson@example.com)
   4. Test message 4 (aperson@example.com)
<BLANKLINE>
--===============...==
Content-Type: message/rfc822
MIME-Version: 1.0
<BLANKLINE>
From: aperson@example.com
To: xtest@example.com
Subject: Test message 1
List-Post: <test@example.com>
<BLANKLINE>
Here is message 1
<BLANKLINE>
--===============...==
Content-Type: message/rfc822
MIME-Version: 1.0
<BLANKLINE>
From: aperson@example.com
To: xtest@example.com
Subject: Test message 2
List-Post: <test@example.com>
<BLANKLINE>
Here is message 2
<BLANKLINE>
--===============...==
Content-Type: message/rfc822
MIME-Version: 1.0
<BLANKLINE>
From: aperson@example.com
To: xtest@example.com
Subject: Test message 3
List-Post: <test@example.com>
<BLANKLINE>
Here is message 3
<BLANKLINE>
--===============...==
Content-Type: message/rfc822
MIME-Version: 1.0
<BLANKLINE>
From: aperson@example.com
To: xtest@example.com
Subject: Test message 4
List-Post: <test@example.com>
<BLANKLINE>
Here is message 4
<BLANKLINE>
--===============...==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Description: Digest Footer
<BLANKLINE>
_______________________________________________
Test mailing list
test@example.com
http://lists.example.com/listinfo/test@example.com
<BLANKLINE>
--===============...==--

The RFC 1153 contains the digest in a single plain text message.

>>> print(rfc1153.msg.as_string())
From: test-request@example.com
Subject: Test Digest, Vol 1, Issue 2
To: test@example.com
Reply-To: test@example.com
Date: ...
Message-ID: ...
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
<BLANKLINE>
Send Test mailing list submissions to
    test@example.com
<BLANKLINE>
To subscribe or unsubscribe via the World Wide Web, visit
    http://lists.example.com/listinfo/test@example.com
or, via email, send a message with subject or body 'help' to
    test-request@example.com
<BLANKLINE>
You can reach the person managing the list at
    test-owner@example.com
<BLANKLINE>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Test digest..."
<BLANKLINE>
Today's Topics:
<BLANKLINE>
   1. Test message 1 (aperson@example.com)
   2. Test message 2 (aperson@example.com)
   3. Test message 3 (aperson@example.com)
   4. Test message 4 (aperson@example.com)
<BLANKLINE>
<BLANKLINE>
----------------------------------------------------------------------
<BLANKLINE>
From: aperson@example.com
Subject: Test message 1
To: xtest@example.com
<BLANKLINE>
Here is message 1
<BLANKLINE>
------------------------------
<BLANKLINE>
From: aperson@example.com
Subject: Test message 2
To: xtest@example.com
<BLANKLINE>
Here is message 2
<BLANKLINE>
------------------------------
<BLANKLINE>
From: aperson@example.com
Subject: Test message 3
To: xtest@example.com
<BLANKLINE>
Here is message 3
<BLANKLINE>
------------------------------
<BLANKLINE>
From: aperson@example.com
Subject: Test message 4
To: xtest@example.com
<BLANKLINE>
Here is message 4
<BLANKLINE>
------------------------------
<BLANKLINE>
Subject: Digest Footer
<BLANKLINE>
_______________________________________________
Test mailing list
test@example.com
http://lists.example.com/listinfo/test@example.com
<BLANKLINE>
<BLANKLINE>
------------------------------
<BLANKLINE>
End of Test Digest, Vol 1, Issue 2
**********************************
<BLANKLINE>

Digest delivery¶

A mailing list’s members can choose to receive normal delivery, plain text digests, or MIME digests.

>>> len(get_queue_messages('virgin'))
0

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)

>>> from mailman.interfaces.member import DeliveryMode, MemberRole
>>> def subscribe(email, mode):
...     address = user_manager.create_address(email)
...     member = mlist.subscribe(address, MemberRole.member)
...     member.preferences.delivery_mode = mode
...     return member

Two regular delivery members subscribe to the mailing list.

>>> member_1 = subscribe('uperson@example.com', DeliveryMode.regular)
>>> member_2 = subscribe('vperson@example.com', DeliveryMode.regular)

Two MIME digest members subscribe to the mailing list.

>>> member_3 = subscribe('wperson@example.com', DeliveryMode.mime_digests)
>>> member_4 = subscribe('xperson@example.com', DeliveryMode.mime_digests)

One RFC 1153 digest member subscribes to the mailing list.

>>> member_5 = subscribe(
...     'yperson@example.com', DeliveryMode.plaintext_digests)
>>> member_6 = subscribe(
...     'zperson@example.com', DeliveryMode.plaintext_digests)

When a digest gets sent, the appropriate recipient list is chosen.

>>> mlist.preferred_language = 'en'
>>> mlist.digest_size_threshold = 0.5
>>> fill_digest()
>>> runner.run()

The digests are sitting in the virgin queue. One of them is the MIME digest and the other is the RFC 1153 digest.

>>> messages = get_queue_messages('virgin')
>>> len(messages)
2

>>> mime, rfc1153 = mime_rfc1153(messages)

Only wperson and xperson get the MIME digests.

>>> sorted(mime.msgdata['recipients'])
['wperson@example.com', 'xperson@example.com']

Only yperson and zperson get the RFC 1153 digests.

>>> sorted(rfc1153.msgdata['recipients'])
['yperson@example.com', 'zperson@example.com']

Now uperson decides that they would like to start receiving digests too.

>>> member_1.preferences.delivery_mode = DeliveryMode.mime_digests
>>> fill_digest()
>>> runner.run()

>>> messages = get_queue_messages('virgin')
>>> len(messages)
2

>>> mime, rfc1153 = mime_rfc1153(messages)
>>> sorted(mime.msgdata['recipients'])
['uperson@example.com', 'wperson@example.com', 'xperson@example.com']

>>> sorted(rfc1153.msgdata['recipients'])
['yperson@example.com', 'zperson@example.com']

At this point, both uperson and wperson decide that they’d rather receive regular deliveries instead of digests. uperson would like to get any last digest that may be sent so that she doesn’t miss anything. wperson does care as much and does not want to receive one last digest.

>>> mlist.send_one_last_digest_to(
...     member_1.address, member_1.preferences.delivery_mode)

>>> member_1.preferences.delivery_mode = DeliveryMode.regular
>>> member_3.preferences.delivery_mode = DeliveryMode.regular

>>> fill_digest()
>>> runner.run()

>>> messages = get_queue_messages('virgin')
>>> mime, rfc1153 = mime_rfc1153(messages)
>>> sorted(mime.msgdata['recipients'])
['uperson@example.com', 'xperson@example.com']

>>> sorted(rfc1153.msgdata['recipients'])
['yperson@example.com', 'zperson@example.com']

Since uperson has received their last digest, they will not get any more of them.

>>> fill_digest()
>>> runner.run()

>>> messages = get_queue_messages('virgin')
>>> len(messages)
2

>>> mime, rfc1153 = mime_rfc1153(messages)
>>> sorted(mime.msgdata['recipients'])
['xperson@example.com']

>>> sorted(rfc1153.msgdata['recipients'])
['yperson@example.com', 'zperson@example.com']

The incoming runner¶

This runner’s sole purpose in life is to decide the disposition of the message. It can either be accepted for delivery, rejected (i.e. bounced), held for moderator approval, or discarded.

The runner operates by processing chains on a message/metadata pair in the context of a mailing list. Each mailing list has a default chain for messages posted to the mailing list. This chain is processed with the message eventually ending up in one of the four disposition states described above.

>>> mlist = create_list('test@example.com')
>>> print(mlist.posting_chain)
default-posting-chain

Sender addresses¶

The incoming runner ensures that the sender addresses on the message are registered with the system. This is used for determining nonmember posting privileges. The addresses will not be linked to a user and will be unverified, so if the real user comes along later and claims the address, it will be linked to their user account (and must be verified).

While configurable, the sender addresses by default are those named in the From:, Sender:, and Reply-To: headers, as well as the envelope sender (though we won’t worry about the latter).

>>> msg = message_from_string("""\
... From: zperson@example.com
... Reply-To: yperson@example.com
... Sender: xperson@example.com
... To: test@example.com
... Subject: This is spiced ham
... Message-ID: <bogus>
...
... """)

>>> from zope.component import getUtility
>>> from mailman.interfaces.usermanager import IUserManager
>>> user_manager = getUtility(IUserManager)
>>> print(user_manager.get_address('xperson@example.com'))
None
>>> print(user_manager.get_address('yperson@example.com'))
None
>>> print(user_manager.get_address('zperson@example.com'))
None

Inject the message into the incoming queue, similar to the way the upstream mail server normally would.

>>> from mailman.app.inject import inject_message
>>> filebase = inject_message(mlist, msg)

The incoming runner runs until it is empty.

>>> from mailman.runners.incoming import IncomingRunner
>>> from mailman.testing.helpers import make_testable_runner
>>> incoming = make_testable_runner(IncomingRunner, 'in')
>>> incoming.run()

And now the addresses are known to the system. As mentioned above, they are not linked to a user and are unverified.

>>> for localpart in ('xperson', 'yperson', 'zperson'):
...     email = '{0}@example.com'.format(localpart)
...     address = user_manager.get_address(email)
...     print('{0}; verified? {1}; user? {2}'.format(
...           address.email,
...           ('No' if address.verified_on is None else 'Yes'),
...           user_manager.get_user(email)))
xperson@example.com; verified? No; user? None
yperson@example.com; verified? No; user? None
zperson@example.com; verified? No; user? None

Accepted messages¶

We have a message that is going to be sent to the mailing list. Once Anne is a member of the mailing list, this message is so perfectly fine for posting that it will be accepted and forward to the pipeline queue.

>>> from mailman.testing.helpers import subscribe
>>> subscribe(mlist, 'Anne')
<Member: Anne Person <aperson@example.com> on test@example.com
         as MemberRole.member>

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: My first post
... Message-ID: <first>
...
... First post!
... """)

Inject the message into the incoming queue and run until the queue is empty.

>>> filebase = inject_message(mlist, msg)
>>> incoming.run()

There are no messages left in the incoming queue.

>>> get_queue_messages('in')
[]

Now the message is in the pipeline queue.

>>> messages = get_queue_messages('pipeline')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
From: aperson@example.com
To: test@example.com
Subject: My first post
Message-ID: <first>
X-Message-ID-Hash: 4CMWUN6BHVCMHMDAOSJZ2Q72G5M32MWB
Date: ...
X-Mailman-Rule-Misses: approved; emergency; loop; member-moderation;
    administrivia; implicit-dest; max-recipients; max-size;
    news-moderation; no-subject; suspicious-header; nonmember-moderation
<BLANKLINE>
First post!
<BLANKLINE>
>>> dump_msgdata(messages[0].msgdata)
_parsemsg    : False
envsender    : noreply@example.com
...

Held messages¶

The list moderator sets the emergency flag on the mailing list. The built-in chain will now hold all posted messages, so nothing will show up in the pipeline queue.

>>> from mailman.interfaces.chain import ChainEvent
>>> def on_chain(event):
...     if isinstance(event, ChainEvent):
...         print(event)
...         print(event.chain)
...         print('From: {0}\nTo: {1}\nMessage-ID: {2}'.format(
...             event.msg['from'], event.msg['to'],
...             event.msg['message-id']))

>>> mlist.emergency = True

>>> from mailman.testing.helpers import event_subscribers
>>> with event_subscribers(on_chain):
...     filebase = inject_message(mlist, msg)
...     incoming.run()
<mailman.interfaces.chain.HoldEvent ...>
<mailman.chains.hold.HoldChain ...>
From: aperson@example.com
To: test@example.com
Message-ID: <first>

>>> mlist.emergency = False

Discarded messages¶

Another possibility is that the message would get immediately discarded. The built-in chain does not have such a disposition by default, so let’s craft a new chain and set it as the mailing list’s start chain.

>>> from mailman.chains.base import Chain, Link
>>> from mailman.interfaces.chain import LinkAction
>>> def make_chain(name, target_chain):
...     truth_rule = config.rules['truth']
...     target_chain = config.chains[target_chain]
...     test_chain = Chain(name, 'Testing {0}'.format(target_chain))
...     config.chains[test_chain.name] = test_chain
...     link = Link(truth_rule, LinkAction.jump, target_chain)
...     test_chain.append_link(link)
...     return test_chain

>>> test_chain = make_chain('always-discard', 'discard')
>>> mlist.posting_chain = test_chain.name

>>> msg.replace_header('message-id', '<second>')
>>> with event_subscribers(on_chain):
...     filebase = inject_message(mlist, msg)
...     incoming.run()
<mailman.interfaces.chain.DiscardEvent ...>
<mailman.chains.discard.DiscardChain ...>
From: aperson@example.com
To: test@example.com
Message-ID: <second>

>>> del config.chains[test_chain.name]

Rejected messages¶

Similar to discarded messages, a message can be rejected, or bounced back to the original sender. Again, the built-in chain doesn’t support this so we’ll just create a new chain that does.

>>> test_chain = make_chain('always-reject', 'reject')
>>> mlist.posting_chain = test_chain.name

>>> msg.replace_header('message-id', '<third>')
>>> with event_subscribers(on_chain):
...     filebase = inject_message(mlist, msg)
...     incoming.run()
<mailman.interfaces.chain.RejectEvent ...>
<mailman.chains.reject.RejectChain ...>
From: aperson@example.com
To: test@example.com
Message-ID: <third>

The rejection message is sitting in the virgin queue waiting to be delivered to the original sender.

>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
Subject: My first post
From: test-owner@example.com
To: aperson@example.com
...
<BLANKLINE>
--===============...
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
<BLANKLINE>
[No bounce details are available]
--===============...
Content-Type: message/rfc822
MIME-Version: 1.0
<BLANKLINE>
From: aperson@example.com
To: test@example.com
Subject: My first post
Message-ID: <third>
Date: ...
<BLANKLINE>
First post!
<BLANKLINE>
--===============...

>>> del config.chains['always-reject']

LTMP server¶

Mailman can accept messages via LMTP (RFC 2033). Most modern mail servers support LMTP local delivery, so this is a very portable way to connect Mailman with your mail server.

Our LMTP server is fairly simple though; all it does is make sure that the message is destined for a valid endpoint, e.g. mylist-join@example.com, that the message bytes can be parsed into a message object, and that the message has a Message-ID header.

Let’s start a testable LMTP runner.

>>> from mailman.testing import helpers
>>> master = helpers.TestableMaster()
>>> master.start('lmtp')

It also helps to have a nice LMTP client.

>>> lmtp = helpers.get_lmtp_client()
(220, b'... GNU Mailman LMTP runner 1.1')
>>> lmtp.lhlo('remote.example.org')
(250, ...)

Posting address¶

Once the mailing list is created, the posting address is valid, and messages can be sent to the list.

>>> create_list('mylist@example.com')
<mailing list "mylist@example.com" at ...>

>>> transaction.commit()
>>> lmtp.sendmail(
...     'anne.person@example.com',
...     ['mylist@example.com'], """\
... From: anne.person@example.com
... To: mylist@example.com
... Subject: An interesting message
... Message-ID: <badger>
...
... This is an interesting message.
... """)
{}

Since the message itself is valid, it gets parsed and lands in the incoming queue.

>>> from mailman.testing.helpers import get_queue_messages
>>> messages = get_queue_messages('in')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
From: anne.person@example.com
To: mylist@example.com
Subject: An interesting message
Message-ID: <badger>
X-Message-ID-Hash: JYMZWSQ4IC2JPKK7ZUONRFRVC4ZYJGKJ
X-MailFrom: anne.person@example.com
<BLANKLINE>
This is an interesting message.
>>> dump_msgdata(messages[0].msgdata)
_parsemsg    : False
listid       : mylist.example.com
original_size: ...
to_list      : True
version      : ...

Sub-addresses¶

The LMTP server understands each of the list’s sub-addreses, such as -join, -leave, -request and so on. The message is accepted if posted to a valid sub-address.

>>> lmtp.sendmail(
...     'anne.person@example.com',
...     ['mylist-request@example.com'], """\
... From: anne.person@example.com
... To: mylist-request@example.com
... Subject: Help
... Message-ID: <dog>
...
... Please help me.
... """)
{}

Request subaddress¶

Depending on the subaddress, there is a message in the appropriate queue for later processing. For example, all -request messages are put into the command queue for processing.

>>> messages = get_queue_messages('command')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
From: anne.person@example.com
To: mylist-request@example.com
Subject: Help
Message-ID: <dog>
X-Message-ID-Hash: 4SKREUSPI62BHDMFBSOZ3BMXFETSQHNA
X-MailFrom: anne.person@example.com
<BLANKLINE>
Please help me.
>>> dump_msgdata(messages[0].msgdata)
_parsemsg    : False
listid       : mylist.example.com
original_size: ...
subaddress   : request
version      : ...

Bounce processor¶

A message to the -bounces address goes to the bounce processor.

>>> lmtp.sendmail(
...     'mail-daemon@example.com',
...     ['mylist-bounces@example.com'], """\
... From: mail-daemon@example.com
... To: mylist-bounces@example.com
... Subject: A bounce
... Message-ID: <elephant>
...
... Bouncy bouncy.
... """)
{}
>>> messages = get_queue_messages('bounces')
>>> len(messages)
1
>>> dump_msgdata(messages[0].msgdata)
_parsemsg    : False
listid       : mylist.example.com
original_size: ...
subaddress   : bounces
version      : ...

Command processor¶

Confirmation messages go to the command processor…

>>> lmtp.sendmail(
...     'anne.person@example.com',
...     ['mylist-confirm@example.com'], """\
... From: anne.person@example.com
... To: mylist-confirm@example.com
... Subject: A bounce
... Message-ID: <falcon>
...
... confirm 123
... """)
{}
>>> messages = get_queue_messages('command')
>>> len(messages)
1
>>> dump_msgdata(messages[0].msgdata)
_parsemsg    : False
listid       : mylist.example.com
original_size: ...
subaddress   : confirm
version      : ...

…as do join messages…

>>> lmtp.sendmail(
...     'anne.person@example.com',
...     ['mylist-join@example.com'], """\
... From: anne.person@example.com
... To: mylist-join@example.com
... Message-ID: <giraffe>
...
... """)
{}
>>> messages = get_queue_messages('command')
>>> len(messages)
1
>>> dump_msgdata(messages[0].msgdata)
_parsemsg    : False
listid       : mylist.example.com
original_size: ...
subaddress   : join
version      : ...

>>> lmtp.sendmail(
...     'anne.person@example.com',
...     ['mylist-subscribe@example.com'], """\
... From: anne.person@example.com
... To: mylist-subscribe@example.com
... Message-ID: <hippopotamus>
...
... """)
{}
>>> messages = get_queue_messages('command')
>>> len(messages)
1
>>> dump_msgdata(messages[0].msgdata)
_parsemsg    : False
listid       : mylist.example.com
original_size: ...
subaddress   : join
version      : ...

…and leave messages.

>>> lmtp.sendmail(
...     'anne.person@example.com',
...     ['mylist-leave@example.com'], """\
... From: anne.person@example.com
... To: mylist-leave@example.com
... Message-ID: <iguana>
...
... """)
{}
>>> messages = get_queue_messages('command')
>>> len(messages)
1
>>> dump_msgdata(messages[0].msgdata)
_parsemsg    : False
listid       : mylist.example.com
original_size: ...
subaddress   : leave
version      : ...

>>> lmtp.sendmail(
...     'anne.person@example.com',
...     ['mylist-unsubscribe@example.com'], """\
... From: anne.person@example.com
... To: mylist-unsubscribe@example.com
... Message-ID: <jackal>
...
... """)
{}
>>> messages = get_queue_messages('command')
>>> len(messages)
1
>>> dump_msgdata(messages[0].msgdata)
_parsemsg    : False
listid       : mylist.example.com
original_size: ...
subaddress   : leave
version      : ...

Incoming processor¶

Messages to the -owner address also go to the incoming processor.

>>> lmtp.sendmail(
...     'anne.person@example.com',
...     ['mylist-owner@example.com'], """\
... From: anne.person@example.com
... To: mylist-owner@example.com
... Message-ID: <kangaroo>
...
... """)
{}
>>> messages = get_queue_messages('in')
>>> len(messages)
1
>>> dump_msgdata(messages[0].msgdata)
_parsemsg    : False
envsender    : noreply@example.com
listid       : mylist.example.com
original_size: ...
subaddress   : owner
to_owner     : True
version      : ...

The NNTP runner¶

The NNTP runner gateways mailing list messages to an NNTP newsgroup.

>>> mlist = create_list('test@example.com')
>>> mlist.linked_newsgroup = 'comp.lang.python'

Get a handle on the NNTP server, which we’ll use later to verify the posted messages.

>>> from mailman.testing.helpers import get_nntp_server
>>> nntpd = get_nntp_server(cleanups)

A message gets posted to the mailing list. It may contain some headers which are prohibited by NNTP servers such as INN.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... NNTP-Posting-Host: news.example.com
... NNTP-Posting-Date: today
... X-Trace: blah blah
... X-Complaints-To: abuse@dom.ain
... Xref: blah blah
... Xref: blah blah
... Date-Received: yesterday
... Posted: tomorrow
... Posting-Version: 99.99
... Relay-Version: 88.88
... Received: blah blah
...
... A message
... """)

The message gets copied to the NNTP queue for preparation and posting.

>>> filebase = config.switchboards['nntp'].enqueue(
...     msg, listid='test.example.com')
>>> from mailman.testing.helpers import make_testable_runner
>>> from mailman.runners.nntp import NNTPRunner
>>> runner = make_testable_runner(NNTPRunner, 'nntp')
>>> runner.run()

The message was successfully posted the NNTP server.

>>> print(nntpd.get_message().as_string())
From: aperson@example.com
To: test@example.com
Newsgroups: comp.lang.python
Message-ID: ...
Lines: 1
<BLANKLINE>
A message
<BLANKLINE>

Outgoing runner¶

The outgoing runner is the process that delivers messages to the directly upstream SMTP server. It is this upstream SMTP server that performs final delivery to the intended recipients.

Messages that appear in the outgoing queue are processed individually through a delivery module, essentially a pluggable interface for determining how the recipient set will be batched, whether messages will be personalized and VERP’d, etc. The outgoing runner doesn’t itself support retrying but it can move messages to the ‘retry queue’ for handling delivery failures.

>>> mlist = create_list('test@example.com')

>>> from mailman.testing.helpers import subscribe
>>> subscribe(mlist, 'Anne')
<Member: Anne Person <aperson@example.com>
         on test@example.com as MemberRole.member>

>>> subscribe(mlist, 'Bart')
<Member: Bart Person <bperson@example.com>
         on test@example.com as MemberRole.member>

>>> subscribe(mlist, 'Cris')
<Member: Cris Person <cperson@example.com>
         on test@example.com as MemberRole.member>

Normally, messages would show up in the outgoing queue after the message has been processed by the rule set and pipeline. But we can simulate that here by injecting a message directly into the outgoing queue. First though, we must call the member-recipients handler so that the message metadata will be populated with the list of addresses to deliver the message to.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: My first post
... Message-ID: <first>
...
... First post!
... """)

>>> msgdata = {}
>>> handler = config.handlers['member-recipients']
>>> handler.process(mlist, msg, msgdata)
>>> outgoing_queue = config.switchboards['out']

The to-outgoing handler populates the message metadata with the destination mailing list name. Simulate that here too.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     tolist=True,
...     listid=mlist.list_id)

Running the outgoing runner processes the message, delivering it to the upstream SMTP.

>>> from mailman.runners.outgoing import OutgoingRunner
>>> from mailman.testing.helpers import make_testable_runner
>>> outgoing = make_testable_runner(OutgoingRunner, 'out')
>>> outgoing.run()

Every recipient got the same copy of the message.

>>> messages = list(smtpd.messages)
>>> len(messages)
1

>>> print(messages[0].as_string())
From: aperson@example.com
To: test@example.com
Subject: My first post
Message-ID: <first>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: aperson@example.com, bperson@example.com, cperson@example.com

First post!

Personalization¶

Mailman supports sending individual messages to each recipient by personalizing delivery. This increases the bandwidth between Mailman and the upstream mail server, and between the upstream mail server and the remote recipient mail servers. The benefit is that personalization provides for a much better user experience, because the messages can be tailored for each recipient.

>>> from mailman.interfaces.mailinglist import Personalization
>>> mlist.personalize = Personalization.individual
>>> transaction.commit()

Now when we send the message, our mail server will get three copies instead of just one.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
3

Since we’ve done no other configuration, the only difference in the messages is the recipient address. Specifically, the Sender header is the same for all recipients.

>>> from operator import itemgetter
>>> def show_headers(messages):
...     for message in sorted(messages, key=itemgetter('x-rcptto')):
...         print(message['X-RcptTo'], message['X-MailFrom'])

>>> show_headers(messages)
aperson@example.com   test-bounces@example.com
bperson@example.com   test-bounces@example.com
cperson@example.com   test-bounces@example.com

VERP¶

An even more interesting personalization opportunity arises if VERP is enabled. Here, Mailman takes advantage of the fact that it’s sending individualized messages anyway, so it also encodes the recipients address in the Sender header.

Forcing VERP¶

A handler can force VERP by setting the verp key in the message metadata.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     verp=True,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> show_headers(messages)
aperson@example.com   test-bounces+aperson=example.com@example.com
bperson@example.com   test-bounces+bperson=example.com@example.com
cperson@example.com   test-bounces+cperson=example.com@example.com

VERP personalized deliveries¶

The site administrator can enable VERP whenever messages are personalized.

>>> config.push('verp', """
... [mta]
... verp_personalized_deliveries: yes
... """)

Again, we get three individual messages, with VERP’d Sender headers.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> show_headers(messages)
aperson@example.com   test-bounces+aperson=example.com@example.com
bperson@example.com   test-bounces+bperson=example.com@example.com
cperson@example.com   test-bounces+cperson=example.com@example.com

>>> config.pop('verp')
>>> mlist.personalize = Personalization.none
>>> transaction.commit()

VERP every once in a while¶

Perhaps personalization is too much of an overhead, but the list owners would still like to occasionally get the benefits of VERP. The site administrator can enable occasional VERPing of messages every so often, by setting a delivery interval. Every N non-personalized deliveries turns on VERP for just the next one.

>>> config.push('verp occasionally', """
... [mta]
... verp_delivery_interval: 3
... """)

# Reset the list's post_id, which is used to calculate the intervals.
>>> mlist.post_id = 1
>>> transaction.commit()

The first message is sent to the list, and it is neither personalized nor VERP’d.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
1

>>> show_headers(messages)
aperson@example.com, bperson@example.com, cperson@example.com
test-bounces@example.com

# Perform post-delivery bookkeeping.
>>> after = config.handlers['after-delivery']
>>> after.process(mlist, msg, msgdata)
>>> transaction.commit()

The second message sent to the list is also not VERP’d.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
1

>>> show_headers(messages)
aperson@example.com, bperson@example.com, cperson@example.com
test-bounces@example.com

# Perform post-delivery bookkeeping.
>>> after.process(mlist, msg, msgdata)
>>> transaction.commit()

The third message though is VERP’d.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> show_headers(messages)
aperson@example.com   test-bounces+aperson=example.com@example.com
bperson@example.com   test-bounces+bperson=example.com@example.com
cperson@example.com   test-bounces+cperson=example.com@example.com

# Perform post-delivery bookkeeping.
>>> after.process(mlist, msg, msgdata)
>>> transaction.commit()

The next one is back to bulk delivery.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
1

>>> show_headers(messages)
aperson@example.com, bperson@example.com, cperson@example.com
test-bounces@example.com

>>> config.pop('verp occasionally')

VERP every time¶

If the site administrator wants to enable VERP for every delivery, even if no personalization is going on, they can set the interval to 1.

>>> config.push('always verp', """
... [mta]
... verp_delivery_interval: 1
... """)

# Reset the list's post_id, which is used to calculate the intervals.
>>> mlist.post_id = 1
>>> transaction.commit()

The first message is VERP’d.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> show_headers(messages)
aperson@example.com   test-bounces+aperson=example.com@example.com
bperson@example.com   test-bounces+bperson=example.com@example.com
cperson@example.com   test-bounces+cperson=example.com@example.com

# Perform post-delivery bookkeeping.
>>> after.process(mlist, msg, msgdata)
>>> transaction.commit()

As is the second message.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> show_headers(messages)
aperson@example.com   test-bounces+aperson=example.com@example.com
bperson@example.com   test-bounces+bperson=example.com@example.com
cperson@example.com   test-bounces+cperson=example.com@example.com

# Perform post-delivery bookkeeping.
>>> after.process(mlist, msg, msgdata)
>>> transaction.commit()

And the third message.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> show_headers(messages)
aperson@example.com   test-bounces+aperson=example.com@example.com
bperson@example.com   test-bounces+bperson=example.com@example.com
cperson@example.com   test-bounces+cperson=example.com@example.com

# Perform post-delivery bookkeeping.
>>> after.process(mlist, msg, msgdata)
>>> transaction.commit()

>>> config.pop('always verp')

Never VERP¶

Similarly, the site administrator can disable occasional VERP’ing of non-personalized messages by setting the interval to zero.

>>> config.push('never verp', """
... [mta]
... verp_delivery_interval: 0
... """)

# Reset the list's post_id, which is used to calculate the intervals.
>>> mlist.post_id = 1
>>> transaction.commit()

Neither the first message…

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
1

>>> show_headers(messages)
aperson@example.com, bperson@example.com, cperson@example.com
test-bounces@example.com

…nor the second message is VERP’d.

>>> ignore = outgoing_queue.enqueue(
...     msg, msgdata,
...     listid=mlist.list_id)
>>> outgoing.run()
>>> messages = list(smtpd.messages)
>>> len(messages)
1

>>> show_headers(messages)
aperson@example.com, bperson@example.com, cperson@example.com
test-bounces@example.com

REST server¶

Mailman is controllable through an administrative REST HTTP server.

>>> from mailman.testing import helpers
>>> master = helpers.TestableMaster(helpers.wait_for_webservice)
>>> master.start('rest')

The RESTful server can be used to access basic version information.

>>> dump_json('http://localhost:9001/3.0/system')
http_etag: "..."
mailman_version: GNU Mailman 3.0... (...)
python_version: ...
self_link: http://localhost:9001/3.0/system/versions

Clean up¶

>>> master.stop()

Moderation¶

Posts by members and nonmembers are subject to moderation checks during incoming processing. Different situations can cause such posts to be held for moderator approval.

>>> mlist = create_list('test@example.com')

Members and nonmembers have a moderation action which can shortcut the normal moderation checks. The built-in chain does just a few checks first, such as seeing if the message has a matching Approved: header, or if the emergency flag has been set on the mailing list, or whether a mail loop has been detected.

After those, the moderation action for the sender is checked. Members generally have a defer action, meaning the normal moderation checks are done, but it is also common for first-time posters to have a hold action, meaning that their messages are held for moderator approval for a while.

Nonmembers almost always have a hold action, though some mailing lists may choose to set this default action to discard, meaning their posts would be immediately thrown away.

Member moderation¶

Posts by list members are moderated if the member’s moderation action is not deferred. The default setting for the moderation action of new members is determined by the mailing list’s settings. By default, a mailing list is not set to moderate new member postings.

>>> from mailman.testing.helpers import subscribe
>>> member = subscribe(mlist, 'Anne', email='anne@example.com')
>>> member
<Member: Anne Person <anne@example.com> on test@example.com
         as MemberRole.member>
>>> print(member.moderation_action)
Action.defer

In order to find out whether the message is held or accepted, we can subscribe to Zope events that are triggered on each case.

>>> from mailman.interfaces.chain import ChainEvent
>>> def on_chain(event):
...     if isinstance(event, ChainEvent):
...         print(event)
...         print(event.chain)
...         print('Subject:', event.msg['subject'])
...         print('Hits:')
...         for hit in event.msgdata.get('rule_hits', []):
...             print('   ', hit)
...         print('Misses:')
...         for miss in event.msgdata.get('rule_misses', []):
...             print('   ', miss)

Anne’s post to the mailing list runs through the incoming runner’s default built-in chain. No rules hit and so the message is accepted.

>>> msg = message_from_string("""\
... From: anne@example.com
... To: test@example.com
... Subject: aardvark
...
... This is a test.
... """)

>>> from mailman.core.chains import process
>>> from mailman.testing.helpers import event_subscribers
>>> with event_subscribers(on_chain):
...     process(mlist, msg, {}, 'default-posting-chain')
<mailman.interfaces.chain.AcceptEvent ...>
<mailman.chains.accept.AcceptChain ...>
Subject: aardvark
Hits:
Misses:
    approved
    emergency
    loop
    member-moderation
    administrivia
    implicit-dest
    max-recipients
    max-size
    news-moderation
    no-subject
    suspicious-header
    nonmember-moderation

However, when Anne’s moderation action is set to hold, her post is held for moderator approval.

>>> from mailman.interfaces.action import Action
>>> member.moderation_action = Action.hold

>>> msg = message_from_string("""\
... From: anne@example.com
... To: test@example.com
... Subject: badger
...
... This is a test.
... """)

>>> with event_subscribers(on_chain):
...     process(mlist, msg, {}, 'default-posting-chain')
<mailman.interfaces.chain.HoldEvent ...>
<mailman.chains.hold.HoldChain ...>
Subject: badger
Hits:
    member-moderation
Misses:
    approved
    emergency
    loop

The list’s member moderation action can also be set to discard…

>>> member.moderation_action = Action.discard

>>> msg = message_from_string("""\
... From: anne@example.com
... To: test@example.com
... Subject: cougar
...
... This is a test.
... """)

>>> with event_subscribers(on_chain):
...     process(mlist, msg, {}, 'default-posting-chain')
<mailman.interfaces.chain.DiscardEvent ...>
<mailman.chains.discard.DiscardChain ...>
Subject: cougar
Hits:
    member-moderation
Misses:
    approved
    emergency
    loop

… or reject.

>>> member.moderation_action = Action.reject

>>> msg = message_from_string("""\
... From: anne@example.com
... To: test@example.com
... Subject: dingo
...
... This is a test.
... """)

>>> with event_subscribers(on_chain):
...     process(mlist, msg, {}, 'default-posting-chain')
<mailman.interfaces.chain.RejectEvent ...>
<mailman.chains.reject.RejectChain ...>
Subject: dingo
Hits:
    member-moderation
Misses:
    approved
    emergency
    loop

Nonmembers¶

Registered nonmembers are handled very similarly to members, the main difference being that they usually have a default moderation action. This is how the incoming runner adds sender addresses as nonmembers.

>>> from zope.component import getUtility
>>> from mailman.interfaces.usermanager import IUserManager
>>> user_manager = getUtility(IUserManager)
>>> address = user_manager.create_address('bart@example.com')
>>> address
<Address: bart@example.com [not verified] at ...>

When the moderation rule runs on a message from this sender, this address will be registered as a nonmember of the mailing list, and it will be held for moderator approval.

>>> msg = message_from_string("""\
... From: bart@example.com
... To: test@example.com
... Subject: elephant
...
... """)

>>> with event_subscribers(on_chain):
...     process(mlist, msg, {}, 'default-posting-chain')
<mailman.interfaces.chain.HoldEvent ...>
<mailman.chains.hold.HoldChain ...>
Subject: elephant
Hits:
    nonmember-moderation
Misses:
    approved
    emergency
    loop
    member-moderation
    administrivia
    implicit-dest
    max-recipients
    max-size
    news-moderation
    no-subject
    suspicious-header

>>> nonmember = mlist.nonmembers.get_member('bart@example.com')
>>> nonmember
<Member: bart@example.com on test@example.com as MemberRole.nonmember>
>>> print(nonmember.moderation_action)
Action.hold

Administrivia¶

The administrivia rule matches when the message contains some common email commands in the Subject: header or first few lines of the payload. This is used to catch messages posted to the list which should have been sent to the -request robot address.

>>> mlist = create_list('_xtest@example.com')
>>> mlist.administrivia = True
>>> rule = config.rules['administrivia']
>>> print(rule.name)
administrivia

For example, if the Subject: header contains the word unsubscribe, the rule matches.

>>> msg_1 = message_from_string("""\
... From: aperson@example.com
... Subject: unsubscribe
...
... """)
>>> rule.check(mlist, msg_1, {})
True

Similarly, if the body of the message contains the word subscribe in the first few lines of text, the rule matches.

>>> msg_2 = message_from_string("""\
... From: aperson@example.com
... Subject: I wish to join your list
...
... subscribe
... """)
>>> rule.check(mlist, msg_2, {})
True

In both cases, administrivia checking can be disabled.

>>> mlist.administrivia = False
>>> rule.check(mlist, msg_1, {})
False
>>> rule.check(mlist, msg_2, {})
False

To make the administrivia heuristics a little more robust, the rule actually looks for a minimum and maximum number of arguments, so that it really does seem like a mis-addressed email command. In this case, the confirm command requires at least one argument. We don’t give that here so the rule will not match.

>>> mlist.administrivia = True
>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: confirm
...
... """)
>>> rule.check(mlist, msg, {})
False

But a real confirm message will match.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: confirm 12345
...
... """)
>>> rule.check(mlist, msg, {})
True

We don’t show all the other possible email commands, but you get the idea.

Non-administrivia¶

Of course, messages that don’t contain administrivia, don’t match the rule.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: examine
...
... persuade
... """)
>>> rule.check(mlist, msg, {})
False

Also, only text/plain parts are checked for administrivia, so any email commands in other content type subparts are ignored.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: some administrivia
... Content-Type: text/x-special
...
... subscribe
... """)
>>> rule.check(mlist, msg, {})
False

Pre-approved postings¶

Messages can contain a pre-approval, which is used to bypass the normal message approval queue. This has several use cases:

A list administrator can send an emergency message to the mailing list from an unregistered address, for example if they are away from their normal email.

An automated script can be programmed to send a message to an otherwise moderated list.

In order to support this, a mailing list can be given a moderator password which is shared among all the administrators.

>>> mlist = create_list('test@example.com')

This password will not be stored in clear text, so it must be hashed using the configured hash protocol.

>>> mlist.moderator_password = config.password_context.encrypt(
...     'super secret')

The approved rule determines whether the message contains the proper approval or not.

>>> rule = config.rules['approved']
>>> print(rule.name)
approved

No approval¶

The preferred header to check for approval is Approved:. If the message does not have this header, the rule will not match.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... An important message.
... """)
>>> rule.check(mlist, msg, {})
False

If the rule has an Approved header, but the value of this header does not match the moderator password, the rule will not match. Note that the header must contain the clear text version of the password.

>>> msg['Approved'] = 'not the password'
>>> rule.check(mlist, msg, {})
False

The message is approved¶

By adding an Approved header with a matching password, the rule will match.

>>> del msg['approved']
>>> msg['Approved'] = 'super secret'
>>> rule.check(mlist, msg, {})
True

Alternative headers¶

Other headers can be used to stash the moderator password. This rule also checks the Approve header.

>>> del msg['approved']
>>> msg['Approve'] = 'super secret'
>>> rule.check(mlist, msg, {})
True

Similarly, an X-Approved header can be used.

>>> del msg['approve']
>>> msg['X-Approved'] = 'super secret'
>>> rule.check(mlist, msg, {})
True

And finally, an X-Approve header can be used.

>>> del msg['x-approved']
>>> msg['X-Approve'] = 'super secret'
>>> rule.check(mlist, msg, {})
True

Removal of header¶

Technically, rules should not have side-effects, however this rule does remove the Approved header (LP: #973790) when it matches.

>>> del msg['x-approved']
>>> msg['Approved'] = 'super secret'
>>> rule.check(mlist, msg, {})
True
>>> print(msg['approved'])
None

It also removes the header when it doesn’t match. If the rule didn’t do this, then the mailing list could be probed for its moderator password.

>>> msg['Approved'] = 'not the password'
>>> rule.check(mlist, msg, {})
False
>>> print(msg['approved'])
None

Using a pseudo-header¶

Mail programs have varying degrees to which they support custom headers like Approved:. For this reason, Mailman also supports using a pseudo-header, which is really just the first non-whitespace line in the payload of the message. If this pseudo-header looks like a matching Approved: header, the message is similarly allowed to pass.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... Approved: super secret
... An important message.
... """)
>>> rule.check(mlist, msg, {})
True

The pseudo-header is always removed from the body of plain text messages.

>>> print(msg.as_string())
From: aperson@example.com
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
<BLANKLINE>
An important message.
<BLANKLINE>

As before, a mismatch in the pseudo-header does not approve the message, but the pseudo-header line is still removed.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... Approved: not the password
... An important message.
... """)
>>> rule.check(mlist, msg, {})
False

>>> print(msg.as_string())
From: aperson@example.com
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"

An important message.

MIME multipart support¶

Mailman searches for the pseudo-header as the first non-whitespace line in the first text/plain message part of the message. This allows the feature to be used with MIME documents.

>>> msg = message_from_string("""\
... From: aperson@example.com
... MIME-Version: 1.0
... Content-Type: multipart/mixed; boundary="AAA"
...
... --AAA
... Content-Type: application/x-ignore
...
... Approved: not the password
... The above line will be ignored.
...
... --AAA
... Content-Type: text/plain
...
... Approved: super secret
... An important message.
... --AAA--
... """)
>>> rule.check(mlist, msg, {})
True

Like before, the pseudo-header is removed, but only from the text parts.

>>> print(msg.as_string())
From: aperson@example.com
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="AAA"
<BLANKLINE>
--AAA
Content-Type: application/x-ignore
<BLANKLINE>
Approved: not the password
The above line will be ignored.
<BLANKLINE>
--AAA
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
<BLANKLINE>
An important message.
--AAA--
<BLANKLINE>

If the correct password is in the non-text/plain part, it is ignored.

>>> msg = message_from_string("""\
... From: aperson@example.com
... MIME-Version: 1.0
... Content-Type: multipart/mixed; boundary="AAA"
...
... --AAA
... Content-Type: application/x-ignore
...
... Approved: super secret
... The above line will be ignored.
...
... --AAA
... Content-Type: text/plain
...
... Approved: not the password
... An important message.
... --AAA--
... """)
>>> rule.check(mlist, msg, {})
False

Pseudo-header is still stripped, but only from the text/plain part.

>>> print(msg.as_string())
From: aperson@example.com
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="AAA"
<BLANKLINE>
--AAA
Content-Type: application/x-ignore
<BLANKLINE>
Approved: super secret
The above line will be ignored.
<BLANKLINE>
--AAA
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
<BLANKLINE>
An important message.
--AAA--

Stripping text/html parts¶

Because some mail programs will include both a text/plain part and a text/html alternative, the rule must search the alternatives and strip anything that looks like an Approved: header.

>>> msg = message_from_string("""\
... From: aperson@example.com
... MIME-Version: 1.0
... Content-Type: multipart/mixed; boundary="AAA"
...
... --AAA
... Content-Type: text/html
...
... <html>
... <head></head>
... <body>
... <b>Approved: super secret</b>
... <p>The above line will be ignored.
... </body>
... </html>
...
... --AAA
... Content-Type: text/plain
...
... Approved: super secret
... An important message.
... --AAA--
... """)
>>> rule.check(mlist, msg, {})
True

And the header-like text in the text/html part was stripped.

>>> print(msg.as_string())
From: aperson@example.com
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="AAA"
<BLANKLINE>
--AAA
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/html; charset="us-ascii"
<BLANKLINE>
<html>
<head></head>
<body>
<b></b>
<p>The above line will be ignored.
</body>
</html>
<BLANKLINE>
--AAA
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
<BLANKLINE>
An important message.
--AAA--
<BLANKLINE>

This is true even if the rule does not match (i.e. the incorrect password was given).

>>> msg = message_from_string("""\
... From: aperson@example.com
... MIME-Version: 1.0
... Content-Type: multipart/mixed; boundary="AAA"
...
... --AAA
... Content-Type: text/html
...
... <html>
... <head></head>
... <body>
... <b>Approved: not the password</b>
... <p>The above line will be ignored.
... </body>
... </html>
...
... --AAA
... Content-Type: text/plain
...
... Approved: not the password
... An important message.
... --AAA--
... """)
>>> rule.check(mlist, msg, {})
False

>>> print(msg.as_string())
From: aperson@example.com
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="AAA"

--AAA
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/html; charset="us-ascii"

<html>
<head></head>
<body>
<b></b>
<p>The above line will be ignored.
</body>
</html>

--AAA
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"

An important message.
--AAA--

Emergency¶

When the mailing list has its emergency flag set, all messages posted to the list are held for moderator approval.

>>> mlist = create_list('test@example.com')
>>> rule = config.rules['emergency']
>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: My first post
... Message-ID: <first>
...
... An important message.
... """)

By default, the mailing list does not have its emergency flag set.

>>> mlist.emergency
False
>>> rule.check(mlist, msg, {})
False

The emergency rule matches if the flag is set on the mailing list.

>>> mlist.emergency = True
>>> rule.check(mlist, msg, {})
True

However, if the message metadata has a moderator_approved key set, then even if the mailing list has its emergency flag set, the message still goes through to the membership.

>>> rule.check(mlist, msg, dict(moderator_approved=True))
False

Header matching¶

Mailman can do pattern based header matching during its normal rule processing. There is a set of site-wide default header matches specified in the configuration file under the [antispam] section.

>>> mlist = create_list('test@example.com')

In this section, the variable header_checks contains a list of the headers to check, and the patterns to check them against. By default, this list is empty.

It is also possible to programmatically extend these header checks. Here, we’ll extend the checks with a pattern that matches 4 or more stars.

>>> chain = config.chains['header-match']
>>> chain.extend('x-spam-score', '[*]{4,}')

First, if the message has no X-Spam-Score: header, the message passes through the chain with no matches.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: Not spam
... Message-ID: <ant>
...
... This is a message.
... """)

By looking at the message metadata after chain processing, we can see that none of the rules matched.

>>> from mailman.core.chains import process
>>> msgdata = {}
>>> process(mlist, msg, msgdata, 'header-match')
>>> hits_and_misses(msgdata)
No rules hit
Rule misses:
    x-spam-score: [*]{4,}

The header may exist but does not match the pattern.

>>> msg['X-Spam-Score'] = '***'
>>> msgdata = {}
>>> process(mlist, msg, msgdata, 'header-match')
>>> hits_and_misses(msgdata)
No rules hit
Rule misses:
    x-spam-score: [*]{4,}

The header may exist and match the pattern. By default, when the header matches, it gets held for moderator approval.

>>> from mailman.interfaces.chain import ChainEvent
>>> from mailman.testing.helpers import event_subscribers
>>> def handler(event):
...     if isinstance(event, ChainEvent):
...         print(event.__class__.__name__,
...               event.chain.name, event.msg['message-id'])

>>> del msg['x-spam-score']
>>> msg['X-Spam-Score'] = '*****'
>>> msgdata = {}
>>> with event_subscribers(handler):
...     process(mlist, msg, msgdata, 'header-match')
HoldEvent hold <ant>

>>> hits_and_misses(msgdata)
Rule hits:
    x-spam-score: [*]{4,}
No rules missed

The configuration file can also specify a different final disposition for messages that match their header checks. For example, we may just want to discard such messages.

>>> from mailman.testing.helpers import configuration
>>> msgdata = {}
>>> with event_subscribers(handler):
...     with configuration('antispam', jump_chain='discard'):
...         process(mlist, msg, msgdata, 'header-match')
DiscardEvent discard <ant>

These programmatically added headers can be removed by flushing the chain. Now, nothing with match this message.

>>> chain.flush()
>>> msgdata = {}
>>> process(mlist, msg, msgdata, 'header-match')
>>> hits_and_misses(msgdata)
No rules hit
No rules missed

List-specific header matching¶

Each mailing list can also be configured with a set of header matching regular expression rules. These are used to impose list-specific header filtering with the same semantics as the global [antispam] section.

The list administrator wants to match not on four stars, but on three plus signs, but only for the current mailing list.

>>> mlist.header_matches = [('x-spam-score', '[+]{3,}')]

A message with a spam score of two pluses does not match.

>>> msgdata = {}
>>> del msg['x-spam-score']
>>> msg['X-Spam-Score'] = '++'
>>> process(mlist, msg, msgdata, 'header-match')
>>> hits_and_misses(msgdata)
No rules hit
Rule misses:
    x-spam-score: [+]{3,}

But a message with a spam score of three pluses does match. Because a message with the previous Message-Id is already in the moderation queue, we need to give this message a new Message-Id.

>>> msgdata = {}
>>> del msg['x-spam-score']
>>> msg['X-Spam-Score'] = '+++'
>>> del msg['message-id']
>>> msg['Message-Id'] = '<bee>'
>>> process(mlist, msg, msgdata, 'header-match')
>>> hits_and_misses(msgdata)
Rule hits:
    x-spam-score: [+]{3,}
No rules missed

As does a message with a spam score of four pluses.

>>> msgdata = {}
>>> del msg['x-spam-score']
>>> msg['X-Spam-Score'] = '++++'
>>> del msg['message-id']
>>> msg['Message-Id'] = '<cat>'
>>> process(mlist, msg, msgdata, 'header-match')
>>> hits_and_misses(msgdata)
Rule hits:
    x-spam-score: [+]{3,}
No rules missed

Implicit destination¶

The implicit-dest rule matches when the mailing list’s posting address is not explicitly mentioned in the set of message recipients.

>>> mlist = create_list('_xtest@example.com')
>>> rule = config.rules['implicit-dest']
>>> print(rule.name)
implicit-dest

In order to check for implicit destinations, we need to adapt the mailing list to the appropriate interface.

>>> from mailman.interfaces.mailinglist import IAcceptableAliasSet
>>> alias_set = IAcceptableAliasSet(mlist)

This rule matches messages that have an implicit destination, meaning that the mailing list’s posting address isn’t included in the explicit recipients.

>>> mlist.require_explicit_destination = True
>>> alias_set.clear()

>>> msg = message_from_string("""\
... From: aperson@example.org
... Subject: An implicit message
...
... """)
>>> rule.check(mlist, msg, {})
True

You can disable implicit destination checks for the mailing list.

>>> mlist.require_explicit_destination = False
>>> rule.check(mlist, msg, {})
False

Even with some recipients, if the posting address is not included, the rule will match.

>>> mlist.require_explicit_destination = True
>>> msg['To'] = 'myfriend@example.com'
>>> rule.check(mlist, msg, {})
True

Add the posting address as a recipient and the rule will no longer match.

>>> msg['Cc'] = '_xtest@example.com'
>>> rule.check(mlist, msg, {})
False

Alternatively, if one of the acceptable aliases is in the recipients list, then the rule will not match.

>>> del msg['cc']
>>> rule.check(mlist, msg, {})
True

>>> alias_set.add('myfriend@example.com')
>>> rule.check(mlist, msg, {})
False

A message gated from NNTP will obviously have an implicit destination. Such gated messages will not be held for implicit destination because it’s assumed that Mailman pulled it from the appropriate news group.

>>> rule.check(mlist, msg, dict(from_usenet=True))
False

Additional aliases can be added.

>>> alias_set.add('other@example.com')
>>> del msg['to']
>>> rule.check(mlist, msg, {})
True

>>> msg['To'] = 'other@example.com'
>>> rule.check(mlist, msg, {})
False

Aliases can be removed.

>>> alias_set.remove('other@example.com')
>>> rule.check(mlist, msg, {})
True

Aliases can also be cleared.

>>> msg['Cc'] = 'myfriend@example.com'
>>> rule.check(mlist, msg, {})
False

>>> alias_set.clear()
>>> rule.check(mlist, msg, {})
True

Alias patterns¶

It’s also possible to specify an alias pattern, i.e. a regular expression to match against the recipients. For example, we can say that if there is a recipient in the example.net domain, then the rule does not match.

>>> alias_set.add('^.*@example.net')
>>> rule.check(mlist, msg, {})
True

>>> msg['To'] = 'you@example.net'
>>> rule.check(mlist, msg, {})
False

Bad aliases¶

You cannot add an alias that looks like neither a pattern nor an email address.

>>> alias_set.add('foobar')
Traceback (most recent call last):
...
ValueError: foobar

Posting loops¶

To avoid a posting loop, Mailman has a rule to check for the existence of an RFC 2369 List-Post: header with the value of the list’s posting address.

>>> mlist = create_list('_xtest@example.com')
>>> rule = config.rules['loop']
>>> print(rule.name)
loop

The header could be missing, in which case the rule does not match.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... An important message.
... """)
>>> rule.check(mlist, msg, {})
False

The header could be present, but not match the list’s posting address.

>>> msg['List-Post'] = 'not-this-list@example.com'
>>> rule.check(mlist, msg, {})
False

If the header is present and does match the posting address, the rule matches.

>>> del msg['list-post']
>>> msg['List-Post'] = mlist.posting_address
>>> rule.check(mlist, msg, {})
True

Even if there are multiple List-Post: headers, as long as one with the posting address exists, the rule matches.

>>> msg = message_from_string("""\
... From: aperson@example.com
... List-Post: not-this-list@example.com
... List-Post: _xtest@example.com
... List-Post: foo@example.com
...
... An important message.
... """)
>>> rule.check(mlist, msg, {})
True

Message size¶

The message-size rule matches when the posted message is bigger than a specified maximum. Generally this is used to prevent huge attachments from getting posted to the list. This value is calculated in terms of KB (1024 bytes).

>>> mlist = create_list('_xtest@example.com')
>>> rule = config.rules['max-size']
>>> print(rule.name)
max-size

For example, setting the maximum message size to 1 means that any message bigger than that will match the rule.

>>> mlist.max_message_size = 1 # 1024 bytes
>>> one_line = 'x' * 79
>>> big_body = '\n'.join([one_line] * 15)
>>> msg = message_from_string("""\
... From: aperson@example.com
... To: _xtest@example.com
...
... """ + big_body)
>>> rule.check(mlist, msg, {})
True

Setting the maximum message size to zero means no size check is performed.

>>> mlist.max_message_size = 0
>>> rule.check(mlist, msg, {})
False

Of course, if the maximum size is larger than the message’s size, then it’s still okay.

>>> mlist.max_message_size = msg.original_size/1024.0 + 1
>>> rule.check(mlist, msg, {})
False

Moderation¶

All members and nonmembers have a moderation action. When the action is not defer, the moderation rule flags the message as needing moderation. This might be to automatically accept, discard, reject, or hold the message.

Two separate rules check for member and nonmember moderation. Member moderation happens early in the built-in chain, while nonmember moderation happens later in the chain, after normal moderation checks.

>>> mlist = create_list('test@example.com')

Member moderation¶

>>> member_rule = config.rules['member-moderation']
>>> print(member_rule.name)
member-moderation

Anne, a mailing list member, sends a message to the mailing list. Her postings are not moderated.

>>> from mailman.testing.helpers import subscribe
>>> subscribe(mlist, 'Anne')
<Member: Anne Person <aperson@example.com> on test@example.com
         as MemberRole.member>

>>> member = mlist.members.get_member('aperson@example.com')
>>> print(member.moderation_action)
Action.defer

Because Anne is not moderated, the member moderation rule does not match.

>>> member_msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: A posted message
...
... """)
>>> member_rule.check(mlist, member_msg, {})
False

Once the member’s moderation action is set to something other than defer, the rule matches. Also, the message metadata has a few extra pieces of information for the eventual moderation chain.

>>> from mailman.interfaces.action import Action
>>> member.moderation_action = Action.hold
>>> msgdata = {}
>>> member_rule.check(mlist, member_msg, msgdata)
True
>>> dump_msgdata(msgdata)
moderation_action: hold
moderation_sender: aperson@example.com

Nonmembers¶

Nonmembers are handled in a similar way, although by default, nonmember postings are held for moderator approval.

>>> nonmember_rule = config.rules['nonmember-moderation']
>>> print(nonmember_rule.name)
nonmember-moderation

Bart, who is not a member of the mailing list, sends a message to the list.

>>> from mailman.interfaces.member import MemberRole
>>> subscribe(mlist, 'Bart', MemberRole.nonmember)
<Member: Bart Person <bperson@example.com> on test@example.com
         as MemberRole.nonmember>

>>> nonmember = mlist.nonmembers.get_member('bperson@example.com')
>>> print(nonmember.moderation_action)
Action.hold

When Bart is registered as a nonmember of the list, his moderation action is set to hold by default. Thus the rule matches and the message metadata again carries some useful information.

>>> nonmember_msg = message_from_string("""\
... From: bperson@example.com
... To: test@example.com
... Subject: A posted message
...
... """)
>>> msgdata = {}
>>> nonmember_rule.check(mlist, nonmember_msg, msgdata)
True
>>> dump_msgdata(msgdata)
moderation_action: hold
moderation_sender: bperson@example.com

Of course, the nonmember action can be set to defer the decision, in which case the rule does not match.

>>> nonmember.moderation_action = Action.defer
>>> nonmember_rule.check(mlist, nonmember_msg, {})
False

Unregistered nonmembers¶

The incoming runner ensures that all sender addresses are registered in the system, but it is the moderation rule that subscribes nonmember addresses to the mailing list if they are not already subscribed.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> address = getUtility(IUserManager).create_address(
...     'cperson@example.com')
>>> address
<Address: cperson@example.com [not verified] at ...>

>>> msg = message_from_string("""\
... From: cperson@example.com
... To: test@example.com
... Subject: A posted message
...
... """)

cperson is neither a member, nor a nonmember of the mailing list.

>>> def memberkey(member):
...     return member.mailing_list, member.address.email, member.role.value

>>> dump_list(mlist.members.members, key=memberkey)
<Member: Anne Person <aperson@example.com>
         on test@example.com as MemberRole.member>
>>> dump_list(mlist.nonmembers.members, key=memberkey)
<Member: Bart Person <bperson@example.com>
         on test@example.com as MemberRole.nonmember>

However, when the nonmember moderation rule runs, it adds the cperson as a nonmember of the list. The rule also matches.

>>> msgdata = {}
>>> nonmember_rule.check(mlist, msg, msgdata)
True
>>> dump_msgdata(msgdata)
moderation_action: hold
moderation_sender: cperson@example.com

>>> dump_list(mlist.members.members, key=memberkey)
<Member: Anne Person <aperson@example.com>
         on test@example.com as MemberRole.member>
>>> dump_list(mlist.nonmembers.members, key=memberkey)
<Member: Bart Person <bperson@example.com>
         on test@example.com as MemberRole.nonmember>
<Member: cperson@example.com
         on test@example.com as MemberRole.nonmember>

Cross-membership checks¶

Of course, the member moderation rule does not match for nonmembers…

>>> member_rule.check(mlist, nonmember_msg, {})
False
>>> nonmember_rule.check(mlist, member_msg, {})
False

Newsgroup moderation¶

The news-moderation rule matches all messages posted to mailing lists that gateway to a moderated newsgroup. The reason for this is that such messages must get forwarded on to the newsgroup moderator. From there it will get posted to the newsgroup, and from there, gated to the mailing list. It’s a circuitous route, but it works nonetheless by holding all messages posted directly to the mailing list.

>>> mlist = create_list('_xtest@example.com')
>>> rule = config.rules['news-moderation']
>>> print(rule.name)
news-moderation

Set the list configuration variable to enable newsgroup moderation.

>>> from mailman.interfaces.nntp import NewsgroupModeration
>>> mlist.newsgroup_moderation = NewsgroupModeration.moderated

And now all messages will match the rule.

>>> msg = message_from_string("""\
... From: aperson@example.org
... Subject: An announcement
...
... Great things are happening.
... """)
>>> rule.check(mlist, msg, {})
True

When moderation is turned off, the rule does not match.

>>> mlist.newsgroup_moderation = NewsgroupModeration.none
>>> rule.check(mlist, msg, {})
False

No Subject header¶

This rule matches if the message has no Subject: header, or if the header is the empty string when stripped.

>>> mlist = create_list('_xtest@example.com')
>>> rule = config.rules['no-subject']
>>> print(rule.name)
no-subject

A message with a non-empty subject does not match the rule.

>>> msg = message_from_string("""\
... From: aperson@example.org
... To: _xtest@example.com
... Subject: A posted message
...
... """)
>>> rule.check(mlist, msg, {})
False

Delete the Subject: header and the rule matches.

>>> del msg['subject']
>>> rule.check(mlist, msg, {})
True

Even a Subject: header with only whitespace still matches the rule.

>>> msg['Subject'] = '    '
>>> rule.check(mlist, msg, {})
True

Maximum number of recipients¶

This rule matches when there are more than the maximum allowed number of explicit recipients addressed by the message.

>>> mlist = create_list('_xtest@example.com')
>>> rule = config.rules['max-recipients']
>>> print(rule.name)
max-recipients

In this case, we’ll create a message with five recipients. These include all addresses in the To: and CC: headers.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: _xtest@example.com, bperson@example.com
... Cc: cperson@example.com
... Cc: dperson@example.com (Dan Person)
... To: Elly Q. Person <eperson@example.com>
...
... Hey folks!
... """)

For backward compatibility, the message must have fewer than the maximum number of explicit recipients.

>>> mlist.max_num_recipients = 5
>>> rule.check(mlist, msg, {})
True

>>> mlist.max_num_recipients = 6
>>> rule.check(mlist, msg, {})
False

Zero means any number of recipients are allowed.

>>> mlist.max_num_recipients = 0
>>> rule.check(mlist, msg, {})
False

Rules¶

Rules are applied to each message as part of a rule chain. Individual rules simply return a boolean specifying whether the rule matches or not. Chain links determine what happens when a rule matches.

All rules¶

Rules are maintained in the configuration object as a dictionary mapping rule names to rule objects.

>>> from zope.interface.verify import verifyObject
>>> from mailman.interfaces.rules import IRule
>>> for rule_name in sorted(config.rules):
...     rule = config.rules[rule_name]
...     print(rule_name, verifyObject(IRule, rule))
administrivia True
any True
approved True
emergency True
implicit-dest True
loop True
max-recipients True
max-size True
member-moderation True
news-moderation True
no-subject True
nonmember-moderation True
suspicious-header True
truth True

You can get a rule by name.

>>> rule = config.rules['emergency']
>>> verifyObject(IRule, rule)
True

Rule checks¶

Individual rules can be checked to see if they match, by running the rule’s check() method. This returns a boolean indicating whether the rule was matched or not.

>>> mlist = create_list('_xtest@example.com')
>>> msg = message_from_string("""\
... From: aperson@example.com
...
... An important message.
... """)

For example, the emergency rule just checks to see if the emergency flag is set on the mailing list, and the message has not been pre-approved by the list administrator.

>>> print(rule.name)
emergency
>>> mlist.emergency = False
>>> rule.check(mlist, msg, {})
False
>>> mlist.emergency = True
>>> rule.check(mlist, msg, {})
True
>>> rule.check(mlist, msg, dict(moderator_approved=True))
False

Suspicious headers¶

Suspicious headers are a way for Mailman to hold messages that match a particular regular expression. This mostly historical feature is fairly confusing to users, and the list attribute that controls this is misnamed.

>>> mlist = create_list('_xtest@example.com')
>>> rule = config.rules['suspicious-header']
>>> print(rule.name)
suspicious-header

Set the so-called suspicious header configuration variable.

>>> mlist.bounce_matching_headers = 'From: .*person@(blah.)?example.com'
>>> msg = message_from_string("""\
... From: aperson@example.com
... To: _xtest@example.com
... Subject: An implicit message
...
... """)
>>> rule.check(mlist, msg, {})
True

But if the header doesn’t match the regular expression, the rule won’t match. This one comes from a .org address.

>>> msg = message_from_string("""\
... From: aperson@example.org
... To: _xtest@example.com
... Subject: An implicit message
...
... """)
>>> rule.check(mlist, msg, {})
False

Truth¶

This rule always matches. This makes it useful as a terminus rule for unconditionally jumping to another chain.

>>> rule = config.rules['truth']
>>> rule.check(False, False, False)
True

Acknowledgment headers¶

Messages that flow through the global pipeline get their headers cooked, which basically means that their headers go through several mostly unrelated transformations. Some headers get added, others get changed. Some of these changes depend on mailing list settings and others depend on how the message is getting sent through the system. We’ll take things one-by-one.

>>> mlist = create_list('_xtest@example.com')
>>> mlist.subject_prefix = ''

When the message’s metadata has a noack key set, an X-Ack: no header is added.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... A message of great import.
... """)

>>> from mailman.handlers.cook_headers import process
>>> process(mlist, msg, dict(noack=True))
>>> print(msg.as_string())
From: aperson@example.com
X-Ack: no
...

Any existing X-Ack header in the original message is removed.

>>> msg = message_from_string("""\
... X-Ack: yes
... From: aperson@example.com
...
... A message of great import.
... """)
>>> process(mlist, msg, dict(noack=True))
>>> print(msg.as_string())
From: aperson@example.com
X-Ack: no
...

Message acknowledgment¶

When a user posts a message to a mailing list, and that user has chosen to receive acknowledgments of their postings, Mailman will sent them such an acknowledgment.

>>> mlist = create_list('test@example.com')
>>> mlist.display_name = 'Test'
>>> mlist.preferred_language = 'en'
>>> mlist.send_welcome_message = False
>>> # XXX This will almost certainly change once we've worked out the web
>>> # space layout for mailing lists now.

>>> # Ensure that the virgin queue is empty, since we'll be checking this
>>> # for new auto-response messages.
>>> from mailman.testing.helpers import get_queue_messages
>>> get_queue_messages('virgin')
[]

Subscribe a user to the mailing list.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)

>>> from mailman.interfaces.member import MemberRole
>>> user_1 = user_manager.create_user('aperson@example.com')
>>> address_1 = list(user_1.addresses)[0]
>>> mlist.subscribe(address_1, MemberRole.member)
<Member: aperson@example.com on test@example.com as MemberRole.member>

Non-member posts¶

Non-members can’t get acknowledgments of their posts to the mailing list.

>>> msg = message_from_string("""\
... From: bperson@example.com
...
... """)

>>> handler = config.handlers['acknowledge']
>>> handler.process(mlist, msg, {})
>>> get_queue_messages('virgin')
[]

We can also specify the original sender in the message’s metadata. If that person is also not a member, no acknowledgment will be sent either.

>>> msg = message_from_string("""\
... From: bperson@example.com
...
... """)
>>> handler.process(mlist, msg,
...     dict(original_sender='cperson@example.com'))
>>> get_queue_messages('virgin')
[]

No acknowledgment requested¶

Unless the user has requested acknowledgments, they will not get one.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)
>>> handler.process(mlist, msg, {})
>>> get_queue_messages('virgin')
[]

Similarly if the original sender is specified in the message metadata, and that sender is a member but not one who has requested acknowledgments, none will be sent.

>>> user_2 = user_manager.create_user('dperson@example.com')
>>> address_2 = list(user_2.addresses)[0]
>>> mlist.subscribe(address_2, MemberRole.member)
<Member: dperson@example.com on test@example.com as MemberRole.member>

>>> handler.process(mlist, msg,
...     dict(original_sender='dperson@example.com'))
>>> get_queue_messages('virgin')
[]

Requested acknowledgments¶

If the member requests acknowledgments, Mailman will send them one when they post to the mailing list.

>>> user_1.preferences.acknowledge_posts = True

The receipt will include the original message’s subject in the response body,

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: Something witty and insightful
...
... """)
>>> handler.process(mlist, msg, {})
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> dump_msgdata(messages[0].msgdata)
_parsemsg           : False
listid              : test.example.com
nodecorate          : True
recipients          : {'aperson@example.com'}
reduced_list_headers: True
...
>>> print(messages[0].msg.as_string())
...
MIME-Version: 1.0
...
Subject: Test post acknowledgment
From: test-bounces@example.com
To: aperson@example.com
...
Precedence: bulk
<BLANKLINE>
Your message entitled
<BLANKLINE>
    Something witty and insightful
<BLANKLINE>
was successfully received by the Test mailing list.
<BLANKLINE>
List info page: http://lists.example.com/listinfo/test@example.com
Your preferences: http://example.com/aperson@example.com
<BLANKLINE>

If there is no subject, then the receipt will use a generic message.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)
>>> handler.process(mlist, msg, {})
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> dump_msgdata(messages[0].msgdata)
_parsemsg           : False
listid              : test.example.com
nodecorate          : True
recipients          : {'aperson@example.com'}
reduced_list_headers: True
...
>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
...
Subject: Test post acknowledgment
From: test-bounces@example.com
To: aperson@example.com
...
Precedence: bulk
<BLANKLINE>
Your message entitled
<BLANKLINE>
    (no subject)
<BLANKLINE>
was successfully received by the Test mailing list.
<BLANKLINE>
List info page: http://lists.example.com/listinfo/test@example.com
Your preferences: http://example.com/aperson@example.com
<BLANKLINE>

After delivery¶

After a message is delivered, or more correctly, after it has been processed by the rest of the handlers in the incoming queue pipeline, a couple of bookkeeping pieces of information are updated.

>>> from datetime import timedelta
>>> from mailman.utilities.datetime import now
>>> mlist = create_list('_xtest@example.com')
>>> post_time = now() - timedelta(minutes=10)
>>> mlist.last_post_time = post_time
>>> mlist.post_id = 10

Processing a message with this handler updates the last_post_time and post_id attributes.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... Something interesting.
... """)

>>> handler = config.handlers['after-delivery']
>>> handler.process(mlist, msg, {})
>>> mlist.last_post_time > post_time
True
>>> mlist.post_id
11

Archives¶

Updating the archives with posted messages is handled by a separate queue, which allows for better memory management and prevents blocking the main delivery processes while messages are archived. This also allows external archivers to work in a separate process from the main Mailman delivery processes.

>>> handler = config.handlers['to-archive']
>>> mlist = create_list('_xtest@example.com')
>>> switchboard = config.switchboards['archive']

A helper function.

>>> def clear():
...     for filebase in switchboard.files:
...         msg, msgdata = switchboard.dequeue(filebase)
...         switchboard.finish(filebase)

The purpose of this handler is to make a simple decision as to whether the message should get archived and if so, to drop the message in the archiving queue. Really the most important things are to determine when a message should not get archived.

For example, no digests should ever get archived.

>>> from mailman.interfaces.archiver import ArchivePolicy
>>> mlist.archive_policy = ArchivePolicy.public
>>> msg = message_from_string("""\
... Subject: A sample message
...
... A message of great import.
... """)
>>> handler.process(mlist, msg, dict(isdigest=True))
>>> switchboard.files
[]

If the mailing list is not configured to archive, then even regular deliveries won’t be archived.

>>> mlist.archive_policy = ArchivePolicy.never
>>> handler.process(mlist, msg, {})
>>> switchboard.files
[]

There are two de-facto standards for a message to indicate that it does not want to be archived. We’ve seen both in the wild so both are supported. The X-No-Archive: header can be used to indicate that the message should not be archived. Confusingly, this header’s value is actually ignored.

>>> mlist.archive_policy = ArchivePolicy.public
>>> msg = message_from_string("""\
... Subject: A sample message
... X-No-Archive: YES
...
... A message of great import.
... """)
>>> handler.process(mlist, msg, dict(isdigest=True))
>>> switchboard.files
[]

Even a no value will stop the archiving of the message.

>>> msg = message_from_string("""\
... Subject: A sample message
... X-No-Archive: No
...
... A message of great import.
... """)
>>> handler.process(mlist, msg, dict(isdigest=True))
>>> switchboard.files
[]

Another header that’s been observed is the X-Archive: header. Here, the header’s case folded value must be no in order to prevent archiving.

>>> msg = message_from_string("""\
... Subject: A sample message
... X-Archive: No
...
... A message of great import.
... """)
>>> handler.process(mlist, msg, dict(isdigest=True))
>>> switchboard.files
[]

But if the value is yes, then the message will be archived.

>>> msg = message_from_string("""\
... Subject: A sample message
... X-Archive: Yes
...
... A message of great import.
... """)
>>> handler.process(mlist, msg, {})
>>> len(switchboard.files)
1
>>> filebase = switchboard.files[0]
>>> qmsg, qdata = switchboard.dequeue(filebase)
>>> switchboard.finish(filebase)
>>> print(qmsg.as_string())
Subject: A sample message
X-Archive: Yes
<BLANKLINE>
A message of great import.
<BLANKLINE>
>>> dump_msgdata(qdata)
_parsemsg: False
version  : 3

Without either archiving header, and all other things being the same, the message will get archived.

>>> msg = message_from_string("""\
... Subject: A sample message
...
... A message of great import.
... """)
>>> handler.process(mlist, msg, {})
>>> len(switchboard.files)
1
>>> filebase = switchboard.files[0]
>>> qmsg, qdata = switchboard.dequeue(filebase)
>>> switchboard.finish(filebase)
>>> print(qmsg.as_string())
Subject: A sample message
<BLANKLINE>
A message of great import.
<BLANKLINE>
>>> dump_msgdata(qdata)
_parsemsg: False
version  : 3

Avoid duplicates¶

This handler implements several strategies to reduce the reception of duplicate messages. It does this by removing certain recipients from the list of recipients calculated earlier.

>>> mlist = create_list('_xtest@example.com')

Create some members we’re going to use.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)

>>> address_a = user_manager.create_address('aperson@example.com')
>>> address_b = user_manager.create_address('bperson@example.com')

>>> from mailman.interfaces.member import MemberRole
>>> member_a = mlist.subscribe(address_a, MemberRole.member)
>>> member_b = mlist.subscribe(address_b, MemberRole.member)
>>> # This is the message metadata dictionary as it would be produced by
>>> # the CalcRecips handler.
>>> recips = dict(
...     recipients=['aperson@example.com', 'bperson@example.com'])

Short circuiting¶

The module short-circuits if there are no recipients.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: A message of great import
...
... Something
... """)
>>> msgdata = {}

>>> handler = config.handlers['avoid-duplicates']
>>> handler.process(mlist, msg, msgdata)
>>> msgdata
{}
>>> print(msg.as_string())
From: aperson@example.com
Subject: A message of great import

Something

Suppressing the list copy¶

Members can elect not to receive a list copy of any message on which they are explicitly named as a recipient. This is done by setting their receive_list_copy preference to False. However, if they aren’t mentioned in one of the recipient headers (i.e. To, CC, Resent-To, or Resent-CC), then they will get a list copy.

>>> member_a.preferences.receive_list_copy = False
>>> msg = message_from_string("""\
... From: Claire Person <cperson@example.com>
...
... Something of great import.
... """)
>>> msgdata = recips.copy()
>>> handler.process(mlist, msg, msgdata)
>>> sorted(msgdata['recipients'])
['aperson@example.com', 'bperson@example.com']
>>> print(msg.as_string())
From: Claire Person <cperson@example.com>
<BLANKLINE>
Something of great import.
<BLANKLINE>

If they’re mentioned on the CC line, they won’t get a list copy.

>>> msg = message_from_string("""\
... From: Claire Person <cperson@example.com>
... CC: aperson@example.com
...
... Something of great import.
... """)
>>> msgdata = recips.copy()
>>> handler.process(mlist, msg, msgdata)
>>> sorted(msgdata['recipients'])
['bperson@example.com']
>>> print(msg.as_string())
From: Claire Person <cperson@example.com>
CC: aperson@example.com
<BLANKLINE>
Something of great import.
<BLANKLINE>

But if they’re mentioned on the CC line and have receive_list_copy set to True (the default), then they still get a list copy.

>>> msg = message_from_string("""\
... From: Claire Person <cperson@example.com>
... CC: bperson@example.com
...
... Something of great import.
... """)
>>> msgdata = recips.copy()
>>> handler.process(mlist, msg, msgdata)
>>> sorted(msgdata['recipients'])
['aperson@example.com', 'bperson@example.com']
>>> print(msg.as_string())
From: Claire Person <cperson@example.com>
CC: bperson@example.com
<BLANKLINE>
Something of great import.
<BLANKLINE>

Other headers checked for recipients include the To…

>>> msg = message_from_string("""\
... From: Claire Person <cperson@example.com>
... To: aperson@example.com
...
... Something of great import.
... """)
>>> msgdata = recips.copy()
>>> handler.process(mlist, msg, msgdata)
>>> sorted(msgdata['recipients'])
['bperson@example.com']
>>> print(msg.as_string())
From: Claire Person <cperson@example.com>
To: aperson@example.com
<BLANKLINE>
Something of great import.
<BLANKLINE>

… Resent-To …

>>> msg = message_from_string("""\
... From: Claire Person <cperson@example.com>
... Resent-To: aperson@example.com
...
... Something of great import.
... """)
>>> msgdata = recips.copy()
>>> handler.process(mlist, msg, msgdata)
>>> sorted(msgdata['recipients'])
['bperson@example.com']
>>> print(msg.as_string())
From: Claire Person <cperson@example.com>
Resent-To: aperson@example.com
<BLANKLINE>
Something of great import.
<BLANKLINE>

…and Resent-CC headers.

>>> msg = message_from_string("""\
... From: Claire Person <cperson@example.com>
... Resent-Cc: aperson@example.com
...
... Something of great import.
... """)
>>> msgdata = recips.copy()
>>> handler.process(mlist, msg, msgdata)
>>> sorted(msgdata['recipients'])
['bperson@example.com']
>>> print(msg.as_string())
From: Claire Person <cperson@example.com>
Resent-Cc: aperson@example.com
<BLANKLINE>
Something of great import.
<BLANKLINE>

Cleansing headers¶

All messages posted to a list get their headers cleansed. Some headers are related to additional permissions that can be granted to the message and other headers can be used to fish for membership.

>>> mlist = create_list('_xtest@example.com')

Headers such as Approved, Approve, (as well as their X- variants) and Urgent are used to grant special permissions to individual messages. All may contain a password; the first two headers are used by list administrators to pre-approve a message normal held for approval. The latter header is used to send a regular message to all members, regardless of whether they get digests or not. Because all three headers contain passwords, they must be removed from any posted message.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Approved: foobar
... Approve: barfoo
... X-Approved: bazbar
... X-Approve: barbaz
... Urgent: notreally
... Subject: A message of great import
...
... Blah blah blah
... """)

>>> handler = config.handlers['cleanse']
>>> handler.process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
Subject: A message of great import

Blah blah blah

Other headers can be used by list members to fish the list for membership, so we don’t let them go through. These are a mix of standard headers and custom headers supported by some mail readers. For example, X-PMRC is supported by Pegasus mail. I don’t remember what program uses X-Confirm-Reading-To though (Some Microsoft product perhaps?).

>>> msg = message_from_string("""\
... From: bperson@example.com
... Reply-To: bperson@example.org
... Sender: asystem@example.net
... Return-Receipt-To: another@example.com
... Disposition-Notification-To: athird@example.com
... X-Confirm-Reading-To: afourth@example.com
... X-PMRQC: afifth@example.com
... Subject: a message to you
...
... How are you doing?
... """)
>>> handler.process(mlist, msg, {})
>>> print(msg.as_string())
From: bperson@example.com
Reply-To: bperson@example.org
Sender: asystem@example.net
Subject: a message to you
<BLANKLINE>
How are you doing?
<BLANKLINE>

Anonymous lists¶

Anonymous mailing lists also try to cleanse certain identifying headers from the original posting, so that it is at least a bit more difficult to determine who sent the message. This isn’t perfect though, for example, the body of the messages are never scrubbed (though that might not be a bad idea). The From and Reply-To headers in the posted message are taken from list attributes.

Hotmail apparently sets X-Originating-Email.

>>> mlist.anonymous_list = True
>>> mlist.description = 'A Test Mailing List'
>>> mlist.preferred_language = 'en'
>>> msg = message_from_string("""\
... From: bperson@example.com
... Reply-To: bperson@example.org
... Sender: asystem@example.net
... X-Originating-Email: cperson@example.com
... Subject: a message to you
...
... How are you doing?
... """)
>>> handler.process(mlist, msg, {})
>>> print(msg.as_string())
Subject: a message to you
From: A Test Mailing List <_xtest@example.com>
Reply-To: _xtest@example.com
<BLANKLINE>
How are you doing?
<BLANKLINE>

Cooking headers¶

Messages that flow through the global pipeline get their headers ‘cooked’, which basically means that their headers go through several mostly unrelated transformations. Some headers get added, others get changed. Some of these changes depend on mailing list settings and others depend on how the message is getting sent through the system. We’ll take things one-by-one.

>>> mlist = create_list('test@example.com')
>>> mlist.subject_prefix = ''

Saving the original sender¶

Because the original sender headers may get deleted or changed, this handler will place the sender in the message metadata for safe keeping.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... A message of great import.
... """)
>>> msgdata = {}

>>> from mailman.handlers.cook_headers import process
>>> process(mlist, msg, msgdata)
>>> print(msgdata['original_sender'])
aperson@example.com

But if there was no original sender, then the empty string will be saved.

>>> msg = message_from_string("""\
... Subject: No original sender
...
... A message of great import.
... """)
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> print(msgdata['original_sender'])
<BLANKLINE>

Mailman version header¶

Mailman will also insert an X-Mailman-Version header…

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... A message of great import.
... """)
>>> process(mlist, msg, {})
>>> from mailman.version import VERSION
>>> msg['x-mailman-version'] == VERSION
True

…but only if one doesn’t already exist.

>>> msg = message_from_string("""\
... From: aperson@example.com
... X-Mailman-Version: 3000
...
... A message of great import.
... """)
>>> process(mlist, msg, {})
>>> print(msg['x-mailman-version'])
3000

Precedence header¶

Mailman will insert a Precedence header, which is a de-facto standard for telling automatic reply software (e.g. vacation(1)) not to respond to this message.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... A message of great import.
... """)
>>> process(mlist, msg, {})
>>> print(msg['precedence'])
list

But Mailman will only add that header if the original message doesn’t already have one of them.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Precedence: junk
...
... A message of great import.
... """)
>>> process(mlist, msg, {})
>>> print(msg['precedence'])
junk

Personalization¶

The To field normally contains the list posting address. However when messages are fully personalized, that header will get overwritten with the address of the recipient. The list’s posting address will be added to one of the recipient headers so that users will be able to reply back to the list.

>>> from mailman.interfaces.mailinglist import (
...     Personalization, ReplyToMunging)
>>> mlist.personalize = Personalization.full
>>> mlist.reply_goes_to_list = ReplyToMunging.no_munging
>>> mlist.description = 'My test mailing list'
>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
X-Mailman-Version: ...
Precedence: list
Cc: My test mailing list <test@example.com>
<BLANKLINE>
<BLANKLINE>

Message decoration¶

Message decoration is the process of adding headers and footers to the original message. A handler module takes care of this based on the settings of the mailing list and the type of message being processed.

>>> mlist = create_list('_xtest@example.com')
>>> msg_text = """\
... From: aperson@example.org
...
... Here is a message.
... """
>>> msg = message_from_string(msg_text)

Short circuiting¶

Digest messages get decorated during the digest creation phase so no extra decorations are added for digest messages.

>>> from mailman.handlers.decorate import process
>>> process(mlist, msg, dict(isdigest=True))
>>> print(msg.as_string())
From: aperson@example.org

Here is a message.

>>> process(mlist, msg, dict(nodecorate=True))
>>> print(msg.as_string())
From: aperson@example.org

Here is a message.

Simple decorations¶

Message decorations are specified by URI and can be specialized by the mailing list and language. Internal Mailman decorations can be referenced by using the mailman:// URL scheme. Here we create a simple English header and footer for all mailing lists in our site.

>>> import os, tempfile
>>> template_dir = tempfile.mkdtemp()
>>> site_dir = os.path.join(template_dir, 'site', 'en')
>>> os.makedirs(site_dir)
>>> config.push('templates', """
... [paths.testing]
... template_dir: {0}
... """.format(template_dir))

>>> myheader_path = os.path.join(site_dir, 'myheader.txt')
>>> with open(myheader_path, 'w') as fp:
...     print('header', file=fp)
>>> myfooter_path = os.path.join(site_dir, 'myfooter.txt')
>>> with open(myfooter_path, 'w') as fp:
...     print('footer', file=fp)

Setting these attributes on the mailing list causes it to use these templates. Since these are site-global templates, we can use a shorter path.

>>> mlist.header_uri = 'mailman:///myheader.txt'
>>> mlist.footer_uri = 'mailman:///myfooter.txt'

Text messages that have no declared content type are, by default encoded in ASCII. When the mailing list’s preferred language is en (i.e. English), the character set of the mailing list and of the message will match, allowing Mailman to simply prepend the header and append the footer verbatim.

>>> mlist.preferred_language = 'en'
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.org
...
<BLANKLINE>
header
Here is a message.
footer

Mailman supports a number of interpolation variables, placeholders in the header and footer for information to be filled in with mailing list specific data. An example of such information is the mailing list’s real name (a short descriptive name for the mailing list).

>>> with open(myheader_path, 'w') as fp:
...     print('$display_name header', file=fp)
>>> with open(myfooter_path, 'w') as fp:
...     print('$display_name footer', file=fp)

>>> msg = message_from_string(msg_text)
>>> mlist.display_name = 'XTest'
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.org
...
XTest header
Here is a message.
XTest footer

You can’t just pick any interpolation variable though; if you do, the variable will remain in the header or footer unchanged.

>>> with open(myheader_path, 'w') as fp:
...     print('$dummy header', file=fp)
>>> with open(myfooter_path, 'w') as fp:
...     print('$dummy footer', file=fp)

>>> msg = message_from_string(msg_text)
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.org
...
$dummy header
Here is a message.
$dummy footer

Handling RFC 3676 ‘format=flowed’ parameters¶

RFC 3676 describes a standard by which text/plain messages can marked by generating MUAs for better readability in compatible receiving MUAs. The format parameter on the text/plain Content-Type header gives hints as to how the receiving MUA may flow and delete trailing whitespace for better display in a proportional font.

When Mailman sees text/plain messages with such RFC 3676 parameters, it preserves these parameters when it concatenates headers and footers to the message payload.

>>> with open(myheader_path, 'w') as fp:
...     print('header', file=fp)
>>> with open(myfooter_path, 'w') as fp:
...     print('footer', file=fp)

>>> mlist.preferred_language = 'en'
>>> msg = message_from_string("""\
... From: aperson@example.org
... Content-Type: text/plain; format=flowed; delsp=no
...
... Here is a message\x20
... with soft line breaks.
... """)
>>> process(mlist, msg, {})
>>> # Don't use 'print' here as above because it won't be obvious from the
>>> # output that the soft-line break space at the end of the 'Here is a
>>> # message' line will be retained in the output.
>>> print(msg['content-type'])
text/plain; format="flowed"; delsp="no"; charset="us-ascii"
>>> for line in msg.get_payload().splitlines():
...     print('>{0}<'.format(line))
>header<
>Here is a message <
>with soft line breaks.<
>footer<

Decorating mixed-charset messages¶

When a message has no explicit character set, it is assumed to be ASCII. However, if the mailing list’s preferred language has a different character set, Mailman will still try to concatenate the header and footer, but it will convert the text to utf-8 and base-64 encode the message payload.

# 'ja' = Japanese; charset = 'euc-jp'
>>> mlist.preferred_language = 'ja'

>>> with open(myheader_path, 'w') as fp:
...     print('$description header', file=fp)
>>> with open(myfooter_path, 'w') as fp:
...     print('$description footer', file=fp)
>>> mlist.description = '\u65e5\u672c\u8a9e'

>>> from email.message import Message
>>> msg = Message()
>>> msg.set_payload('Fran\xe7aise', 'iso-8859-1')
>>> print(msg.as_string())
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<BLANKLINE>
Fran=E7aise
>>> process(mlist, msg, {})
>>> print(msg.as_string())
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
<BLANKLINE>
5pel5pys6KqeIGhlYWRlcgpGcmFuw6dhaXNlCuaXpeacrOiqniBmb290ZXIK

Sometimes the message even has an unknown character set. In this case, Mailman has no choice but to decorate the original message with MIME attachments.

>>> mlist.preferred_language = 'en'
>>> with open(myheader_path, 'w') as fp:
...     print('header', file=fp)
>>> with open(myfooter_path, 'w') as fp:
...     print('footer', file=fp)

>>> msg = message_from_string("""\
... From: aperson@example.org
... Content-Type: text/plain; charset=unknown
... Content-Transfer-Encoding: 7bit
...
... Here is a message.
... """)

>>> process(mlist, msg, {})
>>> msg.set_boundary('BOUNDARY')
>>> print(msg.as_string())
From: aperson@example.org
Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

header
--BOUNDARY
Content-Type: text/plain; charset=unknown
Content-Transfer-Encoding: 7bit

Here is a message.

--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

footer
--BOUNDARY--

Decorating multipart messages¶

Multipart messages have to be decorated differently. The header and footer cannot be simply concatenated into the payload because that will break the MIME structure of the message. Instead, the header and footer are attached as separate MIME subparts.

When the outer part is multipart/mixed, the header and footer can have a Content-Disposition of inline so that MUAs can display these headers as if they were simply concatenated.

>>> part_1 = message_from_string("""\
... From: aperson@example.org
...
... Here is the first message.
... """)
>>> part_2 = message_from_string("""\
... From: bperson@example.com
...
... Here is the second message.
... """)
>>> from email.mime.multipart import MIMEMultipart
>>> msg = MIMEMultipart('mixed', boundary='BOUNDARY',
...                     _subparts=(part_1, part_2))
>>> process(mlist, msg, {})
>>> print(msg.as_string())
Content-Type: multipart/mixed; boundary="BOUNDARY"
MIME-Version: 1.0
<BLANKLINE>
--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
<BLANKLINE>
header
--BOUNDARY
From: aperson@example.org
<BLANKLINE>
Here is the first message.
<BLANKLINE>
--BOUNDARY
From: bperson@example.com
<BLANKLINE>
Here is the second message.
<BLANKLINE>
--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
<BLANKLINE>
footer
--BOUNDARY--

Decorating other content types¶

Non-multipart non-text content types will get wrapped in a multipart/mixed so that the header and footer can be added as attachments.

>>> msg = message_from_string("""\
... From: aperson@example.org
... Content-Type: image/x-beautiful
...
... IMAGEDATAIMAGEDATAIMAGEDATA
... """)
>>> process(mlist, msg, {})
>>> msg.set_boundary('BOUNDARY')
>>> print(msg.as_string())
From: aperson@example.org
...
--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
<BLANKLINE>
header
--BOUNDARY
Content-Type: image/x-beautiful
<BLANKLINE>
IMAGEDATAIMAGEDATAIMAGEDATA
<BLANKLINE>
--BOUNDARY
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
<BLANKLINE>
footer
--BOUNDARY--

Digests¶

Digests are a way for a user to receive list traffic in collections instead of as individual messages when immediately posted. There are several forms of digests, although only two are currently supported: MIME digests and RFC 1153 (a.k.a. plain text) digests.

>>> mlist = create_list('xtest@example.com')

This is a helper function used to iterate through all the accumulated digest messages, in the order in which they were posted. This makes it easier to update the tests when we switch to a different mailbox format.

>>> from mailman.testing.helpers import digest_mbox
>>> from itertools import count
>>> from string import Template

>>> def message_factory():
...     for i in count(1):
...         text = Template("""\
... From: aperson@example.com
... To: xtest@example.com
... Subject: Test message $i
...
... Here is message $i
... """).substitute(i=i)
...         yield message_from_string(text)
>>> message_factory = message_factory()

Short circuiting¶

When a message is posted to the mailing list, it is generally added to a mailbox, unless the mailing list does not allow digests.

>>> mlist.digestable = False
>>> msg = next(message_factory)
>>> process = config.handlers['to-digest'].process
>>> process(mlist, msg, {})
>>> sum(1 for msg in digest_mbox(mlist))
0
>>> digest_queue = config.switchboards['digest']
>>> digest_queue.files
[]

…or they may allow digests but the message is already a digest.

>>> mlist.digestable = True
>>> process(mlist, msg, dict(isdigest=True))
>>> sum(1 for msg in digest_mbox(mlist))
0
>>> digest_queue.files
[]

Sending a digest¶

For messages which are not digests, but which are posted to a digesting mailing list, the messages will be stored until they reach a criteria triggering the sending of the digest. If none of those criteria are met, then the message will just sit in the mailbox for a while.

>>> mlist.digest_size_threshold = 10000
>>> process(mlist, msg, {})
>>> digest_queue.files
[]
>>> digest = digest_mbox(mlist)
>>> sum(1 for msg in digest)
1
>>> import os
>>> os.remove(digest._path)

When the size of the digest mailbox reaches the maximum size threshold, a marker message is placed into the digest runner’s queue. The digest is not actually crafted by the handler.

>>> mlist.digest_size_threshold = 1
>>> mlist.volume = 2
>>> mlist.next_digest_number = 10
>>> digest_path = os.path.join(mlist.data_path, 'digest.mmdf')
>>> size = 0
>>> for msg in message_factory:
...     process(mlist, msg, {})
...     # When the digest reaches the proper size, it is renamed.  So we
...     # can break out of this list when the file disappears.
...     if not os.path.exists(digest_path):
...         break

>>> sum(1 for msg in digest_mbox(mlist))
0
>>> len(digest_queue.files)
1

The digest has been moved to a unique file.

>>> from mailman.utilities.mailbox import Mailbox
>>> from mailman.testing.helpers import get_queue_messages
>>> item = get_queue_messages('digest')[0]
>>> for msg in Mailbox(item.msgdata['digest_path']):
...     print(msg['subject'])
Test message 2
Test message 3
Test message 4
Test message 5
Test message 6
Test message 7
Test message 8
Test message 9

Digests are actually crafted and sent by a separate digest runner.

File recipients¶

Mailman can calculate the recipients for a message from a Sendmail-style include file. This file must be called members.txt and it must live in the list’s data directory.

>>> mlist = create_list('_xtest@example.com')

Short circuiting¶

If the message’s metadata already has recipients, this handler immediately returns.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... A message.
... """)
>>> msgdata = {'recipients': 7}

>>> handler = config.handlers['file-recipients']
>>> handler.process(mlist, msg, msgdata)
>>> print(msg.as_string())
From: aperson@example.com

A message.

>>> dump_msgdata(msgdata)
recipients: 7

Existing file¶

If the file exists, it contains a list of addresses, one per line. These addresses are returned as the set of recipients.

>>> import os
>>> file_path = os.path.join(mlist.data_path, 'members.txt')
>>> with open(file_path, 'w', encoding='utf-8') as fp:
...     print('bperson@example.com', file=fp)
...     print('cperson@example.com', file=fp)
...     print('dperson@example.com', file=fp)
...     print('eperson@example.com', file=fp)
...     print('fperson@example.com', file=fp)
...     print('gperson@example.com', file=fp)

>>> msgdata = {}
>>> handler.process(mlist, msg, msgdata)
>>> dump_list(msgdata['recipients'])
bperson@example.com
cperson@example.com
dperson@example.com
eperson@example.com
fperson@example.com
gperson@example.com

However, if the sender of the original message is a member of the list and their address is in the include file, the sender’s address is not included in the recipients list.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> address_1 = getUtility(IUserManager).create_address(
...     'cperson@example.com')

>>> from mailman.interfaces.member import MemberRole
>>> mlist.subscribe(address_1, MemberRole.member)
<Member: cperson@example.com on _xtest@example.com as MemberRole.member>

>>> msg = message_from_string("""\
... From: cperson@example.com
...
... A message.
... """)
>>> msgdata = {}
>>> handler.process(mlist, msg, msgdata)
>>> dump_list(msgdata['recipients'])
bperson@example.com
dperson@example.com
eperson@example.com
fperson@example.com
gperson@example.com

Content filtering¶

Mailman can filter the content of messages posted to a mailing list by stripping MIME subparts, and possibly reorganizing the MIME structure of a message.

>>> mlist = create_list('test@example.com')

Several mailing list options control content filtering. First, the feature must be enabled, then there are two options that control which MIME types get filtered and which get passed. Finally, there is an option to control whether text/html parts will get converted to plain text. Let’s set up some defaults for these variables, then we’ll explain them in more detail below.

>>> mlist.filter_content = True
>>> mlist.filter_types = []
>>> mlist.pass_types = []
>>> mlist.convert_html_to_plaintext = False

Filtering the outer content type¶

A simple filtering setting will just search the content types of the messages parts, discarding all parts with a matching MIME type. If the message’s outer content type matches the filter, the entire message will be discarded. However, if we turn off content filtering altogether, then the handler short-circuits.

>>> from mailman.interfaces.mime import FilterAction

>>> mlist.filter_types = ['image/jpeg']
>>> mlist.filter_action = FilterAction.discard

>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: image/jpeg
... MIME-Version: 1.0
...
... xxxxx
... """)

>>> process = config.handlers['mime-delete'].process
>>> mlist.filter_content = False
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> print(msg.as_string())
From: aperson@example.com
Content-Type: image/jpeg
MIME-Version: 1.0

xxxxx
>>> msgdata
{}

Similarly, no content filtering is performed on digest messages, which are crafted internally by Mailman.

>>> mlist.filter_content = True
>>> msgdata = {'isdigest': True}
>>> process(mlist, msg, msgdata)
>>> print(msg.as_string())
From: aperson@example.com
Content-Type: image/jpeg
MIME-Version: 1.0
<BLANKLINE>
xxxxx
>>> dump_msgdata(msgdata)
isdigest: True

Simple multipart filtering¶

If one of the subparts in a multipart message matches the filter type, then just that subpart will be stripped.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=BOUNDARY
... MIME-Version: 1.0
...
... --BOUNDARY
... Content-Type: image/jpeg
... MIME-Version: 1.0
...
... xxx
...
... --BOUNDARY
... Content-Type: image/gif
... MIME-Version: 1.0
...
... yyy
... --BOUNDARY--
... """)

>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
Content-Type: multipart/mixed; boundary=BOUNDARY
MIME-Version: 1.0
X-Content-Filtered-By: Mailman/MimeDel ...

--BOUNDARY
Content-Type: image/gif
MIME-Version: 1.0

yyy
--BOUNDARY--

Collapsing multipart/alternative messages¶

When content filtering encounters a multipart/alternative part, and the results of filtering leave only one of the subparts, then the multipart/alternative may be collapsed. For example, in the following message, the outer content type is a multipart/mixed. Inside this part is just a single subpart that has a content type of multipart/alternative. This inner multipart has two subparts, a jpeg and a gif.

Content filtering will remove the jpeg part, leaving the multipart/alternative with only a single gif subpart. Because there’s only one subpart left, the MIME structure of the message will be reorganized, removing the inner multipart/alternative so that the outer multipart/mixed has just a single gif subpart.

>>> mlist.collapse_alternatives = True
>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=BOUNDARY
... MIME-Version: 1.0
...
... --BOUNDARY
... Content-Type: multipart/alternative; boundary=BOUND2
... MIME-Version: 1.0
...
... --BOUND2
... Content-Type: image/jpeg
... MIME-Version: 1.0
...
... xxx
...
... --BOUND2
... Content-Type: image/gif
... MIME-Version: 1.0
...
... yyy
... --BOUND2--
...
... --BOUNDARY--
... """)
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
Content-Type: multipart/mixed; boundary=BOUNDARY
MIME-Version: 1.0
X-Content-Filtered-By: Mailman/MimeDel ...
<BLANKLINE>
--BOUNDARY
Content-Type: image/gif
MIME-Version: 1.0
<BLANKLINE>
yyy
--BOUNDARY--
<BLANKLINE>

When the outer part is a multipart/alternative and filtering leaves this outer part with just one subpart, the entire message is converted to the left over part’s content type. In other words, the left over inner part is promoted to being the outer part.

>>> mlist.filter_types = ['image/jpeg', 'text/html']
>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/alternative; boundary=AAA
...
... --AAA
... Content-Type: text/html
...
... <b>This is some html</b>
... --AAA
... Content-Type: text/plain
...
... This is plain text
... --AAA--
... """)

>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
Content-Type: text/plain
X-Content-Filtered-By: Mailman/MimeDel ...

This is plain text

Clean up.

>>> mlist.filter_types = ['image/jpeg']

Conversion to plain text¶

Some mailing lists prohibit HTML email, and in fact, such email can be a phishing or spam vector. However, many mail readers will send HTML email by default because users think it looks pretty. One approach to handling this would be to filter out text/html parts and rely on multipart/alternative collapsing to leave just a plain text part. This works because many mail readers that send HTML email actually send a plain text part in the second subpart of such multipart/alternatives.

While this is a good suggestion for plain text-only mailing lists, often a mail reader will send only a text/html part with no plain text alternative. in this case, the site administer can enable text/html to text/plain conversion by defining a conversion command. A list administrator still needs to enable such conversion for their list though.

>>> mlist.convert_html_to_plaintext = True

By default, Mailman sends the message through lynx, but since this program is not guaranteed to exist, we’ll craft a simple, but stupid script to simulate the conversion process. The script expects a single argument, which is the name of the file containing the message payload to filter.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: text/html
... MIME-Version: 1.0
...
... <html><head></head>
... <body></body></html>
... """)

>>> from mailman.handlers.tests.test_mimedel import dummy_script
>>> with dummy_script():
...     process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
MIME-Version: 1.0
Content-Type: text/plain
X-Content-Filtered-By: Mailman/MimeDel ...

Converted text/html to text/plain
Filename: ...

Discarding empty parts¶

Similarly, if after filtering a multipart section ends up empty, then the entire multipart is discarded. For example, here’s a message where an inner multipart/mixed contains two jpeg subparts. Both jpegs are filtered out, so the entire inner multipart/mixed is discarded.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=AAA
...
... --AAA
... Content-Type: multipart/mixed; boundary=BBB
...
... --BBB
... Content-Type: image/jpeg
...
... xxx
... --BBB
... Content-Type: image/jpeg
...
... yyy
... --BBB---
... --AAA
... Content-Type: multipart/alternative; boundary=CCC
...
... --CCC
... Content-Type: text/html
...
... <h2>This is a header</h2>
...
... --CCC
... Content-Type: text/plain
...
... A different message
... --CCC--
... --AAA
... Content-Type: image/gif
...
... zzz
... --AAA
... Content-Type: image/gif
...
... aaa
... --AAA--
... """)

>>> with dummy_script():
...     process(mlist, msg, {})

>>> print(msg.as_string())
From: aperson@example.com
Content-Type: multipart/mixed; boundary=AAA
X-Content-Filtered-By: Mailman/MimeDel ...

--AAA
MIME-Version: 1.0
Content-Type: text/plain

Converted text/html to text/plain
Filename: ...

--AAA
Content-Type: image/gif

zzz
--AAA
Content-Type: image/gif

aaa
--AAA--

Passing MIME types¶

XXX Describe the pass_mime_types setting and how it interacts with filter_mime_types.

Calculating recipients¶

Every message that makes it through to the list membership gets sent to a set of recipient addresses. These addresses are calculated by one of the handler modules and depends on a host of factors.

>>> mlist = create_list('test@example.com')

Recipients are calculate from the list membership, so first some people subscribe to the mailing list…

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)

>>> address_a = user_manager.create_address('aperson@example.com')
>>> address_b = user_manager.create_address('bperson@example.com')
>>> address_c = user_manager.create_address('cperson@example.com')
>>> address_d = user_manager.create_address('dperson@example.com')
>>> address_e = user_manager.create_address('eperson@example.com')
>>> address_f = user_manager.create_address('fperson@example.com')

…then subscribe these addresses to the mailing list as members…

>>> from mailman.interfaces.member import MemberRole
>>> member_a = mlist.subscribe(address_a, MemberRole.member)
>>> member_b = mlist.subscribe(address_b, MemberRole.member)
>>> member_c = mlist.subscribe(address_c, MemberRole.member)
>>> member_d = mlist.subscribe(address_d, MemberRole.member)
>>> member_e = mlist.subscribe(address_e, MemberRole.member)
>>> member_f = mlist.subscribe(address_f, MemberRole.member)

…then make some of the members digest members.

>>> from mailman.interfaces.member import DeliveryMode
>>> member_d.preferences.delivery_mode = DeliveryMode.plaintext_digests
>>> member_e.preferences.delivery_mode = DeliveryMode.mime_digests
>>> member_f.preferences.delivery_mode = DeliveryMode.summary_digests

Regular delivery recipients¶

Regular delivery recipients are those people who get messages from the list as soon as they are posted. In other words, these folks are not digest members.

>>> msg = message_from_string("""\
... From: Xavier Person <xperson@example.com>
...
... Something of great import.
... """)
>>> msgdata = {}
>>> handler = config.handlers['member-recipients']
>>> handler.process(mlist, msg, msgdata)
>>> dump_list(msgdata['recipients'])
aperson@example.com
bperson@example.com
cperson@example.com

Members can elect not to receive a list copy of their own postings.

>>> member_c.preferences.receive_own_postings = False
>>> msg = message_from_string("""\
... From: Claire Person <cperson@example.com>
...
... Something of great import.
... """)
>>> msgdata = {}
>>> handler.process(mlist, msg, msgdata)
>>> dump_list(msgdata['recipients'])
aperson@example.com
bperson@example.com

Members can also elect not to receive a list copy of any message on which they are explicitly named as a recipient. However, see the avoid duplicates handler for details.

Digest recipients¶

XXX Test various digest deliveries.

Urgent messages¶

XXX Test various urgent deliveries:

test_urgent_moderator()
test_urgent_admin()
test_urgent_reject()

NNTP Gateway¶

Mailman has an NNTP gateway, whereby messages posted to the mailing list can be forwarded onto an NNTP newsgroup.

>>> mlist = create_list('test@example.com')

Gatewaying from the mailing list to the newsgroup happens through a separate nntp queue and happen immediately when the message is posted through to the list. Note that gatewaying from the newsgroup to the list happens via a separate process.

There are several situations which prevent a message from being gatewayed to the newsgroup. The feature could be disabled, as is the default.

>>> mlist.gateway_to_news = False
>>> msg = message_from_string("""\
... Subject: An important message
...
... Something of great import.
... """)

>>> handler = config.handlers['to-usenet']
>>> handler.process(mlist, msg, {})
>>> from mailman.testing.helpers import get_queue_messages
>>> get_queue_messages('nntp')
[]

Even if enabled, messages that came from the newsgroup are never gated back to the newsgroup.

>>> mlist.gateway_to_news = True
>>> handler.process(mlist, msg, dict(fromusenet=True))
>>> get_queue_messages('nntp')
[]

Neither are digests ever gated to the newsgroup.

>>> handler.process(mlist, msg, dict(isdigest=True))
>>> get_queue_messages('nntp')
[]

However, other posted messages get gated to the newsgroup via the nntp queue. The list owner can set the linked newsgroup and the nntp host that its messages are gated to.

>>> mlist.linked_newsgroup = 'comp.lang.thing'
>>> mlist.nntp_host = 'news.example.com'
>>> handler.process(mlist, msg, {})
>>> messages = get_queue_messages('nntp')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
Subject: An important message

Something of great import.


>>> dump_msgdata(messages[0].msgdata)
_parsemsg: False
listid   : test.example.com
version  : 3

List owner recipients¶

When a message is posted to a mailing list’s -owners address, all of the list’s administrators will receive a copy. The administrators are defined as the set of owners and moderators.

>>> mlist_1 = create_list('alpha@example.com')

Anne is the owner of the list and Bart is a moderator of the list.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)
>>> anne_addr = user_manager.create_address('anne@example.com')
>>> bart_addr = user_manager.create_address('bart@example.com')
>>> from mailman.interfaces.member import MemberRole
>>> anne = mlist_1.subscribe(anne_addr, MemberRole.owner)
>>> bart = mlist_1.subscribe(bart_addr, MemberRole.moderator)

The recipients list for the -owners address includes both Anne and Bart.

>>> msg = message_from_string("""\
... From: Xavier Person <xperson@example.com>
... To: alpha@example.com
...
... """)
>>> msgdata = {}
>>> handler = config.handlers['owner-recipients']
>>> handler.process(mlist_1, msg, msgdata)
>>> dump_list(msgdata['recipients'])
anne@example.com
bart@example.com

Anne disables her owner delivery, so she will not receive -owner emails.

>>> from mailman.interfaces.member import DeliveryStatus
>>> anne.preferences.delivery_status = DeliveryStatus.by_user
>>> msgdata = {}
>>> handler.process(mlist_1, msg, msgdata)
>>> dump_list(msgdata['recipients'])
bart@example.com

If Bart also disables his owner delivery, then no one could contact the list’s owners. Since this is unacceptable, the site owner is used as a fallback.

>>> bart.preferences.delivery_status = DeliveryStatus.by_user
>>> msgdata = {}
>>> handler.process(mlist_1, msg, msgdata)
>>> dump_list(msgdata['recipients'])
noreply@example.com

For mailing lists which have no owners at all, the site owner is also used as a fallback.

>>> mlist_2 = create_list('beta@example.com')
>>> print(mlist_2.administrators.member_count)
0
>>> msgdata = {}
>>> handler.process(mlist_2, msg, msgdata)
>>> dump_list(msgdata['recipients'])
noreply@example.com

Reply-to munging¶

Messages that flow through the global pipeline get their headers cooked, which basically means that their headers go through several mostly unrelated transformations. Some headers get added, others get changed. Some of these changes depend on mailing list settings and others depend on how the message is getting sent through the system. We’ll take things one-by-one.

>>> mlist = create_list('_xtest@example.com')

Reply-to munging refers to the behavior where a mailing list can be configured to change or augment an existing Reply-To header in a message posted to the list. Reply-to munging is fairly controversial, with arguments made either for or against munging.

The Mailman developers, and I believe the majority consensus is to do no reply-to munging, under several principles. Primarily, most reply-to munging is requested by people who do not have both a Reply and Reply All button on their mail reader. If you do not munge Reply-To, then these buttons will work properly, but if you munge the header, it is impossible for these buttons to work right, because both will reply to the list. This leads to unfortunate accidents where a private message is accidentally posted to the entire list.

However, Mailman gives list owners the option to do reply-To munging anyway, mostly as a way to shut up the really vocal minority who seem to insist on this mis-feature.

Reply to list¶

A list can be configured to add a Reply-To header pointing back to the mailing list’s posting address. If there’s no Reply-To header in the original message, the list’s posting address simply gets inserted.

>>> from mailman.interfaces.mailinglist import ReplyToMunging
>>> mlist.reply_goes_to_list = ReplyToMunging.point_to_list
>>> mlist.preferred_language = 'en'
>>> mlist.description = ''
>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)

>>> from mailman.handlers.cook_headers import process
>>> process(mlist, msg, {})
>>> len(msg.get_all('reply-to'))
1
>>> print(msg['reply-to'])
_xtest@example.com

It’s also possible to strip any existing Reply-To header first, before adding the list’s posting address.

>>> mlist.first_strip_reply_to = True
>>> msg = message_from_string("""\
... From: aperson@example.com
... Reply-To: bperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> len(msg.get_all('reply-to'))
1
>>> print(msg['reply-to'])
_xtest@example.com

If you don’t first strip the header, then the list’s posting address will just get appended to whatever the original version was.

>>> mlist.first_strip_reply_to = False
>>> msg = message_from_string("""\
... From: aperson@example.com
... Reply-To: bperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> len(msg.get_all('reply-to'))
1
>>> print(msg['reply-to'])
bperson@example.com, _xtest@example.com

Explicit Reply-To¶

The list can also be configured to have an explicit Reply-To header.

>>> mlist.reply_goes_to_list = ReplyToMunging.explicit_header
>>> mlist.reply_to_address = 'my-list@example.com'
>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> len(msg.get_all('reply-to'))
1
>>> print(msg['reply-to'])
my-list@example.com

And as before, it’s possible to either strip any existing Reply-To header…

>>> mlist.first_strip_reply_to = True
>>> msg = message_from_string("""\
... From: aperson@example.com
... Reply-To: bperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> len(msg.get_all('reply-to'))
1
>>> print(msg['reply-to'])
my-list@example.com

…or not.

>>> mlist.first_strip_reply_to = False
>>> msg = message_from_string("""\
... From: aperson@example.com
... Reply-To: bperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> len(msg.get_all('reply-to'))
1
>>> print(msg['reply-to'])
my-list@example.com, bperson@example.com

Automatic response handler¶

Mailman has a autoreply handler that sends automatic responses to messages it receives on its posting address, owner address, or robot address. Automatic responses are subject to various conditions, such as headers in the original message or the amount of time since the last auto-response.

>>> mlist = create_list('_xtest@example.com')
>>> mlist.display_name = 'XTest'

Basic automatic responding¶

Basic automatic responding occurs when the list is set up to respond to either its -owner address, its -request address, or to the posting address, and a message is sent to one of these addresses. A mailing list also has an automatic response grace period which specifies how much time must pass before a second response will be sent, with 0 meaning “there is no grace period”.

>>> from datetime import timedelta
>>> from mailman.interfaces.autorespond import ResponseAction

>>> mlist.autorespond_owner = ResponseAction.respond_and_continue
>>> mlist.autoresponse_grace_period = timedelta()
>>> mlist.autoresponse_owner_text = 'owner autoresponse text'

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: _xtest-owner@example.com
...
... help
... """)

The preceding message to the mailing list’s owner will trigger an automatic response.

>>> from mailman.testing.helpers import get_queue_messages

>>> handler = config.handlers['replybot']
>>> handler.process(mlist, msg, dict(to_owner=True))
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> dump_msgdata(messages[0].msgdata)
_parsemsg           : False
listid              : _xtest.example.com
nodecorate          : True
recipients          : {'aperson@example.com'}
reduced_list_headers: True
version             : 3

>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Subject: Auto-response for your message to the "XTest" mailing list
From: _xtest-bounces@example.com
To: aperson@example.com
X-Mailer: The Mailman Replybot
X-Ack: No
Message-ID: <...>
Date: ...
Precedence: bulk

owner autoresponse text

Short circuiting¶

Several headers in the original message determine whether an automatic response should even be sent. For example, if the message has an X-Ack: No header, no auto-response is sent.

>>> msg = message_from_string("""\
... From: aperson@example.com
... X-Ack: No
...
... help me
... """)

>>> handler.process(mlist, msg, dict(to_owner=True))
>>> get_queue_messages('virgin')
[]

Mailman itself can suppress automatic responses for certain types of internally crafted messages, by setting the noack metadata key.

>>> msg = message_from_string("""\
... From: mailman@example.com
...
... help for you
... """)

>>> handler.process(mlist, msg, dict(noack=True, to_owner=True))
>>> get_queue_messages('virgin')
[]

If there is a Precedence: header with any of the values bulk, junk, or list, then the automatic response is also suppressed.

>>> msg = message_from_string("""\
... From: asystem@example.com
... Precedence: bulk
...
... hey!
... """)

>>> handler.process(mlist, msg, dict(to_owner=True))
>>> get_queue_messages('virgin')
[]

>>> msg.replace_header('precedence', 'junk')
>>> handler.process(mlist, msg, dict(to_owner=True))
>>> get_queue_messages('virgin')
[]

>>> msg.replace_header('precedence', 'list')
>>> handler.process(mlist, msg, dict(to_owner=True))
>>> get_queue_messages('virgin')
[]

Unless the X-Ack: header has a value of yes, in which case, the Precedence header is ignored.

>>> msg['X-Ack'] = 'yes'
>>> handler.process(mlist, msg, dict(to_owner=True))
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> dump_msgdata(messages[0].msgdata)
_parsemsg           : False
listid              : _xtest.example.com
nodecorate          : True
recipients          : {'asystem@example.com'}
reduced_list_headers: True
version             : 3

>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Subject: Auto-response for your message to the "XTest" mailing list
From: _xtest-bounces@example.com
To: asystem@example.com
X-Mailer: The Mailman Replybot
X-Ack: No
Message-ID: <...>
Date: ...
Precedence: bulk

owner autoresponse text

Available auto-responses¶

As shown above, a message sent to the -owner address will get an auto-response with the text set for owner responses. Two other types of email will get auto-responses: those sent to the -request address…

>>> mlist.autorespond_requests = ResponseAction.respond_and_continue
>>> mlist.autoresponse_request_text = 'robot autoresponse text'

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: _xtest-request@example.com
...
... help me
... """)

>>> handler.process(mlist, msg, dict(to_request=True))
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Subject: Auto-response for your message to the "XTest" mailing list
From: _xtest-bounces@example.com
To: aperson@example.com
X-Mailer: The Mailman Replybot
X-Ack: No
Message-ID: <...>
Date: ...
Precedence: bulk

robot autoresponse text

…and those sent to the posting address.

>>> mlist.autorespond_postings = ResponseAction.respond_and_continue
>>> mlist.autoresponse_postings_text = 'postings autoresponse text'

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: _xtest@example.com
...
... help me
... """)

>>> handler.process(mlist, msg, dict(to_list=True))
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Subject: Auto-response for your message to the "XTest" mailing list
From: _xtest-bounces@example.com
To: aperson@example.com
X-Mailer: The Mailman Replybot
X-Ack: No
Message-ID: <...>
Date: ...
Precedence: bulk

postings autoresponse text

Grace periods¶

Automatic responses have a grace period, during which no additional responses will be sent. This is so as not to bombard the sender with responses. The grace period is measured in days.

>>> mlist.autoresponse_grace_period = timedelta(days=10)

When a response is sent to a person via any of the owner, request, or postings addresses, the response date is recorded. The grace period is usually measured in days.

>>> msg = message_from_string("""\
... From: bperson@example.com
... To: _xtest-owner@example.com
...
... help
... """)

This is the first response to bperson, so it gets sent.

>>> handler.process(mlist, msg, dict(to_owner=True))
>>> len(get_queue_messages('virgin'))
1

But with a grace period greater than zero, no subsequent response will be sent right now.

>>> handler.process(mlist, msg, dict(to_owner=True))
>>> len(get_queue_messages('virgin'))
0

Fast forward 9 days and you still don’t get a response.

>>> from mailman.utilities.datetime import factory
>>> factory.fast_forward(days=9)

>>> handler.process(mlist, msg, dict(to_owner=True))
>>> len(get_queue_messages('virgin'))
0

But tomorrow, the sender will get a new auto-response.

>>> factory.fast_forward()
>>> handler.process(mlist, msg, dict(to_owner=True))
>>> len(get_queue_messages('virgin'))
1

Of course, everything works the same way for messages to the request address, even if the sender is the same person…

>>> msg = message_from_string("""\
... From: bperson@example.com
... To: _xtest-request@example.com
...
... help
... """)

>>> handler.process(mlist, msg, dict(to_request=True))
>>> len(get_queue_messages('virgin'))
1

>>> handler.process(mlist, msg, dict(to_request=True))
>>> len(get_queue_messages('virgin'))
0

>>> factory.fast_forward(days=9)
>>> handler.process(mlist, msg, dict(to_request=True))
>>> len(get_queue_messages('virgin'))
0

>>> factory.fast_forward()
>>> handler.process(mlist, msg, dict(to_request=True))
>>> len(get_queue_messages('virgin'))
1

…and for messages to the posting address.

>>> msg = message_from_string("""\
... From: bperson@example.com
... To: _xtest@example.com
...
... help
... """)

>>> handler.process(mlist, msg, dict(to_list=True))
>>> len(get_queue_messages('virgin'))
1

>>> handler.process(mlist, msg, dict(to_list=True))
>>> len(get_queue_messages('virgin'))
0

>>> factory.fast_forward(days=9)
>>> handler.process(mlist, msg, dict(to_list=True))
>>> len(get_queue_messages('virgin'))
0

>>> factory.fast_forward()
>>> handler.process(mlist, msg, dict(to_list=True))
>>> len(get_queue_messages('virgin'))
1

RFC 2919 and 2369 headers¶

RFC 2919 and RFC 2369 define headers for mailing list actions. These headers generally start with the List- prefix.

>>> mlist = create_list('test@example.com')
>>> mlist.preferred_language = 'en'
>>> from mailman.interfaces.archiver import ArchivePolicy
>>> mlist.archive_policy = ArchivePolicy.never

The rfc-2369 handler adds the List- headers. List-Id is always added.

>>> from mailman.handlers.rfc_2369 import process
>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> list_headers(msg, 'list-id')
---start---
list-id: <test.example.com>
---end---

Fewer headers¶

Some people don’t like these headers because their mail readers aren’t good about hiding them. A list owner can turn these headers off.

>>> mlist.include_rfc2369_headers = False
>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> list_headers(msg)
---start---
---end---

Messages which Mailman generates itself, such as user or owner notifications, have a reduced set of List- headers. Specifically, there is no List-Post, List-Archive or Archived-At header. ..

>>> mlist.include_rfc2369_headers = True
>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)

>>> process(mlist, msg, dict(reduced_list_headers=True))
>>> list_headers(msg)
---start---
list-help: <mailto:test-request@example.com?subject=help>
list-id: <test.example.com>
list-subscribe: <http://lists.example.com/listinfo/test@example.com>,
    <mailto:test-join@example.com>
list-unsubscribe: <http://lists.example.com/listinfo/test@example.com>,
    <mailto:test-leave@example.com>
---end---

List-Post header¶

Discussion lists, to which any subscriber can post, also have a List-Post header which contains the mailto: URL used to send messages to the list.

>>> mlist.include_rfc2369_headers = True
>>> mlist.allow_list_posts = True
>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> list_headers(msg)
---start---
list-help: <mailto:test-request@example.com?subject=help>
list-id: <test.example.com>
list-post: <mailto:test@example.com>
list-subscribe: <http://lists.example.com/listinfo/test@example.com>,
    <mailto:test-join@example.com>
list-unsubscribe: <http://lists.example.com/listinfo/test@example.com>,
    <mailto:test-leave@example.com>
---end---

Some mailing lists are announce, or one-way lists, not discussion lists. Because the general membership cannot post to these mailing lists, the list owner can set a flag which adds a special List-Post header value, according to RFC 2369.

>>> mlist.allow_list_posts = False
>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> list_headers(msg)
---start---
list-help: <mailto:test-request@example.com?subject=help>
list-id: <test.example.com>
list-post: NO
list-subscribe: <http://lists.example.com/listinfo/test@example.com>,
    <mailto:test-join@example.com>
list-unsubscribe: <http://lists.example.com/listinfo/test@example.com>,
    <mailto:test-leave@example.com>
---end---

List-Id header¶

If the mailing list has a description, then it is included in the List-Id header.

>>> mlist.allow_list_posts = True
>>> mlist.description = 'My test mailing list'
>>> msg = message_from_string("""\
... From: aperson@example.com
...
... """)
>>> process(mlist, msg, {})
>>> list_headers(msg)
---start---
list-help: <mailto:test-request@example.com?subject=help>
list-id: My test mailing list <test.example.com>
list-post: <mailto:test@example.com>
list-subscribe: <http://lists.example.com/listinfo/test@example.com>,
    <mailto:test-join@example.com>
list-unsubscribe: <http://lists.example.com/listinfo/test@example.com>,
    <mailto:test-leave@example.com>
---end---

Any existing List-Id headers are removed from the original message.

>>> msg = message_from_string("""\
... From: aperson@example.com
... List-ID: <123.456.789>
...
... """)

>>> process(mlist, msg, {})
>>> list_headers(msg, only='list-id')
---start---
list-id: My test mailing list <test.example.com>
---end---

Archive headers¶

When the mailing list is configured to enable archiving, List-Archive headers will be added for each web accessible archiver that is enabled.

RFC 5064 defines the Archived-At header which contains the url to the individual message in the archives. Archivers which don’t support pre-calculation of the archive url cannot add the Archived-At header. However, other archivers can calculate the url, and do add this header.

If the mailing list isn’t being archived, neither the List-Archive nor Archived-At headers will be added.

Subject prefixes¶

Mailing lists can define a subject prefix which gets added to the front of any Subject text. This can be used to quickly identify which mailing list the message was posted to.

>>> mlist = create_list('test@example.com')

The default list style gives the mailing list a default prefix.

>>> print(mlist.subject_prefix)
[Test]

This can be changed to anything, but typically ends with a trailing space.

>>> mlist.subject_prefix = '[XTest] '
>>> process = config.handlers['subject-prefix'].process

No Subject¶

If the original message has no Subject, then a canned one is used.

>>> msg = message_from_string("""\
... From: aperson@example.com
...
... A message of great import.
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'])
[XTest] (no subject)

Inserting a prefix¶

If the original message had a Subject header, then the prefix is inserted at the beginning of the header’s value.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: Something important
...
... A message of great import.
... """)
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> print(msg['subject'])
[XTest] Something important

The original Subject is available in the metadata.

>>> print(msgdata['original_subject'])
Something important

If a Subject header already has a prefix, usually following a Re: marker, another one will not be added but the prefix will be moved to the front of the header text.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: Re: [XTest] Something important
...
... A message of great import.
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'])
[XTest] Re: Something important

If the Subject header has a prefix at the front of the header text, that’s where it will stay.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: [XTest] Re: Something important
...
... A message of great import.
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'])
[XTest] Re: Something important

Sometimes the incoming Subject header has a pathological sequence of Re: like markers. These should all be collapsed up to the first non-Re: marker.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: [XTest] Re: RE : Re: Re: Re: Re: Re: Something important
...
... A message of great import.
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'])
[XTest] Re: Something important

Internationalized headers¶

Internationalization adds some interesting twists to the handling of subject prefixes. Part of what makes this interesting is the encoding of i18n headers using RFC 2047, and lists whose preferred language is in a different character set than the encoded header.

>>> msg = message_from_string("""\
... Subject: =?iso-2022-jp?b?GyRCJWEhPCVrJV4lcxsoQg==?=
...
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'].encode())
[XTest] =?iso-2022-jp?b?GyRCJWEhPCVrJV4lcxsoQg==?=
>>> print(str(msg['subject']))
[XTest] メールマン

Prefix numbers¶

Subject prefixes support a placeholder for the numeric post id. Every time a message is posted to the mailing list, a post id gets incremented. This is a purely sequential integer that increases monotonically. By added a %d placeholder to the subject prefix, this post id can be included in the prefix.

>>> mlist.subject_prefix = '[XTest %d] '
>>> mlist.post_id = 456
>>> msg = message_from_string("""\
... Subject: Something important
...
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'])
[XTest 456] Something important

This works even when the message is a reply, except that in this case, the numeric post id in the generated subject prefix is updated with the new post id.

>>> msg = message_from_string("""\
... Subject: [XTest 123] Re: Something important
...
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'])
[XTest 456] Re: Something important

If the Subject header had old style prefixing, the prefix is moved to the front of the header text.

>>> msg = message_from_string("""\
... Subject: Re: [XTest 123] Something important
...
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'])
[XTest 456] Re: Something important

And of course, the proper thing is done when posting id numbers are included in the subject prefix, and the subject is encoded non-ASCII.

>>> msg = message_from_string("""\
... Subject: =?iso-2022-jp?b?GyRCJWEhPCVrJV4lcxsoQg==?=
...
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'].encode())
[XTest 456] =?iso-2022-jp?b?GyRCJWEhPCVrJV4lcxsoQg==?=
>>> print(msg['subject'])
[XTest 456] メールマン

Even more fun is when the internationalized Subject header already has a prefix, possibly with a different posting number.

>>> msg = message_from_string("""\
... Subject: [XTest 123] Re: =?iso-2022-jp?b?GyRCJWEhPCVrJV4lcxsoQg==?=
...
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'].encode())
[XTest 456] Re: =?iso-2022-jp?b?GyRCJWEhPCVrJV4lcxsoQg==?=
>>> print(msg['subject'])
[XTest 456] Re: メールマン

As before, old style subject prefixes are re-ordered.

>>> msg = message_from_string("""\
... Subject: Re: [XTest 123] =?iso-2022-jp?b?GyRCJWEhPCVrJV4lcxsoQg==?=
...
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'].encode())
[XTest 456] Re:
  =?iso-2022-jp?b?GyRCJWEhPCVrJV4lcxsoQg==?=
>>> print(msg['subject'])
[XTest 456]  Re: メールマン

In this test case, we get an extra space between the prefix and the original subject. It’s because the original is crooked. Note that a Subject starting with ‘n ‘ is generated by some version of Eudora Japanese edition.

>>> mlist.subject_prefix = '[XTest] '
>>> msg = message_from_string("""\
... Subject:
...  Important message
...
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'])
[XTest]  Important message

And again, with an RFC 2047 encoded header.

>>> msg = message_from_string("""\
... Subject:
...  =?iso-2022-jp?b?GyRCJWEhPCVrJV4lcxsoQg==?=
...
... """)
>>> process(mlist, msg, {})
>>> print(msg['subject'].encode())
[XTest] =?iso-2022-jp?b?GyRCJWEhPCVrJV4lcxsoQg==?=
>>> print(msg['subject'])
[XTest]  メールマン

Message tagger¶

Mailman has a topics system which works like this: a mailing list administrator sets up one or more topics, which is essentially a named regular expression. The topic name can be any arbitrary string, and the name serves double duty as the topic tag. Each message that flows the mailing list has its Subject: and Keywords: headers compared against these regular expressions. The message then gets tagged with the topic names of each hit.

>>> mlist = create_list('_xtest@example.com')

Topics must be enabled for Mailman to do any topic matching, even if topics are defined.

>>> mlist.topics = [('bar fight', '.*bar.*', 'catch any bars', False)]
>>> mlist.topics_enabled = False
>>> mlist.topics_bodylines_limit = 0

>>> msg = message_from_string("""\
... Subject: foobar
... Keywords: barbaz
...
... """)
>>> msgdata = {}

>>> from mailman.handlers.tagger import process
>>> process(mlist, msg, msgdata)
>>> print(msg.as_string())
Subject: foobar
Keywords: barbaz


>>> msgdata
{}

However, once topics are enabled, message will be tagged. There are two artifacts of tagging; an X-Topics: header is added with the topic name, and the message metadata gets a key with a list of matching topic names.

>>> mlist.topics_enabled = True
>>> msg = message_from_string("""\
... Subject: foobar
... Keywords: barbaz
...
... """)
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> print(msg.as_string())
Subject: foobar
Keywords: barbaz
X-Topics: bar fight
<BLANKLINE>
<BLANKLINE>
>>> msgdata['topichits']
['bar fight']

Scanning body lines¶

The tagger can also look at a certain number of body lines, but only for Subject: and Keyword: header-like lines. When set to zero, no body lines are scanned.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: nothing
... Keywords: at all
...
... X-Ignore: something else
... Subject: foobar
... Keywords: barbaz
... """)
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> print(msg.as_string())
From: aperson@example.com
Subject: nothing
Keywords: at all
<BLANKLINE>
X-Ignore: something else
Subject: foobar
Keywords: barbaz
<BLANKLINE>
>>> msgdata
{}

But let the tagger scan a few body lines and the matching headers will be found.

>>> mlist.topics_bodylines_limit = 5
>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: nothing
... Keywords: at all
...
... X-Ignore: something else
... Subject: foobar
... Keywords: barbaz
... """)
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> print(msg.as_string())
From: aperson@example.com
Subject: nothing
Keywords: at all
X-Topics: bar fight
<BLANKLINE>
X-Ignore: something else
Subject: foobar
Keywords: barbaz
<BLANKLINE>
>>> msgdata['topichits']
['bar fight']

However, scanning stops at the first body line that doesn’t look like a header.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: nothing
... Keywords: at all
...
... This is not a header
... Subject: foobar
... Keywords: barbaz
... """)
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> print(msg.as_string())
From: aperson@example.com
Subject: nothing
Keywords: at all
<BLANKLINE>
This is not a header
Subject: foobar
Keywords: barbaz
>>> msgdata
{}

When set to a negative number, all body lines will be scanned.

>>> mlist.topics_bodylines_limit = -1
>>> lots_of_headers = '\n'.join(['X-Ignore: zip'] * 100)
>>> msg = message_from_string("""\
... From: aperson@example.com
... Subject: nothing
... Keywords: at all
...
... %s
... Subject: foobar
... Keywords: barbaz
... """ % lots_of_headers)
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> # Rather than print out 100 X-Ignore: headers, let's just prove that
>>> # the X-Topics: header exists, meaning that the tagger did its job.
>>> print(msg['x-topics'])
bar fight
>>> msgdata['topichits']
['bar fight']

Scanning sub-parts¶

The tagger will also scan the body lines of text subparts in a multipart message, using the same rules as if all those body lines lived in a single text payload.

>>> msg = message_from_string("""\
... Subject: Was
... Keywords: Raw
... Content-Type: multipart/alternative; boundary="BOUNDARY"
...
... --BOUNDARY
... From: sabo
... To: obas
...
... Subject: farbaw
... Keywords: barbaz
...
... --BOUNDARY--
... """)
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> print(msg.as_string())
Subject: Was
Keywords: Raw
Content-Type: multipart/alternative; boundary="BOUNDARY"
X-Topics: bar fight
<BLANKLINE>
--BOUNDARY
From: sabo
To: obas
<BLANKLINE>
Subject: farbaw
Keywords: barbaz
<BLANKLINE>
--BOUNDARY--
<BLANKLINE>
>>> msgdata['topichits']
['bar fight']

But the tagger will not descend into non-text parts.

>>> msg = message_from_string("""\
... Subject: Was
... Keywords: Raw
... Content-Type: multipart/alternative; boundary=BOUNDARY
...
... --BOUNDARY
... From: sabo
... To: obas
... Content-Type: message/rfc822
...
... Subject: farbaw
... Keywords: barbaz
...
... --BOUNDARY
... From: sabo
... To: obas
... Content-Type: message/rfc822
...
... Subject: farbaw
... Keywords: barbaz
...
... --BOUNDARY--
... """)
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> print(msg['x-topics'])
None
>>> msgdata
{}

The outgoing handler¶

Mailman’s outgoing queue is used as the wrapper around SMTP delivery to the upstream mail server. The to-outgoing handler does little more than drop the message into the outgoing queue.

>>> mlist = create_list('test@example.com')

Craft a message destined for the outgoing queue. Include some random metadata as if this message had passed through some other handlers.

>>> msg = message_from_string("""\
... Subject: Here is a message
...
... Something of great import.
... """)

>>> msgdata = dict(foo=1, bar=2, verp=True)
>>> handler = config.handlers['to-outgoing']
>>> handler.process(mlist, msg, msgdata)

While the queued message will not be changed, the queued metadata will have an additional key set: the mailing list name.

>>> from mailman.testing.helpers import get_queue_messages
>>> messages = get_queue_messages('out')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
Subject: Here is a message
<BLANKLINE>
Something of great import.
>>> dump_msgdata(messages[0].msgdata)
_parsemsg: False
bar      : 2
foo      : 1
listid   : test.example.com
verp     : True
version  : 3

Addresses¶

The REST API can be used to manage addresses.

There are no addresses yet.

>>> dump_json('http://localhost:9001/3.0/addresses')
http_etag: "..."
start: 0
total_size: 0

When an address is created via the internal API, it is available in the REST API.

>>> from zope.component import getUtility
>>> from mailman.interfaces.usermanager import IUserManager
>>> user_manager = getUtility(IUserManager)
>>> anne = user_manager.create_address('anne@example.com')
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/addresses')
entry 0:
    email: anne@example.com
    http_etag: "..."
    original_email: anne@example.com
    registered_on: 2005-08-01T07:49:23
    self_link: http://localhost:9001/3.0/addresses/anne@example.com
http_etag: "..."
start: 0
total_size: 1

Anne’s address can also be accessed directly.

>>> dump_json('http://localhost:9001/3.0/addresses/anne@example.com')
email: anne@example.com
http_etag: "..."
original_email: anne@example.com
registered_on: 2005-08-01T07:49:23
self_link: http://localhost:9001/3.0/addresses/anne@example.com

Bart registers with a mixed-case address. The canonical URL always includes the lower-case version.

>>> bart = user_manager.create_address('Bart.Person@example.com')
>>> transaction.commit()
>>> dump_json(
...     'http://localhost:9001/3.0/addresses/bart.person@example.com')
email: bart.person@example.com
http_etag: "..."
original_email: Bart.Person@example.com
registered_on: 2005-08-01T07:49:23
self_link: http://localhost:9001/3.0/addresses/bart.person@example.com

But his address record can be accessed with the case-preserved version too.

>>> dump_json(
...     'http://localhost:9001/3.0/addresses/Bart.Person@example.com')
email: bart.person@example.com
http_etag: "..."
original_email: Bart.Person@example.com
registered_on: 2005-08-01T07:49:23
self_link: http://localhost:9001/3.0/addresses/bart.person@example.com

When an address has a real name associated with it, this is also available in the REST API.

>>> cris = user_manager.create_address('cris@example.com', 'Cris Person')
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com')
display_name: Cris Person
email: cris@example.com
http_etag: "..."
original_email: cris@example.com
registered_on: 2005-08-01T07:49:23
self_link: http://localhost:9001/3.0/addresses/cris@example.com

Verifying¶

When the address gets verified, this attribute is available in the REST representation.

>>> from mailman.utilities.datetime import now
>>> anne.verified_on = now()
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/addresses/anne@example.com')
email: anne@example.com
http_etag: "..."
original_email: anne@example.com
registered_on: 2005-08-01T07:49:23
self_link: http://localhost:9001/3.0/addresses/anne@example.com
verified_on: 2005-08-01T07:49:23

Addresses can also be verified through the REST API, by POSTing to the ‘verify’ sub-resource. The POST data is ignored.

>>> dump_json('http://localhost:9001/3.0/addresses/'
...           'cris@example.com/verify', {})
content-length: 0
date: ...
server: ...
status: 204

Now Cris’s address is verified.

>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com')
display_name: Cris Person
email: cris@example.com
http_etag: "..."
original_email: cris@example.com
registered_on: 2005-08-01T07:49:23
self_link: http://localhost:9001/3.0/addresses/cris@example.com
verified_on: 2005-08-01T07:49:23

If you should ever need to ‘unverify’ an address, POST to the ‘unverify’ sub-resource. Again, the POST data is ignored.

>>> dump_json('http://localhost:9001/3.0/addresses/'
...           'cris@example.com/unverify', {})
content-length: 0
date: ...
server: ...
status: 204

Now Cris’s address is unverified.

>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com')
display_name: Cris Person
email: cris@example.com
http_etag: "..."
original_email: cris@example.com
registered_on: 2005-08-01T07:49:23
self_link: http://localhost:9001/3.0/addresses/cris@example.com

The user¶

To link an address to a user, a POST request can be sent to the /user sub-resource of the address. If the user does not exist, it will be created.

>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com/user',
...           {'display_name': 'Cris X. Person'})
content-length: 0
date: ...
location: http://localhost:9001/3.0/users/1
server: ...
status: 201

The user is now created and the address is linked to it:

>>> cris.user
<User "Cris X. Person" (1) at 0x...>
>>> cris_user = user_manager.get_user('cris@example.com')
>>> cris_user
<User "Cris X. Person" (1) at 0x...>
>>> cris.user == cris_user
True
>>> [a.email for a in cris_user.addresses]
['cris@example.com']

A link to the user resource is now available as a sub-resource.

>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com')
display_name: Cris Person
email: cris@example.com
http_etag: "..."
original_email: cris@example.com
registered_on: 2005-08-01T07:49:23
self_link: http://localhost:9001/3.0/addresses/cris@example.com
user: http://localhost:9001/3.0/users/1

To prevent automatic user creation from taking place, add the auto_create parameter to the POST request and set it to a false-equivalent value like 0:

>>> dump_json('http://localhost:9001/3.0/addresses/anne@example.com/user',
...           {'display_name': 'Anne User', 'auto_create': 0})
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 403: ...

A request to the /user sub-resource will return the linked user’s representation:

>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com/user')
created_on: 2005-08-01T07:49:23
display_name: Cris X. Person
http_etag: "..."
is_server_owner: False
password: ...
self_link: http://localhost:9001/3.0/users/1
user_id: 1

The address and the user can be unlinked by sending a DELETE request on the /user resource. The user itself is not deleted, only the link.

>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com/user',
...           method='DELETE')
content-length: 0
date: ...
server: ...
status: 204
>>> transaction.abort()
>>> cris.user == None
True
>>> from uuid import UUID
>>> user_manager.get_user_by_id(UUID(int=1))
<User "Cris X. Person" (1) at 0x...>
>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com/user')
Traceback (most recent call last):
...
urllib.error.HTTPError: HTTP Error 404: ...

You can link an existing user to an address by passing the user’s ID in the POST request.

>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com/user',
...           {'user_id': 1})
content-length: 0
date: ...
server: ...
status: 200
>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com/user')
created_on: ...
display_name: Cris X. Person
http_etag: ...
password: ...
self_link: http://localhost:9001/3.0/users/1
user_id: 1

To link an address to a different user, you can either send a DELETE request followed by a POST request, or you can send a PUT request.

>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com/user',
...           {'display_name': 'Cris Q Person'}, method="PUT")
content-length: 0
date: ...
location: http://localhost:9001/3.0/users/2
server: ...
status: 201
>>> dump_json('http://localhost:9001/3.0/addresses/cris@example.com/user')
created_on: ...
display_name: Cris Q Person
http_etag: ...
password: ...
self_link: http://localhost:9001/3.0/users/2
user_id: 2

User addresses¶

Users control addresses. The canonical URLs for these user-controlled addresses live in the /addresses namespace.

>>> dave = user_manager.create_user('dave@example.com', 'Dave Person')
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/users/dave@example.com/addresses')
entry 0:
    display_name: Dave Person
    email: dave@example.com
    http_etag: "..."
    original_email: dave@example.com
    registered_on: 2005-08-01T07:49:23
    self_link: http://localhost:9001/3.0/addresses/dave@example.com
    user: http://localhost:9001/3.0/users/3
http_etag: "..."
start: 0
total_size: 1

>>> dump_json('http://localhost:9001/3.0/addresses/dave@example.com')
display_name: Dave Person
email: dave@example.com
http_etag: "..."
original_email: dave@example.com
registered_on: 2005-08-01T07:49:23
self_link: http://localhost:9001/3.0/addresses/dave@example.com
user: http://localhost:9001/3.0/users/3

A user can be associated with multiple email addresses. You can add new addresses to an existing user.

>>> dump_json(
...     'http://localhost:9001/3.0/users/dave@example.com/addresses', {
...           'email': 'dave.person@example.org'
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/addresses/dave.person@example.org
server: ...
status: 201

When you add the new address, you can give it an optional display name.

>>> dump_json(
...     'http://localhost:9001/3.0/users/dave@example.com/addresses', {
...           'email': 'dp@example.org',
...           'display_name': 'Davie P',
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/addresses/dp@example.org
server: ...
status: 201

The user controls these new addresses.

>>> dump_json('http://localhost:9001/3.0/users/dave@example.com/addresses')
entry 0:
    email: dave.person@example.org
    http_etag: "..."
    original_email: dave.person@example.org
    registered_on: 2005-08-01T07:49:23
    self_link: http://localhost:9001/3.0/addresses/dave.person@example.org
    user: http://localhost:9001/3.0/users/3
entry 1:
    display_name: Dave Person
    email: dave@example.com
    http_etag: "..."
    original_email: dave@example.com
    registered_on: 2005-08-01T07:49:23
    self_link: http://localhost:9001/3.0/addresses/dave@example.com
    user: http://localhost:9001/3.0/users/3
entry 2:
    display_name: Davie P
    email: dp@example.org
    http_etag: "..."
    original_email: dp@example.org
    registered_on: 2005-08-01T07:49:23
    self_link: http://localhost:9001/3.0/addresses/dp@example.org
    user: http://localhost:9001/3.0/users/3
http_etag: "..."
start: 0
total_size: 3

Memberships¶

Addresses can be subscribed to mailing lists. When they are, all the membership records for that address are easily accessible via the REST API.

Elle registers several email addresses.

>>> elle = user_manager.create_user('elle@example.com', 'Elle Person')
>>> subscriber = list(elle.addresses)[0]
>>> elle.register('eperson@example.com')
<Address: eperson@example.com [not verified] at ...>
>>> elle.register('elle.person@example.com')
<Address: elle.person@example.com [not verified] at ...>

Elle subscribes to two mailing lists with one of her addresses.

>>> ant = create_list('ant@example.com')
>>> bee = create_list('bee@example.com')
>>> ant.subscribe(subscriber)
<Member: Elle Person <elle@example.com> on ant@example.com
         as MemberRole.member>
>>> bee.subscribe(subscriber)
<Member: Elle Person <elle@example.com> on bee@example.com
         as MemberRole.member>
>>> transaction.commit()

Elle can get her memberships for each of her email addresses.

>>> dump_json('http://localhost:9001/3.0/addresses/'
...           'elle@example.com/memberships')
entry 0:
    address: http://localhost:9001/3.0/addresses/elle@example.com
    delivery_mode: regular
    email: elle@example.com
    http_etag: "..."
    list_id: ant.example.com
    member_id: 1
    role: member
    self_link: http://localhost:9001/3.0/members/1
    user: http://localhost:9001/3.0/users/4
entry 1:
    address: http://localhost:9001/3.0/addresses/elle@example.com
    delivery_mode: regular
    email: elle@example.com
    http_etag: "..."
    list_id: bee.example.com
    member_id: 2
    role: member
    self_link: http://localhost:9001/3.0/members/2
    user: http://localhost:9001/3.0/users/4
http_etag: "..."
start: 0
total_size: 2

>>> dump_json('http://localhost:9001/3.0/addresses/'
...           'eperson@example.com/memberships')
http_etag: "..."
start: 0
total_size: 0

When Elle subscribes to the bee list again with a different address, this does not show up in the list of memberships for his other address.

>>> subscriber = user_manager.get_address('eperson@example.com')
>>> bee.subscribe(subscriber)
<Member: eperson@example.com on bee@example.com as MemberRole.member>
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/addresses/'
...           'elle@example.com/memberships')
entry 0:
    address: http://localhost:9001/3.0/addresses/elle@example.com
    delivery_mode: regular
    email: elle@example.com
    http_etag: "..."
    list_id: ant.example.com
    member_id: 1
    role: member
    self_link: http://localhost:9001/3.0/members/1
    user: http://localhost:9001/3.0/users/4
entry 1:
    address: http://localhost:9001/3.0/addresses/elle@example.com
    delivery_mode: regular
    email: elle@example.com
    http_etag: "..."
    list_id: bee.example.com
    member_id: 2
    role: member
    self_link: http://localhost:9001/3.0/members/2
    user: http://localhost:9001/3.0/users/4
http_etag: "..."
start: 0
total_size: 2

>>> dump_json('http://localhost:9001/3.0/addresses/'
...           'eperson@example.com/memberships')
entry 0:
    address: http://localhost:9001/3.0/addresses/eperson@example.com
    delivery_mode: regular
    email: eperson@example.com
    http_etag: "..."
    list_id: bee.example.com
    member_id: 3
    role: member
    self_link: http://localhost:9001/3.0/members/3
    user: http://localhost:9001/3.0/users/4
http_etag: "..."
start: 0
total_size: 1

Deleting¶

Addresses can be deleted via the REST API.

>>> fred = user_manager.create_address('fred@example.com', 'Fred Person')
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/addresses/fred@example.com')
display_name: Fred Person
email: fred@example.com
http_etag: "..."
original_email: fred@example.com
registered_on: 2005-08-01T07:49:23
self_link: http://localhost:9001/3.0/addresses/fred@example.com

>>> dump_json('http://localhost:9001/3.0/addresses/fred@example.com',
...     method='DELETE')
content-length: 0
date: ...
server: ...
status: 204
>>> transaction.abort()

>>> print(user_manager.get_address('fred@example.com'))
None

If an address is linked to a user, deleting the address does not delete the user, it just unlinks it.

>>> gwen = user_manager.create_user('gwen@example.com', 'Gwen Person')
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/users/5/addresses')
entry 0:
    display_name: Gwen Person
    email: gwen@example.com
    http_etag: "..."
    original_email: gwen@example.com
    registered_on: 2005-08-01T07:49:23
    self_link: http://localhost:9001/3.0/addresses/gwen@example.com
    user: http://localhost:9001/3.0/users/5
http_etag: "795b0680c57ec2df3dceb68ccce2619fecdc7225"
start: 0
total_size: 1

>>> dump_json('http://localhost:9001/3.0/addresses/gwen@example.com',
...     method='DELETE')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/users/5/addresses')
http_etag: "..."
start: 0
total_size: 0

REST server¶

Mailman exposes a REST HTTP server for administrative control.

The server listens for connections on a configurable host name and port.

It is always protected by HTTP basic authentication using a single global user name and password. The credentials are set in the [webservice] section of the configuration using the admin_user and admin_pass properties.

Because the REST server has full administrative access, it should always be run only on localhost, unless you really know what you’re doing. In addition you should set the user name and password to secure values and distribute them to any REST clients with reasonable precautions.

The Mailman major and minor version numbers are in the URL.

Credentials¶

When the Authorization header contains the proper credentials, the request succeeds.

>>> from httplib2 import Http
>>> headers = {
...     'Content-Type': 'application/x-www-form-urlencode',
...     'Authorization': 'Basic cmVzdGFkbWluOnJlc3RwYXNz',
...     }
>>> url = 'http://localhost:9001/3.0/system/versions'
>>> response, content = Http().request(url, 'GET', None, headers)
>>> print(response.status)
200

Version information¶

System version information can be retrieved from the server, in the form of a JSON encoded response.

>>> dump_json('http://localhost:9001/3.0/system/versions')
http_etag: "..."
mailman_version: GNU Mailman 3.0... (...)
python_version: ...
self_link: http://localhost:9001/3.0/system/versions

Domains¶

Domains are how Mailman interacts with email host names and web host names.

# The test framework starts out with an example domain, so let's delete
# that first.
>>> from mailman.interfaces.domain import IDomainManager
>>> from zope.component import getUtility
>>> domain_manager = getUtility(IDomainManager)

>>> domain_manager.remove('example.com')
<Domain example.com...>
>>> transaction.commit()

The REST API can be queried for the set of known domains, of which there are initially none.

>>> dump_json('http://localhost:9001/3.0/domains')
http_etag: "..."
start: 0
total_size: 0

Once a domain is added, it is accessible through the API.

>>> domain_manager.add(
...     'example.com', 'An example domain', 'http://lists.example.com')
<Domain example.com, An example domain, base_url: http://lists.example.com>
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/domains')
entry 0:
    base_url: http://lists.example.com
    description: An example domain
    http_etag: "..."
    mail_host: example.com
    self_link: http://localhost:9001/3.0/domains/example.com
    url_host: lists.example.com
http_etag: "..."
start: 0
total_size: 1

At the top level, all domains are returned as separate entries.

>>> domain_manager.add(
...     'example.org',
...     base_url='http://mail.example.org')
<Domain example.org, base_url: http://mail.example.org>
>>> domain_manager.add(
...     'lists.example.net',
...     'Porkmasters',
...     'http://example.net')
<Domain lists.example.net, Porkmasters, base_url: http://example.net>
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/domains')
entry 0:
    base_url: http://lists.example.com
    description: An example domain
    http_etag: "..."
    mail_host: example.com
    self_link: http://localhost:9001/3.0/domains/example.com
    url_host: lists.example.com
entry 1:
    base_url: http://mail.example.org
    description: None
    http_etag: "..."
    mail_host: example.org
    self_link: http://localhost:9001/3.0/domains/example.org
    url_host: mail.example.org
entry 2:
    base_url: http://example.net
    description: Porkmasters
    http_etag: "..."
    mail_host: lists.example.net
    self_link: http://localhost:9001/3.0/domains/lists.example.net
    url_host: example.net
http_etag: "..."
start: 0
total_size: 3

Individual domains¶

The information for a single domain is available by following one of the self_links from the above collection.

>>> dump_json('http://localhost:9001/3.0/domains/lists.example.net')
base_url: http://example.net
description: Porkmasters
http_etag: "..."
mail_host: lists.example.net
self_link: http://localhost:9001/3.0/domains/lists.example.net
url_host: example.net

You can also list all the mailing lists for a given domain. At first, the example.com domain does not contain any mailing lists.

>>> dump_json('http://localhost:9001/3.0/domains/example.com/lists')
http_etag: "..."
start: 0
total_size: 0

>>> dump_json('http://localhost:9001/3.0/lists', {
...           'fqdn_listname': 'test-domains@example.com',
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/lists/test-domains.example.com
...

>>> dump_json('http://localhost:9001/3.0/domains/example.com/lists')
entry 0:
    display_name: Test-domains
    fqdn_listname: test-domains@example.com
    http_etag: "..."
    ...
    member_count: 0
    self_link: http://localhost:9001/3.0/lists/test-domains.example.com
    volume: 1
http_etag: "..."
start: 0
total_size: 1

Other domains continue to contain no mailing lists.

>>> dump_json('http://localhost:9001/3.0/domains/lists.example.net/lists')
http_etag: "..."
start: 0
total_size: 0

Creating new domains¶

New domains can be created by posting to the domains url.

>>> dump_json('http://localhost:9001/3.0/domains', {
...           'mail_host': 'lists.example.com',
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/domains/lists.example.com
...

Now the web service knows about our new domain.

>>> dump_json('http://localhost:9001/3.0/domains/lists.example.com')
base_url: http://lists.example.com
description: None
http_etag: "..."
mail_host: lists.example.com
self_link: http://localhost:9001/3.0/domains/lists.example.com
url_host: lists.example.com

And the new domain is in our database.

>>> domain_manager['lists.example.com']
<Domain lists.example.com, base_url: http://lists.example.com>

# Unlock the database.
>>> transaction.abort()

You can also create a new domain with a description, a base url, and a contact address.

>>> dump_json('http://localhost:9001/3.0/domains', {
...           'mail_host': 'my.example.com',
...           'description': 'My new domain',
...           'base_url': 'http://allmy.example.com'
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/domains/my.example.com
...

>>> dump_json('http://localhost:9001/3.0/domains/my.example.com')
base_url: http://allmy.example.com
description: My new domain
http_etag: "..."
mail_host: my.example.com
self_link: http://localhost:9001/3.0/domains/my.example.com
url_host: allmy.example.com

>>> domain_manager['my.example.com']
<Domain my.example.com, My new domain, base_url: http://allmy.example.com>

# Unlock the database.
>>> transaction.abort()

Deleting domains¶

Domains can also be deleted via the API.

>>> dump_json('http://localhost:9001/3.0/domains/lists.example.com',
...           method='DELETE')
content-length: 0
date: ...
server: ...
status: 204

Domain owners¶

Domains can have owners. By posting some addresses to the owners resource, you can add some domain owners. Currently our domain has no owners:

>>> dump_json('http://localhost:9001/3.0/domains/my.example.com/owners')
http_etag: ...
start: 0
total_size: 0

Anne and Bart volunteer to be a domain owners.

>>> dump_json('http://localhost:9001/3.0/domains/my.example.com/owners', (
...     ('owner', 'anne@example.com'), ('owner', 'bart@example.com')
...     ))
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/domains/my.example.com/owners')
entry 0:
    created_on: 2005-08-01T07:49:23
    http_etag: ...
    is_server_owner: False
    self_link: http://localhost:9001/3.0/users/1
    user_id: 1
entry 1:
    created_on: 2005-08-01T07:49:23
    http_etag: ...
    is_server_owner: False
    self_link: http://localhost:9001/3.0/users/2
    user_id: 2
http_etag: ...
start: 0
total_size: 2

We can delete all the domain owners.

>>> dump_json('http://localhost:9001/3.0/domains/my.example.com/owners',
...           method='DELETE')
content-length: 0
date: ...
server: ...
status: 204

Now there are no owners.

>>> dump_json('http://localhost:9001/3.0/domains/my.example.com/owners')
http_etag: ...
start: 0
total_size: 0

New domains can be created with owners.

>>> dump_json('http://localhost:9001/3.0/domains', (
...           ('mail_host', 'your.example.com'),
...           ('owner', 'anne@example.com'),
...           ('owner', 'bart@example.com'),
...           ))
content-length: 0
date: ...
location: http://localhost:9001/3.0/domains/your.example.com
server: ...
status: 201

The new domain has the expected owners.

>>> dump_json('http://localhost:9001/3.0/domains/your.example.com/owners')
entry 0:
    created_on: 2005-08-01T07:49:23
    http_etag: ...
    is_server_owner: False
    self_link: http://localhost:9001/3.0/users/1
    user_id: 1
entry 1:
    created_on: 2005-08-01T07:49:23
    http_etag: ...
    is_server_owner: False
    self_link: http://localhost:9001/3.0/users/2
    user_id: 2
http_etag: ...
start: 0
total_size: 2

REST API helpers¶

There are a number of helpers that make building out the REST API easier.

Resource paths¶

For example, most resources don’t have to worry about where they are rooted. They only need to know where they are relative to the root URI, and this function can return them the full path to the resource.

>>> from mailman.rest.helpers import path_to
>>> print(path_to('system'))
http://localhost:9001/3.0/system

Parameters like the scheme, host, port, and API version number can be set in the configuration file.

>>> config.push('helpers', """
... [webservice]
... hostname: geddy
... port: 2112
... use_https: yes
... api_version: 4.2
... """)
>>> cleanups.append((config.pop, 'helpers'))

>>> print(path_to('system'))
https://geddy:2112/4.2/system

Etags¶

HTTP etags are a way for clients to decide whether their copy of a resource has changed or not. Mailman’s REST API calculates this in a cheap and dirty way. Pass in the dictionary representing the resource and that dictionary gets modified to contain the etag under the http_etag key.

>>> from mailman.rest.helpers import etag
>>> resource = dict(geddy='bass', alex='guitar', neil='drums')
>>> json_data = etag(resource)
>>> print(resource['http_etag'])
"6929ecfbda2282980a4818fb75f82e812077f77a"

For convenience, the etag function also returns the JSON representation of the dictionary after tagging, since that’s almost always what you want.

>>> import json
>>> data = json.loads(json_data)

# This is pretty close to what we want, so it's convenient to use.
>>> dump_msgdata(data)
alex     : guitar
geddy    : bass
http_etag: "6929ecfbda2282980a4818fb75f82e812077f77a"
neil     : drums

POST and PUT unpacking¶

Another helper unpacks POST and PUT request variables, validating and converting their values.

>>> from mailman.rest.validator import Validator
>>> validator = Validator(one=int, two=str, three=bool)

>>> class FakeRequest:
...     params = None
>>> FakeRequest.params = dict(one='1', two='two', three='yes')

On valid input, the validator can be used as a **keyword argument.

>>> def print_request(one, two, three):
...     print(repr(one), repr(two), repr(three))
>>> print_request(**validator(FakeRequest))
1 'two' True

On invalid input, an exception is raised.

>>> FakeRequest.params['one'] = 'hello'
>>> print_request(**validator(FakeRequest))
Traceback (most recent call last):
...
ValueError: Cannot convert parameters: one

On missing input, an exception is raised.

>>> del FakeRequest.params['one']
>>> print_request(**validator(FakeRequest))
Traceback (most recent call last):
...
ValueError: Missing parameters: one

If more than one key is missing, it will be reflected in the error message.

>>> del FakeRequest.params['two']
>>> print_request(**validator(FakeRequest))
Traceback (most recent call last):
...
ValueError: Missing parameters: one, two

Extra keys are also not allowed.

>>> FakeRequest.params = dict(one='1', two='two', three='yes',
...                           four='', five='')
>>> print_request(**validator(FakeRequest))
Traceback (most recent call last):
...
ValueError: Unexpected parameters: five, four

However, if optional keys are missing, it’s okay.

>>> validator = Validator(one=int, two=str, three=bool,
...                       four=int, five=int,
...                       _optional=('four', 'five'))

>>> FakeRequest.params = dict(one='1', two='two', three='yes',
...                           four='4', five='5')
>>> def print_request(one, two, three, four=None, five=None):
...     print(repr(one), repr(two), repr(three), repr(four), repr(five))
>>> print_request(**validator(FakeRequest))
1 'two' True 4 5

>>> del FakeRequest.params['four']
>>> print_request(**validator(FakeRequest))
1 'two' True None 5

>>> del FakeRequest.params['five']
>>> print_request(**validator(FakeRequest))
1 'two' True None None

But if the optional values are present, they must of course also be valid.

>>> FakeRequest.params = dict(one='1', two='two', three='yes',
...                           four='no', five='maybe')
>>> print_request(**validator(FakeRequest))
Traceback (most recent call last):
...
ValueError: Cannot convert parameters: five, four

Arrays¶

Some POST forms include more than one value for a particular key. This is how lists and arrays are modeled. The validator does the right thing with such form data. Specifically, when a key shows up multiple times in the form data, a list is given to the validator.

# We can't use a normal dictionary because we'll have multiple keys, but
# the validator only wants to call .items() on the object.
>>> class MultiDict:
...     def __init__(self, *params): self.values = list(params)
...     def items(self): return iter(self.values)
>>> form_data = MultiDict(
...     ('one', '1'),
...     ('many', '3'),
...     ('many', '4'),
...     ('many', '5'),
...     )

This is a validation function that ensures the value is a list.

>>> def must_be_list(value):
...     if not isinstance(value, list):
...         raise ValueError('not a list')
...     return [int(item) for item in value]

This is a validation function that ensure the value is not a list.

>>> def must_be_scalar(value):
...     if isinstance(value, list):
...         raise ValueError('is a list')
...     return int(value)

And a validator to pull it all together.

>>> validator = Validator(one=must_be_scalar, many=must_be_list)
>>> FakeRequest.params = form_data
>>> values = validator(FakeRequest)
>>> print(values['one'])
1
>>> print(values['many'])
[3, 4, 5]

The list values are guaranteed to be in the same order they show up in the form data.

>>> FakeRequest.params = MultiDict(
...     ('one', '1'),
...     ('many', '3'),
...     ('many', '5'),
...     ('many', '4'),
...     )
>>> values = validator(FakeRequest)
>>> print(values['one'])
1
>>> print(values['many'])
[3, 5, 4]

Mailing list configuration¶

Mailing lists can be configured via the REST API.

>>> mlist = create_list('ant@example.com')
>>> transaction.commit()

Reading a configuration¶

All readable attributes for a list are available on a sub-resource.

>>> dump_json('http://localhost:9001/3.0/lists/ant@example.com/config')
acceptable_aliases: []
admin_immed_notify: True
admin_notify_mchanges: False
administrivia: True
advertised: True
allow_list_posts: True
anonymous_list: False
archive_policy: public
autorespond_owner: none
autorespond_postings: none
autorespond_requests: none
autoresponse_grace_period: 90d
autoresponse_owner_text:
autoresponse_postings_text:
autoresponse_request_text:
bounces_address: ant-bounces@example.com
collapse_alternatives: True
convert_html_to_plaintext: False
created_at: 20...T...
default_member_action: defer
default_nonmember_action: hold
description:
digest_last_sent_at: None
digest_size_threshold: 30.0
display_name: Ant
filter_content: False
first_strip_reply_to: False
fqdn_listname: ant@example.com
http_etag: "..."
include_rfc2369_headers: True
join_address: ant-join@example.com
last_post_at: None
leave_address: ant-leave@example.com
list_name: ant
mail_host: example.com
next_digest_number: 1
no_reply_address: noreply@example.com
owner_address: ant-owner@example.com
post_id: 1
posting_address: ant@example.com
posting_pipeline: default-posting-pipeline
reply_goes_to_list: no_munging
reply_to_address:
request_address: ant-request@example.com
scheme: http
send_welcome_message: True
subject_prefix: [Ant]
subscription_policy: confirm
volume: 1
web_host: lists.example.com
welcome_message_uri: mailman:///welcome.txt

Changing the full configuration¶

Not all of the readable attributes can be set through the web interface. The ones that can, can either be set via PUT or PATCH. PUT changes all the writable attributes in one request.

When using PUT, all writable attributes must be included.

>>> dump_json('http://localhost:9001/3.0/lists/'
...           'ant@example.com/config',
...           dict(
...             acceptable_aliases=['one@example.com', 'two@example.com'],
...             admin_immed_notify=False,
...             admin_notify_mchanges=True,
...             administrivia=False,
...             advertised=False,
...             anonymous_list=True,
...             archive_policy='never',
...             autorespond_owner='respond_and_discard',
...             autorespond_postings='respond_and_continue',
...             autorespond_requests='respond_and_discard',
...             autoresponse_grace_period='45d',
...             autoresponse_owner_text='the owner',
...             autoresponse_postings_text='the mailing list',
...             autoresponse_request_text='the robot',
...             display_name='Fnords',
...             description='This is my mailing list',
...             include_rfc2369_headers=False,
...             allow_list_posts=False,
...             digest_size_threshold=10.5,
...             posting_pipeline='virgin',
...             filter_content=True,
...             first_strip_reply_to=True,
...             convert_html_to_plaintext=True,
...             collapse_alternatives=False,
...             reply_goes_to_list='point_to_list',
...             reply_to_address='bee@example.com',
...             send_welcome_message=False,
...             subject_prefix='[ant]',
...             subscription_policy='moderate',
...             welcome_message_uri='mailman:///welcome.txt',
...             default_member_action='hold',
...             default_nonmember_action='discard',
...             ),
...           'PUT')
content-length: 0
date: ...
server: WSGIServer/...
status: 204

These values are changed permanently.

>>> dump_json('http://localhost:9001/3.0/lists/'
...           'ant@example.com/config')
acceptable_aliases: ['one@example.com', 'two@example.com']
admin_immed_notify: False
admin_notify_mchanges: True
administrivia: False
advertised: False
allow_list_posts: False
anonymous_list: True
archive_policy: never
autorespond_owner: respond_and_discard
autorespond_postings: respond_and_continue
autorespond_requests: respond_and_discard
autoresponse_grace_period: 45d
autoresponse_owner_text: the owner
autoresponse_postings_text: the mailing list
autoresponse_request_text: the robot
...
collapse_alternatives: False
convert_html_to_plaintext: True
...
default_member_action: hold
default_nonmember_action: discard
description: This is my mailing list
...
digest_size_threshold: 10.5
display_name: Fnords
filter_content: True
first_strip_reply_to: True
...
include_rfc2369_headers: False
...
posting_pipeline: virgin
reply_goes_to_list: point_to_list
reply_to_address: bee@example.com
...
send_welcome_message: False
subject_prefix: [ant]
subscription_policy: moderate
...
welcome_message_uri: mailman:///welcome.txt

Changing a partial configuration¶

Using PATCH, you can change just one attribute.

>>> dump_json('http://localhost:9001/3.0/lists/'
...           'ant@example.com/config',
...           dict(display_name='My List'),
...           'PATCH')
content-length: 0
date: ...
server: ...
status: 204

These values are changed permanently.

>>> print(mlist.display_name)
My List

Sub-resources¶

Mailing list configuration variables are actually available as sub-resources on the mailing list. Their values can be retrieved and set through the sub-resource.

Simple resources¶

You can view the current value of the sub-resource.

>>> dump_json('http://localhost:9001/3.0/lists/ant.example.com'
...           '/config/display_name')
display_name: My List
http_etag: ...

The resource can be changed by PUTting to it. Note that the value still requires a dictionary, and that dictionary must have a single key matching the name of the resource.

>>> dump_json('http://localhost:9001/3.0/lists/ant.example.com'
...           '/config/display_name',
...           dict(display_name='Your List'),
...           'PUT')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/lists/ant.example.com'
...           '/config/display_name')
display_name: Your List
http_etag: ...

PATCH works the same way, with the same effect, so you can choose to use either method.

>>> dump_json('http://localhost:9001/3.0/lists/ant.example.com'
...           '/config/display_name',
...           dict(display_name='Their List'),
...           'PATCH')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/lists/ant.example.com'
...           '/config/display_name')
display_name: Their List
http_etag: ...

Acceptable aliases¶

These are recipient aliases that can be used in the To: and CC: headers instead of the posting address. They are often used in forwarded emails. By default, a mailing list has no acceptable aliases.

>>> from mailman.interfaces.mailinglist import IAcceptableAliasSet
>>> IAcceptableAliasSet(mlist).clear()
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/lists/'
...           'ant@example.com/config/acceptable_aliases')
acceptable_aliases: []
http_etag: "..."

We can add a few by PUT-ing them on the sub-resource. The keys in the dictionary are ignored.

>>> dump_json('http://localhost:9001/3.0/lists/'
...           'ant@example.com/config/acceptable_aliases',
...           dict(acceptable_aliases=['foo@example.com',
...                                    'bar@example.net']),
...           'PUT')
content-length: 0
date: ...
server: WSGIServer/...
status: 204

Aliases are returned as a list on the aliases key.

>>> response = call_http(
...     'http://localhost:9001/3.0/lists/'
...     'ant@example.com/config/acceptable_aliases')
>>> for alias in response['acceptable_aliases']:
...     print(alias)
bar@example.net
foo@example.com

The mailing list has its aliases set.

>>> from mailman.interfaces.mailinglist import IAcceptableAliasSet
>>> aliases = IAcceptableAliasSet(mlist)
>>> for alias in sorted(aliases.aliases):
...     print(alias)
bar@example.net
foo@example.com

Mailing lists¶

The REST API can be queried for the set of known mailing lists. There is a top level collection that can return all the mailing lists. There aren’t any yet though.

>>> dump_json('http://localhost:9001/3.0/lists')
http_etag: "..."
start: 0
total_size: 0

Create a mailing list in a domain and it’s accessible via the API.

>>> mlist = create_list('ant@example.com')
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/lists')
entry 0:
    display_name: Ant
    fqdn_listname: ant@example.com
    http_etag: "..."
    list_id: ant.example.com
    list_name: ant
    mail_host: example.com
    member_count: 0
    self_link: http://localhost:9001/3.0/lists/ant.example.com
    volume: 1
http_etag: "..."
start: 0
total_size: 1

You can also query for lists from a particular domain.

>>> dump_json('http://localhost:9001/3.0/domains/example.com/lists')
entry 0:
    display_name: Ant
    fqdn_listname: ant@example.com
    http_etag: "..."
    list_id: ant.example.com
    list_name: ant
    mail_host: example.com
    member_count: 0
    self_link: http://localhost:9001/3.0/lists/ant.example.com
    volume: 1
http_etag: "..."
start: 0
total_size: 1

Paginating over list records¶

Instead of returning all the list records at once, it’s possible to return them in pages by adding the GET parameters count and page to the request URI. Page 1 is the first page and count defines the size of the page.

>>> mlist = create_list('bird@example.com')
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/domains/example.com/lists'
...           '?count=1&page=1')
entry 0:
    display_name: Ant
    fqdn_listname: ant@example.com
    http_etag: "..."
    list_id: ant.example.com
    list_name: ant
    mail_host: example.com
    member_count: 0
    self_link: http://localhost:9001/3.0/lists/ant.example.com
    volume: 1
http_etag: "..."
start: 0
total_size: 2

>>> dump_json('http://localhost:9001/3.0/domains/example.com/lists'
...           '?count=1&page=2')
entry 0:
    display_name: Bird
    fqdn_listname: bird@example.com
    http_etag: "..."
    list_id: bird.example.com
    list_name: bird
    mail_host: example.com
    member_count: 0
    self_link: http://localhost:9001/3.0/lists/bird.example.com
    volume: 1
http_etag: "..."
start: 1
total_size: 2

Creating lists via the API¶

New mailing lists can also be created through the API, by posting to the lists URL.

>>> dump_json('http://localhost:9001/3.0/lists', {
...           'fqdn_listname': 'bee@example.com',
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/lists/bee.example.com
...

The mailing list exists in the database.

>>> from mailman.interfaces.listmanager import IListManager
>>> from zope.component import getUtility
>>> list_manager = getUtility(IListManager)

>>> bee = list_manager.get('bee@example.com')
>>> bee
<mailing list "bee@example.com" at ...>

The mailing list was created using the default style, which allows list posts.

>>> bee.allow_list_posts
True

It is also available in the REST API via the location given in the response.

>>> dump_json('http://localhost:9001/3.0/lists/bee.example.com')
display_name: Bee
fqdn_listname: bee@example.com
http_etag: "..."
list_id: bee.example.com
list_name: bee
mail_host: example.com
member_count: 0
self_link: http://localhost:9001/3.0/lists/bee.example.com
volume: 1

Normally, you access the list via its RFC 2369 list-id as shown above, but for backward compatibility purposes, you can also access it via the list’s posting address, if that has never been changed (since the list-id is immutable, but the posting address is not).

>>> dump_json('http://localhost:9001/3.0/lists/bee@example.com')
display_name: Bee
fqdn_listname: bee@example.com
http_etag: "..."
list_id: bee.example.com
list_name: bee
mail_host: example.com
member_count: 0
self_link: http://localhost:9001/3.0/lists/bee.example.com
volume: 1

Apply a style at list creation time¶

List styles allow you to more easily create mailing lists of a particular type, e.g. discussion lists. We can see which styles are available, and which is the default style.

>>> dump_json('http://localhost:9001/3.0/lists/styles')
default: legacy-default
http_etag: "..."
style_names: ['legacy-announce', 'legacy-default']

When creating a list, if we don’t specify a style to apply, the default style is used. However, we can provide a style name in the POST data to choose a different style.

>>> dump_json('http://localhost:9001/3.0/lists', {
...           'fqdn_listname': 'cat@example.com',
...           'style_name': 'legacy-announce',
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/lists/cat.example.com
...

We can tell that the list was created using the legacy-announce style, because announce lists don’t allow posting by the general public.

>>> cat = list_manager.get('cat@example.com')
>>> cat.allow_list_posts
False

Deleting lists via the API¶

Existing mailing lists can be deleted through the API, by doing an HTTP DELETE on the mailing list URL.

>>> dump_json('http://localhost:9001/3.0/lists/bee.example.com',
...           method='DELETE')
content-length: 0
date: ...
server: ...
status: 204

The mailing list does not exist.

>>> print(list_manager.get('bee@example.com'))
None

For backward compatibility purposes, you can delete a list via its posting address as well.

>>> dump_json('http://localhost:9001/3.0/lists/ant@example.com',
...           method='DELETE')
content-length: 0
date: ...
server: ...
status: 204

The mailing list does not exist.

>>> print(list_manager.get('ant@example.com'))
None

Managing mailing list archivers¶

The Mailman system has some site-wide enabled archivers, and each mailing list can enable or disable these archivers individually. This gives list owners control over where traffic to their list is archived. You can see which archivers are available, and whether they are enabled for this mailing list.

>>> mlist = create_list('dog@example.com')
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/lists/dog@example.com/archivers')
http_etag: "..."
mail-archive: True
mhonarc: True
prototype: True

You can set all the archiver states by putting new state flags on the resource.

>>> dump_json(
...     'http://localhost:9001/3.0/lists/dog@example.com/archivers', {
...         'mail-archive': False,
...         'mhonarc': True,
...         'prototype': False,
...         }, method='PUT')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/lists/dog@example.com/archivers')
http_etag: "..."
mail-archive: False
mhonarc: True
prototype: False

You can change the state of a subset of the list archivers.

>>> dump_json(
...     'http://localhost:9001/3.0/lists/dog@example.com/archivers', {
...         'mhonarc': False,
...         }, method='PATCH')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/lists/dog@example.com/archivers')
http_etag: "..."
mail-archive: False
mhonarc: False
prototype: False

Membership¶

The REST API can be used to subscribe and unsubscribe users to mailing lists. A subscribed user is called a member. There is a top level collection that returns all the members of all known mailing lists.

There are no mailing lists and no members yet.

>>> dump_json('http://localhost:9001/3.0/members')
http_etag: "..."
start: 0
total_size: 0

We create a mailing list, which starts out with no members.

>>> bee = create_list('bee@example.com')
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/members')
http_etag: "..."
start: 0
total_size: 0

Subscribers¶

After Bart subscribes to the mailing list, his subscription is available via the REST interface.

>>> from mailman.interfaces.member import MemberRole
>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)

>>> from mailman.testing.helpers import subscribe
>>> subscribe(bee, 'Bart')
<Member: Bart Person <bperson@example.com> on bee@example.com
         as MemberRole.member>

>>> dump_json('http://localhost:9001/3.0/members')
entry 0:
    address: http://localhost:9001/3.0/addresses/bperson@example.com
    delivery_mode: regular
    email: bperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 1
    role: member
    self_link: http://localhost:9001/3.0/members/1
    user: http://localhost:9001/3.0/users/1
http_etag: "..."
start: 0
total_size: 1

Bart’s specific membership can be accessed directly:

>>> dump_json('http://localhost:9001/3.0/members/1')
address: http://localhost:9001/3.0/addresses/bperson@example.com
delivery_mode: regular
email: bperson@example.com
http_etag: ...
list_id: bee.example.com
member_id: 1
role: member
self_link: http://localhost:9001/3.0/members/1
user: http://localhost:9001/3.0/users/1

When Cris also joins the mailing list, her subscription is also available via the REST interface.

>>> subscribe(bee, 'Cris')
<Member: Cris Person <cperson@example.com> on bee@example.com
         as MemberRole.member>

>>> dump_json('http://localhost:9001/3.0/members')
entry 0:
    address: http://localhost:9001/3.0/addresses/bperson@example.com
    delivery_mode: regular
    email: bperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 1
    role: member
    self_link: http://localhost:9001/3.0/members/1
    user: http://localhost:9001/3.0/users/1
entry 1:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 2
    role: member
    self_link: http://localhost:9001/3.0/members/2
    user: http://localhost:9001/3.0/users/2
http_etag: "..."
start: 0
total_size: 2

The subscribed members are returned in alphabetical order, so when Anna subscribes, she is returned first.

>>> subscribe(bee, 'Anna')
<Member: Anna Person <aperson@example.com> on bee@example.com
         as MemberRole.member>

>>> dump_json('http://localhost:9001/3.0/members')
entry 0:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 3
    role: member
    self_link: http://localhost:9001/3.0/members/3
    user: http://localhost:9001/3.0/users/3
entry 1:
    address: http://localhost:9001/3.0/addresses/bperson@example.com
    delivery_mode: regular
    email: bperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 1
    role: member
    self_link: http://localhost:9001/3.0/members/1
    user: http://localhost:9001/3.0/users/1
entry 2:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 2
    role: member
    self_link: http://localhost:9001/3.0/members/2
    user: http://localhost:9001/3.0/users/2
http_etag: "..."
start: 0
total_size: 3

Subscriptions are also returned alphabetically by mailing list posting address. Anna and Cris subscribe to this new mailing list.

>>> ant = create_list('ant@example.com')
>>> subscribe(ant, 'Anna')
<Member: Anna Person <aperson@example.com> on ant@example.com
         as MemberRole.member>
>>> subscribe(ant, 'Cris')
<Member: Cris Person <cperson@example.com> on ant@example.com
         as MemberRole.member>

User ids are different than member ids.

>>> dump_json('http://localhost:9001/3.0/members')
entry 0:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 4
    role: member
    self_link: http://localhost:9001/3.0/members/4
    user: http://localhost:9001/3.0/users/3
entry 1:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 5
    role: member
    self_link: http://localhost:9001/3.0/members/5
    user: http://localhost:9001/3.0/users/2
entry 2:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 3
    role: member
    self_link: http://localhost:9001/3.0/members/3
    user: http://localhost:9001/3.0/users/3
entry 3:
    address: http://localhost:9001/3.0/addresses/bperson@example.com
    delivery_mode: regular
    email: bperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 1
    role: member
    self_link: http://localhost:9001/3.0/members/1
    user: http://localhost:9001/3.0/users/1
entry 4:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 2
    role: member
    self_link: http://localhost:9001/3.0/members/2
    user: http://localhost:9001/3.0/users/2
http_etag: "..."
start: 0
total_size: 5

We can also get just the members of a single mailing list.

>>> dump_json(
...     'http://localhost:9001/3.0/lists/ant@example.com/roster/member')
entry 0:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 4
    role: member
    self_link: http://localhost:9001/3.0/members/4
    user: http://localhost:9001/3.0/users/3
entry 1:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 5
    role: member
    self_link: http://localhost:9001/3.0/members/5
    user: http://localhost:9001/3.0/users/2
http_etag: ...
start: 0
total_size: 2

Paginating over member records¶

Instead of returning all the member records at once, it’s possible to return them in pages by adding the GET parameters count and page to the request URI. Page 1 is the first page and count defines the size of the page.

>>> dump_json(
...     'http://localhost:9001/3.0/lists/ant@example.com/roster/member'
...     '?count=1&page=1')
entry 0:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 4
    role: member
    self_link: http://localhost:9001/3.0/members/4
    user: http://localhost:9001/3.0/users/3
http_etag: ...
start: 0
total_size: 2

This works with members of a single list as well as with all members.

>>> dump_json(
...     'http://localhost:9001/3.0/members?count=1&page=1')
entry 0:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 4
    role: member
    self_link: http://localhost:9001/3.0/members/4
    user: http://localhost:9001/3.0/users/3
http_etag: ...
start: 0
total_size: 5

Owners and moderators¶

Mailing list owners and moderators also show up in the REST API. Cris becomes an owner of the ant mailing list and Dave becomes a moderator of the bee mailing list.

>>> dump_json('http://localhost:9001/3.0/members', {
...           'list_id': 'ant.example.com',
...           'subscriber': 'dperson@example.com',
...           'role': 'moderator',
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/members/6
server: ...
status: 201

>>> dump_json('http://localhost:9001/3.0/members', {
...           'list_id': 'bee.example.com',
...           'subscriber': 'cperson@example.com',
...           'role': 'owner',
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/members/7
server: ...
status: 201

>>> dump_json('http://localhost:9001/3.0/members')
entry 0:
    address: http://localhost:9001/3.0/addresses/dperson@example.com
    delivery_mode: regular
    email: dperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 6
    role: moderator
    self_link: http://localhost:9001/3.0/members/6
    user: http://localhost:9001/3.0/users/4
entry 1:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 4
    role: member
    self_link: http://localhost:9001/3.0/members/4
    user: http://localhost:9001/3.0/users/3
entry 2:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 5
    role: member
    self_link: http://localhost:9001/3.0/members/5
    user: http://localhost:9001/3.0/users/2
entry 3:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 7
    role: owner
    self_link: http://localhost:9001/3.0/members/7
    user: http://localhost:9001/3.0/users/2
entry 4:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 3
    role: member
    self_link: http://localhost:9001/3.0/members/3
    user: http://localhost:9001/3.0/users/3
entry 5:
    address: http://localhost:9001/3.0/addresses/bperson@example.com
    delivery_mode: regular
    email: bperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 1
    role: member
    self_link: http://localhost:9001/3.0/members/1
    user: http://localhost:9001/3.0/users/1
entry 6:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 2
    role: member
    self_link: http://localhost:9001/3.0/members/2
    user: http://localhost:9001/3.0/users/2
http_etag: "..."
start: 0
total_size: 7

We can access all the owners of a list.

>>> dump_json(
...     'http://localhost:9001/3.0/lists/bee@example.com/roster/owner')
entry 0:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 7
    role: owner
    self_link: http://localhost:9001/3.0/members/7
    user: http://localhost:9001/3.0/users/2
http_etag: ...
start: 0
total_size: 1

Finding members¶

A specific member can always be referenced by their role and address.

>>> dump_json('http://localhost:9001/3.0/lists/'
...           'bee@example.com/owner/cperson@example.com')
address: http://localhost:9001/3.0/addresses/cperson@example.com
delivery_mode: regular
email: cperson@example.com
http_etag: ...
list_id: bee.example.com
member_id: 7
role: owner
self_link: http://localhost:9001/3.0/members/7
user: http://localhost:9001/3.0/users/2

You can find a specific member based on several different criteria. For example, we can search for all the memberships of a particular address.

>>> dump_json('http://localhost:9001/3.0/members/find', {
...           'subscriber': 'aperson@example.com',
...           })
entry 0:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 4
    role: member
    self_link: http://localhost:9001/3.0/members/4
    user: http://localhost:9001/3.0/users/3
entry 1:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 3
    role: member
    self_link: http://localhost:9001/3.0/members/3
    user: http://localhost:9001/3.0/users/3
http_etag: ...
start: 0
total_size: 2

Or, we can find all the memberships for a particular mailing list.

>>> dump_json('http://localhost:9001/3.0/members/find', {
...           'list_id': 'bee.example.com',
...           })
entry 0:
    address: http://localhost:9001/3.0/addresses/aperson@example.com
    delivery_mode: regular
    email: aperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 3
    role: member
    self_link: http://localhost:9001/3.0/members/3
    user: http://localhost:9001/3.0/users/3
entry 1:
    address: http://localhost:9001/3.0/addresses/bperson@example.com
    delivery_mode: regular
    email: bperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 1
    role: member
    self_link: http://localhost:9001/3.0/members/1
    user: http://localhost:9001/3.0/users/1
entry 2:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 2
    role: member
    self_link: http://localhost:9001/3.0/members/2
    user: http://localhost:9001/3.0/users/2
entry 3:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 7
    role: owner
    self_link: http://localhost:9001/3.0/members/7
    user: http://localhost:9001/3.0/users/2
http_etag: "..."
start: 0
total_size: 4

Or, we can find all the memberships for an address on a particular mailing list.

>>> dump_json('http://localhost:9001/3.0/members/find', {
...           'subscriber': 'cperson@example.com',
...           'list_id': 'bee.example.com',
...           })
entry 0:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 2
    role: member
    self_link: http://localhost:9001/3.0/members/2
    user: http://localhost:9001/3.0/users/2
entry 1:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 7
    role: owner
    self_link: http://localhost:9001/3.0/members/7
    user: http://localhost:9001/3.0/users/2
http_etag: ...
start: 0
total_size: 2

Or, we can find all the memberships for an address with a specific role.

>>> dump_json('http://localhost:9001/3.0/members/find', {
...           'subscriber': 'cperson@example.com',
...           'role': 'member',
...           })
entry 0:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 5
    role: member
    self_link: http://localhost:9001/3.0/members/5
    user: http://localhost:9001/3.0/users/2
entry 1:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 2
    role: member
    self_link: http://localhost:9001/3.0/members/2
    user: http://localhost:9001/3.0/users/2
http_etag: ...
start: 0
total_size: 2

Finally, we can search for a specific member given all three criteria.

>>> dump_json('http://localhost:9001/3.0/members/find', {
...           'subscriber': 'cperson@example.com',
...           'list_id': 'bee.example.com',
...           'role': 'member',
...           })
entry 0:
    address: http://localhost:9001/3.0/addresses/cperson@example.com
    delivery_mode: regular
    email: cperson@example.com
    http_etag: ...
    list_id: bee.example.com
    member_id: 2
    role: member
    self_link: http://localhost:9001/3.0/members/2
    user: http://localhost:9001/3.0/users/2
http_etag: ...
start: 0
total_size: 1

Joining a mailing list¶

A user can be subscribed to a mailing list via the REST API, either by a specific address, or more generally by their preferred address. A subscribed user is called a member.

The list owner wants to subscribe Elly to the ant mailing list. Since Elly’s email address is not yet known to Mailman, a user is created for her. By default, get gets a regular delivery.

>>> dump_json('http://localhost:9001/3.0/members', {
...           'list_id': 'ant.example.com',
...           'subscriber': 'eperson@example.com',
...           'display_name': 'Elly Person',
...           'pre_verified': True,
...           'pre_confirmed': True,
...           'pre_approved': True,
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/members/8
server: ...
status: 201

Elly is now a known user, and a member of the mailing list.

>>> elly = user_manager.get_user('eperson@example.com')
>>> elly
<User "Elly Person" (...) at ...>

>>> set(member.list_id for member in elly.memberships.members)
{'ant.example.com'}

>>> dump_json('http://localhost:9001/3.0/members')
entry 0:
...
entry 3:
    address: http://localhost:9001/3.0/addresses/eperson@example.com
    delivery_mode: regular
    email: eperson@example.com
    http_etag: ...
    list_id: ant.example.com
    member_id: 8
    role: member
    self_link: http://localhost:9001/3.0/members/8
    user: http://localhost:9001/3.0/users/5
...

Gwen is a user with a preferred address. She subscribes to the ant mailing list with her preferred address.

>>> from mailman.utilities.datetime import now
>>> gwen = user_manager.create_user('gwen@example.com', 'Gwen Person')
>>> preferred = list(gwen.addresses)[0]
>>> preferred.verified_on = now()
>>> gwen.preferred_address = preferred

# Note that we must extract the user id before we commit the transaction.
# This is because accessing the .user_id attribute will lock the database
# in the testing process, breaking the REST queue process.  Also, the
# user_id is a UUID internally, but an integer (represented as a string)
# is required by the REST API.
>>> user_id = gwen.user_id.int
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/members', {
...     'list_id': 'ant.example.com',
...     'subscriber': user_id,
...     'pre_verified': True,
...     'pre_confirmed': True,
...     'pre_approved': True,
...     })
content-length: 0
date: ...
location: http://localhost:9001/3.0/members/9
server: ...
status: 201

>>> dump_json('http://localhost:9001/3.0/members')
entry 0:
...
entry 4:
    address: http://localhost:9001/3.0/addresses/gwen@example.com
    delivery_mode: regular
    email: gwen@example.com
    http_etag: "..."
    list_id: ant.example.com
    member_id: 9
    role: member
    self_link: http://localhost:9001/3.0/members/9
    user: http://localhost:9001/3.0/users/6
...
total_size: 9

When Gwen changes her preferred address, her subscription automatically tracks the new address.

>>> new_preferred = gwen.register('gwen.person@example.com')
>>> new_preferred.verified_on = now()
>>> gwen.preferred_address = new_preferred
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/members')
entry 0:
...
entry 4:
    address: http://localhost:9001/3.0/addresses/gwen.person@example.com
    delivery_mode: regular
    email: gwen.person@example.com
    http_etag: "..."
    list_id: ant.example.com
    member_id: 9
    role: member
    self_link: http://localhost:9001/3.0/members/9
    user: http://localhost:9001/3.0/users/6
...
total_size: 9

Leaving a mailing list¶

Elly decides she does not want to be a member of the mailing list after all, so she leaves from the mailing list.

# Ensure our previous reads don't keep the database lock.
>>> transaction.abort()
>>> dump_json('http://localhost:9001/3.0/members/8',
...           method='DELETE')
content-length: 0
...
status: 204

Elly is no longer a member of the mailing list.

>>> set(member.mailing_list for member in elly.memberships.members)
set()

Changing delivery address¶

As shown above, Gwen is subscribed to a mailing list with her preferred email address. If she changes her preferred address, this automatically changes the address she will receive deliveries at for all such memberships.

However, when Herb subscribes to a couple of mailing lists with explicit addresses, he must change each subscription explicitly.

Herb controls multiple email addresses. All of these addresses are verified.

>>> herb = user_manager.create_user('herb@example.com', 'Herb Person')
>>> herb_1 = list(herb.addresses)[0]
>>> herb_2 = herb.register('hperson@example.com')
>>> herb_3 = herb.register('herb.person@example.com')
>>> for address in herb.addresses:
...     address.verified_on = now()

Herb subscribes to both the ant and bee mailing lists with one of his addresses.

>>> ant.subscribe(herb_1)
<Member: Herb Person <herb@example.com> on
         ant@example.com as MemberRole.member>
>>> bee.subscribe(herb_1)
<Member: Herb Person <herb@example.com> on
         bee@example.com as MemberRole.member>
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/members')
entry 0:
...
entry 4:
    address: http://localhost:9001/3.0/addresses/herb@example.com
    delivery_mode: regular
    email: herb@example.com
    http_etag: "..."
    list_id: ant.example.com
    member_id: 10
    role: member
    self_link: http://localhost:9001/3.0/members/10
    user: http://localhost:9001/3.0/users/7
...
entry 9:
    address: http://localhost:9001/3.0/addresses/herb@example.com
    delivery_mode: regular
    email: herb@example.com
    http_etag: "..."
    list_id: bee.example.com
    member_id: 11
    role: member
    self_link: http://localhost:9001/3.0/members/11
    user: http://localhost:9001/3.0/users/7
http_etag: "..."
start: 0
total_size: 10

In order to change all of his subscriptions to use a different email address, Herb must iterate through his memberships explicitly.

>>> from mailman.testing.helpers import call_api
>>> content, response = call_api('http://localhost:9001/3.0/addresses/'
...                              'herb@example.com/memberships')
>>> memberships = [entry['self_link'] for entry in content['entries']]
>>> for url in sorted(memberships):
...     print(url)
http://localhost:9001/3.0/members/10
http://localhost:9001/3.0/members/11

For each membership resource, the subscription address is changed by PATCH’ing the address attribute.

>>> dump_json('http://localhost:9001/3.0/members/10', {
...           'address': 'hperson@example.com',
...           }, method='PATCH')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/members/11', {
...           'address': 'hperson@example.com',
...           }, method='PATCH')
content-length: 0
date: ...
server: ...
status: 204

Herb’s memberships with the old address are gone.

>>> dump_json('http://localhost:9001/3.0/addresses/'
...           'herb@example.com/memberships')
http_etag: "..."
start: 0
total_size: 0

Herb’s memberships have been updated with his new email address. Of course, his membership ids have not changed.

>>> dump_json('http://localhost:9001/3.0/addresses/'
...           'hperson@example.com/memberships')
entry 0:
    address: http://localhost:9001/3.0/addresses/hperson@example.com
    delivery_mode: regular
    email: hperson@example.com
    http_etag: "..."
    list_id: ant.example.com
    member_id: 10
    role: member
    self_link: http://localhost:9001/3.0/members/10
    user: http://localhost:9001/3.0/users/7
entry 1:
    address: http://localhost:9001/3.0/addresses/hperson@example.com
    delivery_mode: regular
    email: hperson@example.com
    http_etag: "..."
    list_id: bee.example.com
    member_id: 11
    role: member
    self_link: http://localhost:9001/3.0/members/11
    user: http://localhost:9001/3.0/users/7
http_etag: "..."
start: 0
total_size: 2

Post Moderation¶

Messages which are held for approval can be accepted, rejected, discarded, or deferred by the list moderators.

Viewing the list of held messages¶

Held messages can be moderated through the REST API. A mailing list starts with no held messages.

>>> ant = create_list('ant@example.com')
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/lists/ant@example.com/held')
http_etag: "..."
start: 0
total_size: 0

When a message gets held for moderator approval, it shows up in this list.

>>> msg = message_from_string("""\
... From: anne@example.com
... To: ant@example.com
... Subject: Something
... Message-ID: <alpha>
...
... Something else.
... """)

>>> from mailman.app.moderator import hold_message
>>> request_id = hold_message(ant, msg, {'extra': 7}, 'Because')
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/lists/ant@example.com/held')
entry 0:
    extra: 7
    hold_date: 2005-08-01T07:49:23
    http_etag: "..."
    message_id: <alpha>
    msg: From: anne@example.com
To: ant@example.com
Subject: Something
Message-ID: <alpha>
X-Message-ID-Hash: GCSMSG43GYWWVUMO6F7FBUSSPNXQCJ6M

Something else.

    reason: Because
    request_id: 1
    sender: anne@example.com
    subject: Something
http_etag: "..."
start: 0
total_size: 1

You can get an individual held message by providing the request id for that message. This will include the text of the message.

>>> def url(request_id):
...     return ('http://localhost:9001/3.0/lists/'
...             'ant@example.com/held/{0}'.format(request_id))

>>> dump_json(url(request_id))
extra: 7
hold_date: 2005-08-01T07:49:23
http_etag: "..."
message_id: <alpha>
msg: From: anne@example.com
To: ant@example.com
Subject: Something
Message-ID: <alpha>
X-Message-ID-Hash: GCSMSG43GYWWVUMO6F7FBUSSPNXQCJ6M

Something else.

reason: Because
request_id: 1
sender: anne@example.com
subject: Something

Disposing of held messages¶

Individual messages can be moderated through the API by POSTing back to the held message’s resource. The POST data requires an action of one of the following:

discard - throw the message away.

reject - bounces the message back to the original author.

defer - defer any action on the message (continue to hold it)

accept - accept the message for posting.

Let’s see what happens when the above message is deferred.

>>> dump_json(url(request_id), {
...     'action': 'defer',
...     })
content-length: 0
date: ...
server: ...
status: 204

The message is still in the moderation queue.

>>> dump_json(url(request_id))
extra: 7
hold_date: 2005-08-01T07:49:23
http_etag: "..."
message_id: <alpha>
msg: From: anne@example.com
To: ant@example.com
Subject: Something
Message-ID: <alpha>
X-Message-ID-Hash: GCSMSG43GYWWVUMO6F7FBUSSPNXQCJ6M
<BLANKLINE>
Something else.
<BLANKLINE>
reason: Because
request_id: 1
sender: anne@example.com
subject: Something

The held message can be discarded.

>>> dump_json(url(request_id), {
...     'action': 'discard',
...     })
content-length: 0
date: ...
server: ...
status: 204

Messages can also be accepted via the REST API. Let’s hold a new message for moderation.

>>> del msg['message-id']
>>> msg['Message-ID'] = '<bravo>'
>>> request_id = hold_message(ant, msg)
>>> transaction.commit()

>>> results = call_http(url(request_id))
>>> print(results['message_id'])
<bravo>

>>> dump_json(url(request_id), {
...     'action': 'accept',
...     })
content-length: 0
date: ...
server: ...
status: 204

>>> from mailman.testing.helpers import get_queue_messages
>>> messages = get_queue_messages('pipeline')
>>> len(messages)
1
>>> print(messages[0].msg['message-id'])
<bravo>

Messages can be rejected via the REST API too. These bounce the message back to the original author.

>>> del msg['message-id']
>>> msg['Message-ID'] = '<charlie>'
>>> request_id = hold_message(ant, msg)
>>> transaction.commit()

>>> results = call_http(url(request_id))
>>> print(results['message_id'])
<charlie>

>>> dump_json(url(request_id), {
...     'action': 'reject',
...     })
content-length: 0
date: ...
server: ...
status: 204

>>> from mailman.testing.helpers import get_queue_messages
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg['subject'])
Request to mailing list "Ant" rejected

Preferences¶

Addresses have preferences.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)

>>> anne = user_manager.create_address('anne@example.com')
>>> transaction.commit()

Although to start with, an address has no preferences.

>>> dump_json('http://localhost:9001/3.0/addresses/anne@example.com'
...           '/preferences')
http_etag: "..."
self_link: http://localhost:9001/3.0/addresses/anne@example.com/preferences

Once the address is given some preferences, they are available through the REST API.

>>> anne.preferences.acknowledge_posts = True
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/addresses/anne@example.com'
...           '/preferences')
acknowledge_posts: True
http_etag: "..."
self_link: http://localhost:9001/3.0/addresses/anne@example.com/preferences

Similarly, users have their own set of preferences, which also start out empty.

>>> bart = user_manager.create_user('bart@example.com')
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/addresses/bart@example.com'
...           '/preferences')
http_etag: "..."
self_link: http://localhost:9001/3.0/addresses/bart@example.com/preferences

Setting a preference on the user’s address does not set them on the user.

>>> list(bart.addresses)[0].preferences.acknowledge_posts = True
>>> transaction.commit()

>>> dump_json('http://localhost:9001/3.0/addresses/bart@example.com'
...           '/preferences')
acknowledge_posts: True
http_etag: "..."
self_link: http://localhost:9001/3.0/addresses/bart@example.com/preferences

>>> dump_json('http://localhost:9001/3.0/users/1/preferences')
http_etag: "..."
self_link: http://localhost:9001/3.0/users/1/preferences

Users have their own set of preferences.

>>> bart.preferences.receive_own_postings = False
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/users/1/preferences')
http_etag: "..."
receive_own_postings: False
self_link: http://localhost:9001/3.0/users/1/preferences

Similarly, members have their own separate set of preferences, and just like the above, setting a preference on the member’s address or user does not set the preference on the member.

>>> from mailman.interfaces.member import MemberRole
>>> mlist = create_list('test@example.com')
>>> bart_member = mlist.subscribe(list(bart.addresses)[0])
>>> bart_member.preferences.receive_list_copy = False
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/members/1/preferences')
http_etag: "..."
receive_list_copy: False
self_link: http://localhost:9001/3.0/members/1/preferences

Changing preferences¶

Preferences for the address, user, or member can be changed through the API. You can change all the preferences for a particular object by using an HTTP PUT operation.

>>> dump_json('http://localhost:9001/3.0/addresses/bart@example.com'
...           '/preferences', {
...           'acknowledge_posts': True,
...           'delivery_mode': 'plaintext_digests',
...           'delivery_status': 'by_user',
...           'hide_address': False,
...           'preferred_language': 'ja',
...           'receive_list_copy': True,
...           'receive_own_postings': False,
...           }, method='PUT')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/addresses/bart@example.com'
...           '/preferences')
acknowledge_posts: True
delivery_mode: plaintext_digests
delivery_status: by_user
hide_address: False
http_etag: "..."
preferred_language: ja
receive_list_copy: True
receive_own_postings: False
self_link: http://localhost:9001/3.0/addresses/bart@example.com/preferences

You can also update just a few of the attributes using PATCH.

>>> dump_json('http://localhost:9001/3.0/addresses/bart@example.com'
...           '/preferences', {
...           'delivery_mode': 'plaintext_digests',
...           'receive_list_copy': False,
...           }, method='PATCH')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/addresses/bart@example.com'
...           '/preferences')
acknowledge_posts: True
delivery_mode: plaintext_digests
delivery_status: by_user
hide_address: False
http_etag: "..."
preferred_language: ja
receive_list_copy: False
receive_own_postings: False
self_link: http://localhost:9001/3.0/addresses/bart@example.com/preferences

Deleting preferences¶

Preferences for any of the levels, member, user, or address, can be entirely deleted.

>>> dump_json('http://localhost:9001/3.0/addresses/anne@example.com'
...           '/preferences', {
...           'preferred_language': 'ja',
...           }, method='PATCH')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/addresses/anne@example.com'
...           '/preferences')
acknowledge_posts: True
http_etag: "..."
preferred_language: ja
self_link: http://localhost:9001/3.0/addresses/anne@example.com/preferences

>>> dump_json('http://localhost:9001/3.0/addresses/anne@example.com'
...           '/preferences', method='DELETE')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/addresses/anne@example.com'
...           '/preferences')
http_etag: "..."
self_link: http://localhost:9001/3.0/addresses/anne@example.com/preferences

Combined member preferences¶

The member resource provides a way to access the set of preference in effect for a specific subscription. This stacks the preferences, so that a value is always available. The preference value is looked up first on the member, falling back to the address, then user, then system preference.

Preferences accessed through this interface are always read only.

>>> dump_json('http://localhost:9001/3.0/members/1/all/preferences')
acknowledge_posts: True
delivery_mode: plaintext_digests
delivery_status: by_user
http_etag: "..."
preferred_language: ja
receive_list_copy: False
receive_own_postings: False
self_link: http://localhost:9001/3.0/members/1/all/preferences

System preferences¶

The Mailman system itself has a default set of preference. All preference look ups fall back to these values, which are read-only.

>>> dump_json('http://localhost:9001/3.0/system/preferences')
acknowledge_posts: False
delivery_mode: regular
delivery_status: enabled
hide_address: True
http_etag: "..."
preferred_language: en
receive_list_copy: True
receive_own_postings: True
self_link: http://localhost:9001/3.0/system/preferences

Queues¶

You can get information about what messages are currently in the Mailman queues by querying the top-level queues resource. Of course, this information may be out-of-date by the time you receive a response, since queue management is asynchronous, but the information will be as current as possible.

You can get the list of all queue names.

>>> dump_json('http://localhost:9001/3.0/queues')
entry 0:
    count: 0
    directory: .../queue/archive
    files: []
    http_etag: ...
    name: archive
    self_link: http://localhost:9001/3.0/queues/archive
entry 1:
    count: 0
    directory: .../queue/bad
    files: []
    http_etag: ...
    name: bad
    self_link: http://localhost:9001/3.0/queues/bad
entry 2:
    count: 0
    directory: .../queue/bounces
    files: []
    http_etag: ...
    name: bounces
    self_link: http://localhost:9001/3.0/queues/bounces
entry 3:
    count: 0
    directory: .../queue/command
    files: []
    http_etag: ...
    name: command
    self_link: http://localhost:9001/3.0/queues/command
entry 4:
    count: 0
    directory: .../queue/digest
    files: []
    http_etag: ...
    name: digest
    self_link: http://localhost:9001/3.0/queues/digest
entry 5:
    count: 0
    directory: .../queue/in
    files: []
    http_etag: ...
    name: in
    self_link: http://localhost:9001/3.0/queues/in
entry 6:
    count: 0
    directory: .../queue/nntp
    files: []
    http_etag: ...
    name: nntp
    self_link: http://localhost:9001/3.0/queues/nntp
entry 7:
    count: 0
    directory: .../queue/out
    files: []
    http_etag: ...
    name: out
    self_link: http://localhost:9001/3.0/queues/out
entry 8:
    count: 0
    directory: .../queue/pipeline
    files: []
    http_etag: ...
    name: pipeline
    self_link: http://localhost:9001/3.0/queues/pipeline
entry 9:
    count: 0
    directory: .../queue/retry
    files: []
    http_etag: ...
    name: retry
    self_link: http://localhost:9001/3.0/queues/retry
entry 10:
    count: 0
    directory: .../queue/shunt
    files: []
    http_etag: ...
    name: shunt
    self_link: http://localhost:9001/3.0/queues/shunt
entry 11:
    count: 0
    directory: .../queue/virgin
    files: []
    http_etag: ...
    name: virgin
    self_link: http://localhost:9001/3.0/queues/virgin
http_etag: ...
self_link: http://localhost:9001/3.0/queues
start: 0
total_size: 12

Query an individual queue to get a count of, and the list of file base names in the queue. There are currently no files in the bad queue.

>>> dump_json('http://localhost:9001/3.0/queues/bad')
count: 0
directory: .../queue/bad
files: []
http_etag: ...
name: bad
self_link: http://localhost:9001/3.0/queues/bad

We can inject a message into the bad queue. It must be destined for an existing mailing list.

>>> dump_json('http://localhost:9001/3.0/lists', {
...     'fqdn_listname': 'ant@example.com',
...     })
content-length: 0
date: ...
location: http://localhost:9001/3.0/lists/ant.example.com
server: WSGIServer/0.2 CPython/...
status: 201

While list creation takes an FQDN list name, injecting a message to the queue requires a List ID.

>>> dump_json('http://localhost:9001/3.0/queues/bad', {
...     'list_id': 'ant.example.com',
...     'text': """\
... From: anne@example.com
... To: ant@example.com
... Subject: Testing
...
... """})
content-length: 0
date: ...
location: http://localhost:9001/3.0/queues/bad/...
server: ...
status: 201

And now the bad queue has at least one message in it.

>>> dump_json('http://localhost:9001/3.0/queues/bad')
count: 1
directory: .../queue/bad
files: ['...']
http_etag: ...
name: bad
self_link: http://localhost:9001/3.0/queues/bad

We can delete the injected message.

>>> json = call_http('http://localhost:9001/3.0/queues/bad')
>>> len(json['files'])
1
>>> dump_json('http://localhost:9001/3.0/queues/bad/{}'.format(
...           json['files'][0]),
...           method='DELETE')
content-length: 0
date: ...
server: ...
status: 204

And now the queue has no files.

>>> dump_json('http://localhost:9001/3.0/queues/bad')
count: 0
directory: .../queue/bad
files: []
http_etag: ...
name: bad
self_link: http://localhost:9001/3.0/queues/bad

Subscription moderation¶

Subscription (and sometimes unsubscription) requests can similarly be accepted, discarded, rejected, or deferred by the list moderators.

Viewing subscription requests¶

A mailing list starts with no pending subscription or unsubscription requests.

>>> ant = create_list('ant@example.com')
>>> ant.admin_immed_notify = False
>>> from mailman.interfaces.mailinglist import SubscriptionPolicy
>>> ant.subscription_policy = SubscriptionPolicy.moderate
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/lists/ant@example.com/requests')
http_etag: "..."
start: 0
total_size: 0

When Anne tries to subscribe to the Ant list, her subscription is held for moderator approval.

>>> from mailman.interfaces.registrar import IRegistrar
>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> registrar = IRegistrar(ant)
>>> manager = getUtility(IUserManager)
>>> anne = manager.create_address('anne@example.com', 'Anne Person')
>>> token, token_owner, member = registrar.register(
...     anne, pre_verified=True, pre_confirmed=True)
>>> print(member)
None

The message is being held for moderator approval.

>>> print(token_owner.name)
moderator

The subscription request can be viewed in the REST API.

>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/lists/ant@example.com/requests')
entry 0:
    display_name: Anne Person
    email: anne@example.com
    http_etag: "..."
    list_id: ant.example.com
    token: ...
    token_owner: moderator
    when: 2005-08-01T07:49:23
http_etag: "..."
start: 0
total_size: 1

Viewing individual requests¶

You can view an individual membership change request by providing the token (a.k.a. request id). Anne’s subscription request looks like this.

>>> dump_json('http://localhost:9001/3.0/lists/ant@example.com/'
...           'requests/{}'.format(token))
display_name: Anne Person
email: anne@example.com
http_etag: "..."
list_id: ant.example.com
token: ...
token_owner: moderator
when: 2005-08-01T07:49:23

Disposing of subscription requests¶

Moderators can dispose of held subscription requests by POSTing back to the request’s resource. The POST data requires an action of one of the following:

discard - throw the request away.

reject - the request is denied and a notification is sent to the email

address requesting the membership change.

defer - defer any action on this membership change (continue to hold it).

accept - accept the membership change.

Anne’s subscription request is accepted.

>>> dump_json('http://localhost:9001/3.0/lists/'
...           'ant@example.com/requests/{}'.format(token),
...           {'action': 'accept'})
content-length: 0
date: ...
server: ...
status: 204

Anne is now a member of the mailing list.

>>> ant.members.get_member('anne@example.com')
<Member: Anne Person <anne@example.com> on ant@example.com
         as MemberRole.member>

There are no more membership change requests.

>>> dump_json('http://localhost:9001/3.0/lists/ant@example.com/requests')
http_etag: "..."
start: 0
total_size: 0

System configuration¶

The entire system configuration is available through the REST API. You can get a list of all defined sections.

>>> dump_json('http://localhost:9001/3.0/system/configuration')
http_etag: ...
sections: ['antispam', 'archiver.mail_archive', 'archiver.master', ...

You can also get all the values for a particular section.

>>> dump_json('http://localhost:9001/3.0/system/configuration/mailman')
default_language: en
email_commands_max_lines: 10
filtered_messages_are_preservable: no
html_to_plain_text_command: /usr/bin/lynx -dump $filename
http_etag: ...
layout: testing
noreply_address: noreply
pending_request_life: 3d
post_hook:
pre_hook:
sender_headers: from from_ reply-to sender
site_owner: noreply@example.com

Dotted section names work too, for example, to get the French language settings section.

>>> dump_json('http://localhost:9001/3.0/system/configuration/language.fr')
charset: iso-8859-1
description: French
enabled: yes
http_etag: ...

Users¶

The REST API can be used to add and remove users, add and remove user addresses, and change their preferred address, password, or name. The API can also be used to verify a user’s password.

Users are different than members; the latter represents an email address subscribed to a specific mailing list. Users are just people that Mailman knows about.

There are no users yet.

>>> dump_json('http://localhost:9001/3.0/users')
http_etag: "..."
start: 0
total_size: 0

Anne is added, with an email address. Her user record gets a user_id.

>>> from zope.component import getUtility
>>> from mailman.interfaces.usermanager import IUserManager
>>> user_manager = getUtility(IUserManager)
>>> anne = user_manager.create_user('anne@example.com', 'Anne Person')
>>> transaction.commit()
>>> int(anne.user_id.int)
1

Anne’s user record is returned as an entry into the collection of all users.

>>> dump_json('http://localhost:9001/3.0/users')
entry 0:
    created_on: 2005-08-01T07:49:23
    display_name: Anne Person
    http_etag: "..."
    is_server_owner: False
    self_link: http://localhost:9001/3.0/users/1
    user_id: 1
http_etag: "..."
start: 0
total_size: 1

A user might not have a display name, in which case, the attribute will not be returned in the REST API.

>>> bart = user_manager.create_user('bart@example.com')
>>> transaction.commit()
>>> dump_json('http://localhost:9001/3.0/users')
entry 0:
    created_on: 2005-08-01T07:49:23
    display_name: Anne Person
    http_etag: "..."
    is_server_owner: False
    self_link: http://localhost:9001/3.0/users/1
    user_id: 1
entry 1:
    created_on: 2005-08-01T07:49:23
    http_etag: "..."
    is_server_owner: False
    self_link: http://localhost:9001/3.0/users/2
    user_id: 2
http_etag: "..."
start: 0
total_size: 2

Paginating over user records¶

Instead of returning all the user records at once, it’s possible to return them in pages by adding the GET parameters count and page to the request URI. Page 1 is the first page and count defines the size of the page.

>>> dump_json('http://localhost:9001/3.0/users?count=1&page=1')
entry 0:
    created_on: 2005-08-01T07:49:23
    display_name: Anne Person
    http_etag: "..."
    is_server_owner: False
    self_link: http://localhost:9001/3.0/users/1
    user_id: 1
http_etag: "..."
start: 0
total_size: 2

>>> dump_json('http://localhost:9001/3.0/users?count=1&page=2')
entry 0:
    created_on: 2005-08-01T07:49:23
    http_etag: "..."
    is_server_owner: False
    self_link: http://localhost:9001/3.0/users/2
    user_id: 2
http_etag: "..."
start: 1
total_size: 2

Creating users¶

New users can be created by POSTing to the users collection. At a minimum, the user’s email address must be provided.

>>> dump_json('http://localhost:9001/3.0/users', {
...           'email': 'cris@example.com',
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/users/3
server: ...
status: 201

Cris is now a user known to the system, but he has no display name.

>>> user_manager.get_user('cris@example.com')
<User "" (3) at ...>

Cris’s user record can also be accessed via the REST API, using her user id. Note that because no password was given when the record was created, a random one was assigned to her.

>>> dump_json('http://localhost:9001/3.0/users/3')
created_on: 2005-08-01T07:49:23
http_etag: "..."
is_server_owner: False
password: {plaintext}...
self_link: http://localhost:9001/3.0/users/3
user_id: 3

Because email addresses just have an @ sign in then, there’s no confusing them with user ids. Thus, Cris’s record can be retrieved via her email address.

>>> dump_json('http://localhost:9001/3.0/users/cris@example.com')
created_on: 2005-08-01T07:49:23
http_etag: "..."
is_server_owner: False
password: {plaintext}...
self_link: http://localhost:9001/3.0/users/3
user_id: 3

Providing a display name¶

When a user is added, a display name can be provided.

>>> transaction.abort()
>>> dump_json('http://localhost:9001/3.0/users', {
...           'email': 'dave@example.com',
...           'display_name': 'Dave Person',
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/users/4
server: ...
status: 201

Dave’s user record includes his display name.

>>> dump_json('http://localhost:9001/3.0/users/4')
created_on: 2005-08-01T07:49:23
display_name: Dave Person
http_etag: "..."
is_server_owner: False
password: {plaintext}...
self_link: http://localhost:9001/3.0/users/4
user_id: 4

Providing passwords¶

To avoid getting assigned a random, and irretrievable password (but one which can be reset), you can provide a password when the user is created. By default, the password is provided in plain text, and it is hashed by Mailman before being stored.

>>> transaction.abort()
>>> dump_json('http://localhost:9001/3.0/users', {
...           'email': 'elly@example.com',
...           'display_name': 'Elly Person',
...           'password': 'supersekrit',
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/users/5
server: ...
status: 201

When we view Elly’s user record, we can tell that her password has been hashed because it has the hash algorithm prefix (i.e. the {plaintext} marker).

>>> dump_json('http://localhost:9001/3.0/users/5')
created_on: 2005-08-01T07:49:23
display_name: Elly Person
http_etag: "..."
is_server_owner: False
password: {plaintext}supersekrit
self_link: http://localhost:9001/3.0/users/5
user_id: 5

Updating users¶

Dave’s display name can be changed through the REST API.

>>> dump_json('http://localhost:9001/3.0/users/4', {
...           'display_name': 'David Person',
...           }, method='PATCH')
content-length: 0
date: ...
server: ...
status: 204

Dave’s display name has been updated.

>>> dump_json('http://localhost:9001/3.0/users/dave@example.com')
created_on: 2005-08-01T07:49:23
display_name: David Person
http_etag: "..."
is_server_owner: False
password: {plaintext}...
self_link: http://localhost:9001/3.0/users/4
user_id: 4

Dave can also be assigned a new password by providing in the new cleartext password. Mailman will hash this before it is stored internally.

>>> dump_json('http://localhost:9001/3.0/users/4', {
...           'cleartext_password': 'clockwork angels',
...           }, method='PATCH')
content-length: 0
date: ...
server: ...
status: 204

As described above, even though you see {plaintext}clockwork angels below, it has still been hashed before storage. The default hashing algorithm for the test suite is a plain text hash, but you can see that it works by the addition of the algorithm prefix.

>>> dump_json('http://localhost:9001/3.0/users/4')
created_on: 2005-08-01T07:49:23
display_name: David Person
http_etag: "..."
is_server_owner: False
password: {plaintext}clockwork angels
self_link: http://localhost:9001/3.0/users/4
user_id: 4

You can change both the display name and the password by PUTing the full resource.

>>> dump_json('http://localhost:9001/3.0/users/4', {
...           'cleartext_password': 'the garden',
...           'display_name': 'David Personhood',
...           'is_server_owner': False,
...           }, method='PUT')
content-length: 0
date: ...
server: ...
status: 204

Dave’s user record has been updated.

>>> dump_json('http://localhost:9001/3.0/users/dave@example.com')
created_on: 2005-08-01T07:49:23
display_name: David Personhood
http_etag: "..."
is_server_owner: False
password: {plaintext}the garden
self_link: http://localhost:9001/3.0/users/4
user_id: 4

Deleting users via the API¶

Users can also be deleted via the API.

>>> dump_json('http://localhost:9001/3.0/users/cris@example.com',
...           method='DELETE')
content-length: 0
date: ...
server: ...
status: 204

User addresses¶

Fred may have any number of email addresses associated with his user account, and we can find them all through the API.

Through some other means, Fred registers a bunch of email addresses and associates them with his user account.

>>> fred = user_manager.create_user('fred@example.com', 'Fred Person')
>>> fred.register('fperson@example.com')
<Address: fperson@example.com [not verified] at ...>
>>> fred.register('fred.person@example.com')
<Address: fred.person@example.com [not verified] at ...>
>>> fred.register('Fred.Q.Person@example.com')
<Address: Fred.Q.Person@example.com [not verified]
          key: fred.q.person@example.com at ...>
>>> transaction.commit()

When we access Fred’s addresses via the REST API, they are sorted in lexical order by original (i.e. case-preserved) email address.

>>> dump_json('http://localhost:9001/3.0/users/fred@example.com/addresses')
entry 0:
    email: fred.q.person@example.com
    http_etag: "..."
    original_email: Fred.Q.Person@example.com
    registered_on: 2005-08-01T07:49:23
    self_link:
        http://localhost:9001/3.0/addresses/fred.q.person@example.com
    user: http://localhost:9001/3.0/users/6
entry 1:
    email: fperson@example.com
    http_etag: "..."
    original_email: fperson@example.com
    registered_on: 2005-08-01T07:49:23
    self_link: http://localhost:9001/3.0/addresses/fperson@example.com
    user: http://localhost:9001/3.0/users/6
entry 2:
    email: fred.person@example.com
    http_etag: "..."
    original_email: fred.person@example.com
    registered_on: 2005-08-01T07:49:23
    self_link: http://localhost:9001/3.0/addresses/fred.person@example.com
    user: http://localhost:9001/3.0/users/6
entry 3:
    display_name: Fred Person
    email: fred@example.com
    http_etag: "..."
    original_email: fred@example.com
    registered_on: 2005-08-01T07:49:23
    self_link: http://localhost:9001/3.0/addresses/fred@example.com
    user: http://localhost:9001/3.0/users/6
http_etag: "..."
start: 0
total_size: 4

In fact, since these are all associated with Fred’s user account, any of the addresses can be used to look up Fred’s user record.

>>> dump_json('http://localhost:9001/3.0/users/fred@example.com')
created_on: 2005-08-01T07:49:23
display_name: Fred Person
http_etag: "..."
is_server_owner: False
self_link: http://localhost:9001/3.0/users/6
user_id: 6

>>> dump_json('http://localhost:9001/3.0/users/fred.person@example.com')
created_on: 2005-08-01T07:49:23
display_name: Fred Person
http_etag: "..."
is_server_owner: False
self_link: http://localhost:9001/3.0/users/6
user_id: 6

>>> dump_json('http://localhost:9001/3.0/users/fperson@example.com')
created_on: 2005-08-01T07:49:23
display_name: Fred Person
http_etag: "..."
is_server_owner: False
self_link: http://localhost:9001/3.0/users/6
user_id: 6

>>> dump_json('http://localhost:9001/3.0/users/Fred.Q.Person@example.com')
created_on: 2005-08-01T07:49:23
display_name: Fred Person
http_etag: "..."
is_server_owner: False
self_link: http://localhost:9001/3.0/users/6
user_id: 6

Verifying passwords¶

A user’s password is stored internally in hashed form. Logging in a user is the process of verifying a provided clear text password against the hashed internal password.

When Elly was added as a user, she provided a password in the clear. Now the password is hashed and getting her user record returns the hashed password.

>>> dump_json('http://localhost:9001/3.0/users/5')
created_on: 2005-08-01T07:49:23
display_name: Elly Person
http_etag: "..."
is_server_owner: False
password: {plaintext}supersekrit
self_link: http://localhost:9001/3.0/users/5
user_id: 5

Unless the client can run the hashing algorithm on the login text that Elly provided, and do its own comparison, the client should let the REST API handle password verification.

This time, Elly successfully logs into Mailman.

>>> dump_json('http://localhost:9001/3.0/users/5/login', {
...           'cleartext_password': 'supersekrit',
...           }, method='POST')
content-length: 0
date: ...
server: ...
status: 204

Server owners¶

Users can be designated as server owners. Elly is not currently a server owner.

>>> dump_json('http://localhost:9001/3.0/users/5')
created_on: 2005-08-01T07:49:23
display_name: Elly Person
http_etag: "..."
is_server_owner: False
password: {plaintext}supersekrit
self_link: http://localhost:9001/3.0/users/5
user_id: 5

Let’s make her a server owner.

>>> dump_json('http://localhost:9001/3.0/users/5', {
...           'is_server_owner': True,
...           }, method='PATCH')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/users/5')
created_on: 2005-08-01T07:49:23
display_name: Elly Person
http_etag: "..."
is_server_owner: True
password: {plaintext}supersekrit
self_link: http://localhost:9001/3.0/users/5
user_id: 5

Elly later retires as server owner.

>>> dump_json('http://localhost:9001/3.0/users/5', {
...           'is_server_owner': False,
...           }, method='PATCH')
content-length: 0
date: ...
server: ...
status: 204

>>> dump_json('http://localhost:9001/3.0/users/5')
created_on: 2005-08-01T07:49:23
display_name: Elly Person
http_etag: "..."
is_server_owner: False
password: {plaintext}...
self_link: http://localhost:9001/3.0/users/5
user_id: 5

Gwen, a new users, takes over as a server owner.

>>> dump_json('http://localhost:9001/3.0/users', {
...           'display_name': 'Gwen Person',
...           'email': 'gwen@example.com',
...           'is_server_owner': True,
...           })
content-length: 0
date: ...
location: http://localhost:9001/3.0/users/7
server: ...
status: 201

>>> dump_json('http://localhost:9001/3.0/users/7')
created_on: 2005-08-01T07:49:23
display_name: Gwen Person
http_etag: "..."
is_server_owner: True
password: {plaintext}...
self_link: http://localhost:9001/3.0/users/7
user_id: 7

Chains¶

When a new message is posted to a mailing list, Mailman uses a set of rule chains to decide whether the message gets accepted for posting, rejected, discarded, or held for moderator approval.

There are a number of built-in chains available that act as end-points in the processing of messages.

The Discard chain¶

The discard chain simply throws the message away.

>>> chain = config.chains['discard']
>>> print(chain.name)
discard
>>> print(chain.description)
Discard a message and stop processing.

>>> mlist = create_list('test@example.com')
>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: My first post
... Message-ID: <first>
...
... An important message.
... """)

>>> def print_msgid(event):
...     print('{0}: {1}'.format(
...         event.chain.name.upper(), event.msg.get('message-id', 'n/a')))

>>> from mailman.core.chains import process
>>> from mailman.testing.helpers import event_subscribers

>>> with event_subscribers(print_msgid):
...     process(mlist, msg, {}, 'discard')
DISCARD: <first>

The Reject chain¶

The reject chain bounces the message back to the original sender, and logs this action.

>>> chain = config.chains['reject']
>>> print(chain.name)
reject
>>> print(chain.description)
Reject/bounce a message and stop processing.

>>> with event_subscribers(print_msgid):
...     process(mlist, msg, {}, 'reject')
REJECT: <first>

The bounce message is now sitting in the virgin queue.

>>> from mailman.testing.helpers import get_queue_messages
>>> qfiles = get_queue_messages('virgin')
>>> len(qfiles)
1
>>> print(qfiles[0].msg.as_string())
Subject: My first post
From: test-owner@example.com
To: aperson@example.com
...
[No bounce details are available]
...
Content-Type: message/rfc822
MIME-Version: 1.0
<BLANKLINE>
From: aperson@example.com
To: test@example.com
Subject: My first post
Message-ID: <first>
<BLANKLINE>
An important message.
<BLANKLINE>
...

The Hold Chain¶

The hold chain places the message into the administrative request database and depending on the list’s settings, sends a notification to both the original sender and the list moderators.

>>> chain = config.chains['hold']
>>> print(chain.name)
hold
>>> print(chain.description)
Hold a message and stop processing.

>>> with event_subscribers(print_msgid):
...     process(mlist, msg, {}, 'hold')
HOLD: <first>

There are now two messages in the virgin queue, one to the list moderators and one to the original author.

>>> qfiles = get_queue_messages('virgin', sort_on='to')
>>> len(qfiles)
2

One of the message is addressed to the mailing list moderators, and the other is addressed to the original sender.

>>> from operator import itemgetter
>>> messages = sorted((item.msg for item in qfiles),
...                   key=itemgetter('to'), reverse=True)

This one is addressed to the list moderators.

>>> print(messages[0].as_string())
Subject: test@example.com post from aperson@example.com requires approval
From: test-owner@example.com
To: test-owner@example.com
MIME-Version: 1.0
...
As list administrator, your authorization is requested for the
following mailing list posting:
<BLANKLINE>
    List:    test@example.com
    From:    aperson@example.com
    Subject: My first post
<BLANKLINE>
The message is being held because:
<BLANKLINE>
    N/A
<BLANKLINE>
At your convenience, visit your dashboard to approve or deny the
request.
<BLANKLINE>
...
Content-Type: message/rfc822
MIME-Version: 1.0
<BLANKLINE>
From: aperson@example.com
To: test@example.com
Subject: My first post
Message-ID: <first>
X-Message-ID-Hash: RXJU4JL6N2OUN3OYMXXPPSCR7P7JE2BW
<BLANKLINE>
An important message.
<BLANKLINE>
...
Content-Type: message/rfc822
MIME-Version: 1.0
<BLANKLINE>
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Subject: confirm ...
From: test-request@example.com
...
<BLANKLINE>
If you reply to this message, keeping the Subject: header intact,
Mailman will discard the held message.  Do this if the message is
spam.  If you reply to this message and include an Approved: header
with the list password in it, the message will be approved for posting
to the list.  The Approved: header can also appear in the first line
of the body of the reply.
...

This message is addressed to the sender of the message.

>>> print(messages[1].as_string())
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Subject: Your message to test@example.com awaits moderator approval
From: test-bounces@example.com
To: aperson@example.com
...
Your mail to 'test@example.com' with the subject
<BLANKLINE>
    My first post
<BLANKLINE>
Is being held until the list moderator can review it for approval.
<BLANKLINE>
The message is being held because:
<BLANKLINE>
    N/A
<BLANKLINE>
Either the message will get posted to the list, or you will receive
notification of the moderator's decision.

The Accept chain¶

The accept chain sends the message on the pipeline queue, where it will be processed and sent on to the list membership.

>>> chain = config.chains['accept']
>>> print(chain.name)
accept
>>> print(chain.description)
Accept a message.

>>> with event_subscribers(print_msgid):
...     process(mlist, msg, {}, 'accept')
ACCEPT: <first>

>>> qfiles = get_queue_messages('pipeline')
>>> len(qfiles)
1
>>> print(qfiles[0].msg.as_string())
From: aperson@example.com
To: test@example.com
Subject: My first post
Message-ID: <first>
X-Message-ID-Hash: RXJU4JL6N2OUN3OYMXXPPSCR7P7JE2BW

An important message.

Run-time chains¶

We can also define chains at run time, and these chains can be mutated. Run-time chains are made up of links where each link associates both a rule and a jump. The rule is really a rule name, which is looked up when needed. The jump names a chain which is jumped to if the rule matches.

There is one built-in posting chain. This is the default chain to use when no other input chain is defined for a mailing list. It runs through the default rules.

>>> chain = config.chains['default-posting-chain']
>>> print(chain.name)
default-posting-chain
>>> print(chain.description)
The built-in moderation chain.

Once the sender is a member of the mailing list, the previously created message is innocuous enough that it should pass through all default rules. This message will end up in the pipeline queue.

>>> from mailman.testing.helpers import subscribe
>>> subscribe(mlist, 'Anne')
<Member: aperson@example.com on test@example.com as MemberRole.member>

>>> with event_subscribers(print_msgid):
...     process(mlist, msg, {})
ACCEPT: <first>

>>> qfiles = get_queue_messages('pipeline')
>>> len(qfiles)
1
>>> print(qfiles[0].msg.as_string())
From: aperson@example.com
To: test@example.com
Subject: My first post
Message-ID: <first>
X-Message-ID-Hash: RXJU4JL6N2OUN3OYMXXPPSCR7P7JE2BW
X-Mailman-Rule-Misses: approved; emergency; loop; member-moderation;
    administrivia; implicit-dest; max-recipients; max-size;
    news-moderation; no-subject; suspicious-header; nonmember-moderation

An important message.

In addition, the message metadata now contains lists of all rules that have hit and all rules that have missed.

>>> dump_list(qfiles[0].msgdata['rule_hits'])
*Empty*
>>> dump_list(qfiles[0].msgdata['rule_misses'])
administrivia
approved
emergency
implicit-dest
loop
max-recipients
max-size
member-moderation
news-moderation
no-subject
nonmember-moderation
suspicious-header

Runners¶

The runners are the processes that perform long-running tasks, such as moving messages around the Mailman queues. Some runners don’t manage queues, such as the LMTP and REST API handling runners. Each runner that manages a queue directory though, is responsible for a slice of the hash space. It processes all the files in its slice, sleeps a little while, then wakes up and runs through its queue files again.

Basic architecture¶

The basic architecture of runner is implemented in the base class that all runners inherit from. This base class implements a .run() method that runs continuously in a loop until the .stop() method is called.

>>> mlist = create_list('test@example.com')

Here is a very simple derived runner class. Runners use a configuration section in the configuration files to determine run characteristics, such as the queue directory to use. Here we push a configuration section for the test runner.

>>> config.push('test-runner', """
... [runner.test]
... max_restarts: 1
... """)

>>> from mailman.core.runner import Runner
>>> class TestableRunner(Runner):
...     def _dispose(self, mlist, msg, msgdata):
...         self.msg = msg
...         self.msgdata = msgdata
...         return False
...
...     def _do_periodic(self):
...         self.stop()
...
...     def _snooze(self, filecnt):
...         return

>>> runner = TestableRunner('test')

This runner doesn’t do much except run once, storing the message and metadata on instance variables.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
...
... A test message.
... """)
>>> switchboard = config.switchboards['test']
>>> filebase = switchboard.enqueue(msg, listid=mlist.list_id,
...                                foo='yes', bar='no')
>>> runner.run()
>>> print(runner.msg.as_string())
From: aperson@example.com
To: test@example.com
<BLANKLINE>
A test message.
<BLANKLINE>
>>> dump_msgdata(runner.msgdata)
_parsemsg: False
bar      : no
foo      : yes
lang     : en
listid   : test.example.com
version  : 3

XXX More of the Runner API should be tested.

The switchboard¶

The switchboard is subsystem that moves messages between queues. Each instance of a switchboard is responsible for one queue directory.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: _xtest@example.com
...
... A test message.
... """)

Create a switchboard by giving its queue name and directory.

>>> import os
>>> queue_directory = os.path.join(config.QUEUE_DIR, 'test')
>>> from mailman.core.switchboard import Switchboard
>>> switchboard = Switchboard('test', queue_directory)
>>> print(switchboard.name)
test
>>> switchboard.queue_directory == queue_directory
True

Here’s a helper function for ensuring things work correctly.

>>> def check_qfiles(directory=None):
...     if directory is None:
...         directory = queue_directory
...     files = {}
...     for qfile in os.listdir(directory):
...         root, ext = os.path.splitext(qfile)
...         files[ext] = files.get(ext, 0) + 1
...     if len(files) == 0:
...         print('empty')
...     for ext in sorted(files):
...         print('{0}: {1}'.format(ext, files[ext]))

Enqueing and dequeing¶

The message can be enqueued with metadata specified in the passed in dictionary.

>>> filebase = switchboard.enqueue(msg)
>>> check_qfiles()
.pck: 1

To read the contents of a queue file, dequeue it.

>>> msg, msgdata = switchboard.dequeue(filebase)
>>> print(msg.as_string())
From: aperson@example.com
To: _xtest@example.com
<BLANKLINE>
A test message.
<BLANKLINE>
>>> dump_msgdata(msgdata)
_parsemsg: False
version  : 3
>>> check_qfiles()
.bak: 1

To complete the dequeing process, removing all traces of the message file, finish it (without preservation).

>>> switchboard.finish(filebase)
>>> check_qfiles()
empty

When enqueing a file, you can provide additional metadata keys by using keyword arguments.

>>> filebase = switchboard.enqueue(msg, {'foo': 1}, bar=2)
>>> msg, msgdata = switchboard.dequeue(filebase)
>>> switchboard.finish(filebase)
>>> dump_msgdata(msgdata)
_parsemsg: False
bar      : 2
foo      : 1
version  : 3

Keyword arguments override keys from the metadata dictionary.

>>> filebase = switchboard.enqueue(msg, {'foo': 1}, foo=2)
>>> msg, msgdata = switchboard.dequeue(filebase)
>>> switchboard.finish(filebase)
>>> dump_msgdata(msgdata)
_parsemsg: False
foo      : 2
version  : 3

Iterating over files¶

There are two ways to iterate over all the files in a switchboard’s queue. Normally, queue files end in .pck (for ‘pickle’) and the easiest way to iterate over just these files is to use the .files attribute.

>>> filebase_1 = switchboard.enqueue(msg, foo=1)
>>> filebase_2 = switchboard.enqueue(msg, foo=2)
>>> filebase_3 = switchboard.enqueue(msg, foo=3)
>>> filebases = sorted((filebase_1, filebase_2, filebase_3))
>>> sorted(switchboard.files) == filebases
True
>>> check_qfiles()
.pck: 3

You can also use the .get_files() method if you want to iterate over all the file bases for some other extension.

>>> for filebase in switchboard.get_files():
...     msg, msgdata = switchboard.dequeue(filebase)
>>> bakfiles = sorted(switchboard.get_files('.bak'))
>>> bakfiles == filebases
True
>>> check_qfiles()
.bak: 3
>>> for filebase in switchboard.get_files('.bak'):
...     switchboard.finish(filebase)
>>> check_qfiles()
empty

Recovering files¶

Calling .dequeue() without calling .finish() leaves .bak backup files in place. These can be recovered when the switchboard is instantiated.

>>> filebase_1 = switchboard.enqueue(msg, foo=1)
>>> filebase_2 = switchboard.enqueue(msg, foo=2)
>>> filebase_3 = switchboard.enqueue(msg, foo=3)
>>> for filebase in switchboard.files:
...     msg, msgdata = switchboard.dequeue(filebase)
...     # Don't call .finish()
>>> check_qfiles()
.bak: 3
>>> switchboard_2 = Switchboard('test', queue_directory, recover=True)
>>> check_qfiles()
.pck: 3

The files can be recovered explicitly.

>>> for filebase in switchboard.files:
...     msg, msgdata = switchboard.dequeue(filebase)
...     # Don't call .finish()
>>> check_qfiles()
.bak: 3
>>> switchboard.recover_backup_files()
>>> check_qfiles()
.pck: 3

But the files will only be recovered at most three times before they are considered defective. In order to prevent mail bombs and loops, once this maximum is reached, the files will be preserved in the ‘bad’ queue.

>>> for filebase in switchboard.files:
...     msg, msgdata = switchboard.dequeue(filebase)
...     # Don't call .finish()
>>> check_qfiles()
.bak: 3
>>> switchboard.recover_backup_files()
>>> check_qfiles()
empty

>>> bad = config.switchboards['bad']
>>> check_qfiles(bad.queue_directory)
.psv: 3

Clean up¶

>>> for file in os.listdir(bad.queue_directory):
...     os.remove(os.path.join(bad.queue_directory, file))
>>> check_qfiles(bad.queue_directory)
empty

Queue slices¶

XXX Add tests for queue slices.

Banning email addresses¶

Email addresses can be banned from ever subscribing, either to a specific mailing list or globally within the Mailman system. Both explicit email addresses and email address patterns can be banned.

Bans are managed through the Ban Manager. There are ban managers for specific lists, and there is a global ban manager. To get access to the global ban manager, adapt None.

>>> from mailman.interfaces.bans import IBanManager
>>> global_bans = IBanManager(None)

At first, no email addresses are banned globally.

>>> global_bans.is_banned('anne@example.com')
False

To get a list-specific ban manager, adapt the mailing list object.

>>> mlist = create_list('test@example.com')
>>> test_bans = IBanManager(mlist)

There are no bans for this particular list.

>>> test_bans.is_banned('bart@example.com')
False

Specific bans¶

An email address can be banned from a specific mailing list by adding a ban to the list’s ban manager.

>>> test_bans.ban('cris@example.com')
>>> test_bans.is_banned('cris@example.com')
True
>>> test_bans.is_banned('bart@example.com')
False

However, this is not a global ban.

>>> global_bans.is_banned('cris@example.com')
False

Global bans¶

An email address can be banned globally, so that it cannot be subscribed to any mailing list.

>>> global_bans.ban('dave@example.com')

Because there is a global ban, Dave is also banned from the mailing list.

>>> test_bans.is_banned('dave@example.com')
True

Even when a new mailing list is created, Dave is still banned from this list because of his global ban.

>>> sample = create_list('sample@example.com')
>>> sample_bans = IBanManager(sample)
>>> sample_bans.is_banned('dave@example.com')
True

Dave is of course banned globally.

>>> global_bans.is_banned('dave@example.com')
True

Cris however is not banned globally.

>>> global_bans.is_banned('cris@example.com')
False

Even though Cris is not banned globally, we can add a global ban for her.

>>> global_bans.ban('cris@example.com')
>>> global_bans.is_banned('cris@example.com')
True

Cris is now banned from all mailing lists.

>>> test_bans.is_banned('cris@example.com')
True
>>> sample_bans.is_banned('cris@example.com')
True

We can remove the global ban to once again just ban her address from just the test list.

>>> global_bans.unban('cris@example.com')
>>> global_bans.is_banned('cris@example.com')
False
>>> test_bans.is_banned('cris@example.com')
True
>>> sample_bans.is_banned('cris@example.com')
False

Regular expression bans¶

Entire email address patterns can be banned, both for a specific mailing list and globally, just as specific addresses can be banned. Use this for example, when an entire domain is a spam faucet. When using a pattern, the email address must start with a caret (^).

>>> test_bans.ban('^.*@example.org')

Now, no one from example.org can subscribe to the test mailing list.

>>> test_bans.is_banned('elle@example.org')
True
>>> test_bans.is_banned('eperson@example.org')
True

example.com addresses are not banned.

>>> test_bans.is_banned('elle@example.com')
False

example.org addresses are not banned globally, nor for any other mailing list.

>>> sample_bans.is_banned('elle@example.org')
False
>>> global_bans.is_banned('elle@example.org')
False

Of course, we can ban everyone from example.org globally too.

>>> global_bans.ban('^.*@example.org')
>>> sample_bans.is_banned('elle@example.org')
True
>>> global_bans.is_banned('elle@example.org')
True

We can remove the mailing list ban on the pattern, though the global ban will still be in place.

>>> test_bans.unban('^.*@example.org')
>>> test_bans.is_banned('elle@example.org')
True
>>> sample_bans.is_banned('elle@example.org')
True
>>> global_bans.is_banned('elle@example.org')
True

But once the global ban is removed, everyone from example.org can subscribe to the mailing lists.

>>> global_bans.unban('^.*@example.org')
>>> test_bans.is_banned('elle@example.org')
False
>>> sample_bans.is_banned('elle@example.org')
False
>>> global_bans.is_banned('elle@example.org')
False

Adding and removing bans¶

It is not an error to add a ban more than once. These are just ignored.

>>> test_bans.ban('fred@example.com')
>>> test_bans.ban('fred@example.com')
>>> test_bans.is_banned('fred@example.com')
True

Nor is it an error to remove a ban more than once.

>>> test_bans.unban('fred@example.com')
>>> test_bans.unban('fred@example.com')
>>> test_bans.is_banned('fred@example.com')
False

Bounces¶

An important feature of Mailman is automatic bounce processing.

Bounces, or message rejection¶

Mailman can bounce messages back to the original sender. This is essentially equivalent to rejecting the message with notification. Mailing lists can bounce a message with an optional error message.

>>> mlist = create_list('text@example.com')

Any message can be bounced.

>>> msg = message_from_string("""\
... To: text@example.com
... From: aperson@example.com
... Subject: Something important
...
... I sometimes say something important.
... """)

Bounce a message by passing in the original message, and an optional error message. The bounced message ends up in the virgin queue, awaiting sending to the original message author.

>>> from mailman.app.bounces import bounce_message
>>> bounce_message(mlist, msg)
>>> from mailman.testing.helpers import get_queue_messages
>>> items = get_queue_messages('virgin')
>>> len(items)
1
>>> print(items[0].msg.as_string())
Subject: Something important
From: text-owner@example.com
To: aperson@example.com
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="..."
Message-ID: ...
Date: ...
Precedence: bulk
<BLANKLINE>
--...
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
<BLANKLINE>
[No bounce details are available]
--...
Content-Type: message/rfc822
MIME-Version: 1.0
<BLANKLINE>
To: text@example.com
From: aperson@example.com
Subject: Something important
<BLANKLINE>
I sometimes say something important.
<BLANKLINE>
--...--

An error message can be given when the message is bounced, and this will be included in the payload of the text/plain part. The error message must be passed in as an instance of a RejectMessage exception.

>>> from mailman.core.errors import RejectMessage
>>> error = RejectMessage("This wasn't very important after all.")
>>> bounce_message(mlist, msg, error)
>>> items = get_queue_messages('virgin')
>>> len(items)
1
>>> print(items[0].msg.as_string())
Subject: Something important
From: text-owner@example.com
To: aperson@example.com
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="..."
Message-ID: ...
Date: ...
Precedence: bulk
<BLANKLINE>
--...
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
<BLANKLINE>
This wasn't very important after all.
--...
Content-Type: message/rfc822
MIME-Version: 1.0
<BLANKLINE>
To: text@example.com
From: aperson@example.com
Subject: Something important
<BLANKLINE>
I sometimes say something important.
<BLANKLINE>
--...--

Hooks¶

Mailman defines two initialization hooks, one which is run early in the initialization process and the other run late in the initialization process. Hooks name an importable callable so it must be accessible on sys.path.

>>> import os, sys
>>> from mailman.config import config
>>> config_directory = os.path.dirname(config.filename)
>>> sys.path.insert(0, config_directory)

>>> hook_path = os.path.join(config_directory, 'hooks.py')
>>> with open(hook_path, 'w') as fp:
...     print("""\
... counter = 1
... def pre_hook():
...     global counter
...     print('pre-hook:', counter)
...     counter += 1
...
... def post_hook():
...     global counter
...     print('post-hook:', counter)
...     counter += 1
... """, file=fp)
>>> fp.close()

Pre-hook¶

We can set the pre-hook in the configuration file.

>>> config_path = os.path.join(config_directory, 'hooks.cfg')
>>> with open(config_path, 'w') as fp:
...     print("""\
... [meta]
... extends: test.cfg
...
... [mailman]
... pre_hook: hooks.pre_hook
... """, file=fp)

The hooks are run in the second and third steps of initialization. However, we can’t run those initialization steps in process, so call a command line script that will produce no output to force the hooks to run.

>>> import subprocess
>>> from mailman.testing.layers import ConfigLayer
>>> def call():
...     exe = os.path.join(os.path.dirname(sys.executable), 'mailman')
...     env = dict(MAILMAN_CONFIG_FILE=config_path,
...                PYTHONPATH=config_directory)
...     test_cfg = os.environ.get('MAILMAN_EXTRA_TESTING_CFG')
...     if test_cfg is not None:
...         env['MAILMAN_EXTRA_TESTING_CFG'] = test_cfg
...     proc = subprocess.Popen(
...         [exe, 'lists', '--domain', 'ignore', '-q'],
...         cwd=ConfigLayer.root_directory, env=env,
...         universal_newlines=True,
...         stdout=subprocess.PIPE, stderr=subprocess.PIPE)
...     stdout, stderr = proc.communicate()
...     assert proc.returncode == 0, stderr
...     print(stdout)

>>> call()
pre-hook: 1


>>> os.remove(config_path)

Post-hook¶

We can set the post-hook in the configuration file.

>>> with open(config_path, 'w') as fp:
...     print("""\
... [meta]
... extends: test.cfg
...
... [mailman]
... post_hook: hooks.post_hook
... """, file=fp)

>>> call()
post-hook: 1


>>> os.remove(config_path)

Running both hooks¶

We can set the pre- and post-hooks in the configuration file.

>>> with open(config_path, 'w') as fp:
...     print("""\
... [meta]
... extends: test.cfg
...
... [mailman]
... pre_hook: hooks.pre_hook
... post_hook: hooks.post_hook
... """, file=fp)

>>> call()
pre-hook: 1
post-hook: 2

Application level list life cycle¶

The low-level way to create and delete a mailing list is to use the IListManager interface. This interface simply adds or removes the appropriate database entries to record the list’s creation.

There is a higher level interface for creating and deleting mailing lists which performs additional tasks such as:

validating the list’s posting address (which also serves as the list’s fully qualified name);

ensuring that the list’s domain is registered;

applying a list style to the new list;

creating and assigning list owners;

notifying watchers of list creation;

creating ancillary artifacts (such as the list’s on-disk directory)

Creating a list with owners¶

You can also specify a list of owner email addresses. If these addresses are not yet known, they will be registered, and new users will be linked to them.

>>> owners = [
...     'aperson@example.com',
...     'bperson@example.com',
...     'cperson@example.com',
...     'dperson@example.com',
...     ]

>>> ant = create_list('ant@example.com', owners)
>>> dump_list(address.email for address in ant.owners.addresses)
aperson@example.com
bperson@example.com
cperson@example.com
dperson@example.com

None of the owner addresses are verified.

>>> any(address.verified_on is not None
...     for address in ant.owners.addresses)
False

However, all addresses are linked to users.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> user_manager = getUtility(IUserManager)
>>> for address in owners:
...     user = user_manager.get_user(address)
...     print(int(user.user_id.int), list(user.addresses)[0])
1 aperson@example.com
2 bperson@example.com
3 cperson@example.com
4 dperson@example.com

If you create a mailing list with owner addresses that are already known to the system, they won’t be created again.

>>> bee = create_list('bee@example.com', owners)
>>> from operator import attrgetter
>>> for user in sorted(bee.owners.users, key=attrgetter('user_id')):
...     print(int(user.user_id.int), list(user.addresses)[0])
1 aperson@example.com
2 bperson@example.com
3 cperson@example.com
4 dperson@example.com

Deleting a list¶

Removing a mailing list deletes the list, all its subscribers, and any related artifacts.

>>> from mailman.app.lifecycle import remove_list
>>> remove_list(bee)

>>> from mailman.interfaces.listmanager import IListManager
>>> print(getUtility(IListManager).get('bee@example.com'))
None

We should now be able to completely recreate the mailing list.

>>> buzz = create_list('bee@example.com', owners)
>>> dump_list(address.email for address in bee.owners.addresses)
aperson@example.com
bperson@example.com
cperson@example.com
dperson@example.com

Messages¶

Mailman has its own Message classes, derived from the standard email.message.Message class, but providing additional useful methods.

User notifications¶

When Mailman needs to send a message to a user, it creates a UserNotification instance, and then calls the .send() method on this object. This method requires a mailing list instance.

>>> mlist = create_list('test@example.com')

The UserNotification constructor takes the recipient address, the sender address, an optional subject, optional body text, and optional language.

>>> from mailman.email.message import UserNotification
>>> msg = UserNotification(
...     'aperson@example.com',
...     'test@example.com',
...     'Something you need to know',
...     'I needed to tell you this.')
>>> msg.send(mlist)

The message will end up in the virgin queue.

>>> from mailman.testing.helpers import get_queue_messages
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Subject: Something you need to know
From: test@example.com
To: aperson@example.com
Message-ID: ...
Date: ...
Precedence: bulk
<BLANKLINE>
I needed to tell you this.

The message above got a Precedence: bulk header added by default. If the message we’re sending already has a Precedence: header, it shouldn’t be changed.

>>> del msg['precedence']
>>> msg['Precedence'] = 'list'
>>> msg.send(mlist)

Again, the message will end up in the virgin queue but with the original Precedence: header.

>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg['precedence'])
list

Sometimes we want to send the message without a Precedence: header such as when we send a probe message.

>>> del msg['precedence']
>>> msg.send(mlist, add_precedence=False)

Again, the message will end up in the virgin queue but without the Precedence: header.

>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg['precedence'])
None

However, if the message already has a Precedence: header, setting the precedence=False argument will have no effect.

>>> msg['Precedence'] = 'junk'
>>> msg.send(mlist, add_precedence=False)
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg['precedence'])
junk

Application level moderation¶

At an application level, moderation involves holding messages and membership changes for moderator approval. This utilizes the lower level interface for list-centric moderation requests.

Moderation is always mailing list-centric.

>>> mlist = create_list('ant@example.com')
>>> mlist.preferred_language = 'en'
>>> mlist.display_name = 'A Test List'
>>> mlist.admin_immed_notify = False

We’ll use the lower level API for diagnostic purposes.

>>> from mailman.interfaces.requests import IListRequests
>>> requests = IListRequests(mlist)

Message moderation¶

Holding messages¶

Anne posts a message to the mailing list, but she is not a member of the list, so the message is held for moderator approval.

>>> msg = message_from_string("""\
... From: anne@example.org
... To: ant@example.com
... Subject: Something important
... Message-ID: <aardvark>
...
... Here's something important about our mailing list.
... """)

Holding a message means keeping a copy of it that a moderator must approve before the message is posted to the mailing list. To hold the message, the message, its metadata, and a reason for the hold must be provided. In this case, we won’t include any additional metadata.

>>> from mailman.app.moderator import hold_message
>>> hold_message(mlist, msg, {}, 'Needs approval')
1

We can also hold a message with some additional metadata.

>>> msg = message_from_string("""\
... From: bart@example.org
... To: ant@example.com
... Subject: Something important
... Message-ID: <badger>
...
... Here's something important about our mailing list.
... """)
>>> msgdata = dict(sender='anne@example.com', approved=True)

>>> hold_message(mlist, msg, msgdata, 'Feeling ornery')
2

Disposing of messages¶

The moderator can select one of several dispositions:

discard - throw the message away.

reject - bounces the message back to the original author.

defer - defer any action on the message (continue to hold it)

accept - accept the message for posting.

The most trivial is to simply defer a decision for now.

>>> from mailman.interfaces.action import Action
>>> from mailman.app.moderator import handle_message
>>> handle_message(mlist, 1, Action.defer)

This leaves the message in the requests database.

>>> key, data = requests.get_request(1)
>>> print(key)
<aardvark>

The moderator can also discard the message.

>>> handle_message(mlist, 1, Action.discard)
>>> print(requests.get_request(1))
None

The message can be rejected, which bounces the message back to the original sender.

>>> handle_message(mlist, 2, Action.reject, 'Off topic')

The message is no longer available in the requests database.

>>> print(requests.get_request(2))
None

And there is one message in the virgin queue - the rejection notice.

>>> from mailman.testing.helpers import get_queue_messages
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
...
Subject: Request to mailing list "A Test List" rejected
From: ant-bounces@example.com
To: bart@example.org
...
<BLANKLINE>
Your request to the ant@example.com mailing list
<BLANKLINE>
    Posting of your message titled "Something important"
<BLANKLINE>
has been rejected by the list moderator.  The moderator gave the
following reason for rejecting your request:
<BLANKLINE>
"Off topic"
<BLANKLINE>
Any questions or comments should be directed to the list administrator
at:
<BLANKLINE>
    ant-owner@example.com
<BLANKLINE>

The bounce gets sent to the original sender.

>>> for recipient in sorted(messages[0].msgdata['recipients']):
...     print(recipient)
bart@example.org

Or the message can be approved.

>>> msg = message_from_string("""\
... From: cris@example.org
... To: ant@example.com
... Subject: Something important
... Message-ID: <caribou>
...
... Here's something important about our mailing list.
... """)
>>> id = hold_message(mlist, msg, {}, 'Needs approval')
>>> handle_message(mlist, id, Action.accept)

This places the message back into the incoming queue for further processing, however the message metadata indicates that the message has been approved.

>>> messages = get_queue_messages('pipeline')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
From: cris@example.org
To: ant@example.com
Subject: Something important
...

>>> dump_msgdata(messages[0].msgdata)
_parsemsg         : False
approved          : True
moderator_approved: True
version           : 3

Forwarding the message¶

The message can be forwarded to another address. This is helpful for getting the message into the inbox of one of the moderators.

>>> msg = message_from_string("""\
... From: elly@example.org
... To: ant@example.com
... Subject: Something important
... Message-ID: <elephant>
...
... Here's something important about our mailing list.
... """)
>>> req_id = hold_message(mlist, msg, {}, 'Needs approval')
>>> handle_message(mlist, req_id, Action.discard,
...                forward=['zack@example.com'])

The forwarded message is in the virgin queue, destined for the moderator.

>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
Subject: Forward of moderated message
From: ant-bounces@example.com
To: zack@example.com
...

>>> for recipient in sorted(messages[0].msgdata['recipients']):
...     print(recipient)
zack@example.com

Holding subscription requests¶

For closed lists, subscription requests will also be held for moderator approval. In this case, several pieces of information related to the subscription must be provided, including the subscriber’s address and real name, what kind of delivery option they are choosing and their preferred language.

>>> from mailman.app.moderator import hold_subscription
>>> from mailman.interfaces.member import DeliveryMode
>>> from mailman.interfaces.subscriptions import RequestRecord
>>> req_id = hold_subscription(
...     mlist,
...     RequestRecord('fred@example.org', 'Fred Person',
...                   DeliveryMode.regular, 'en'))

Disposing of membership change requests¶

Just as with held messages, the moderator can select one of several dispositions for this membership change request. The most trivial is to simply defer a decision for now.

>>> from mailman.app.moderator import handle_subscription
>>> handle_subscription(mlist, req_id, Action.defer)
>>> requests.get_request(req_id) is not None
True

The held subscription can also be discarded.

>>> handle_subscription(mlist, req_id, Action.discard)
>>> print(requests.get_request(req_id))
None

Gwen tries to subscribe to the mailing list, but…

>>> req_id = hold_subscription(
...     mlist,
...     RequestRecord('gwen@example.org', 'Gwen Person',
...                   DeliveryMode.regular, 'en'))

…her request is rejected…

>>> handle_subscription(
...     mlist, req_id, Action.reject, 'This is a closed list')
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

…and she receives a rejection notice.

>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
...
Subject: Request to mailing list "A Test List" rejected
From: ant-bounces@example.com
To: gwen@example.org
...
Your request to the ant@example.com mailing list
<BLANKLINE>
    Subscription request
<BLANKLINE>
has been rejected by the list moderator.  The moderator gave the
following reason for rejecting your request:
<BLANKLINE>
"This is a closed list"
...

The subscription can also be accepted. This subscribes the address to the mailing list.

>>> mlist.send_welcome_message = False
>>> req_id = hold_subscription(
...     mlist,
...     RequestRecord('herb@example.org', 'Herb Person',
...                   DeliveryMode.regular, 'en'))

The moderators accept the subscription request.

>>> handle_subscription(mlist, req_id, Action.accept)

And now Herb is a member of the mailing list.

>>> print(mlist.members.get_member('herb@example.org').address)
Herb Person <herb@example.org>

Holding unsubscription requests¶

Some lists require moderator approval for unsubscriptions. In this case, only the unsubscribing address is required.

Herb now wants to leave the mailing list, but his request must be approved.

>>> from mailman.app.moderator import hold_unsubscription
>>> req_id = hold_unsubscription(mlist, 'herb@example.org')

As with subscription requests, the unsubscription request can be deferred.

>>> from mailman.app.moderator import handle_unsubscription
>>> handle_unsubscription(mlist, req_id, Action.defer)
>>> print(mlist.members.get_member('herb@example.org').address)
Herb Person <herb@example.org>

The held unsubscription can also be discarded, and the member will remain subscribed.

>>> handle_unsubscription(mlist, req_id, Action.discard)
>>> print(mlist.members.get_member('herb@example.org').address)
Herb Person <herb@example.org>

The request can be rejected, in which case a message is sent to the member, and the person remains a member of the mailing list.

>>> req_id = hold_unsubscription(mlist, 'herb@example.org')
>>> handle_unsubscription(mlist, req_id, Action.reject, 'No can do')
>>> print(mlist.members.get_member('herb@example.org').address)
Herb Person <herb@example.org>

Herb gets a rejection notice.

>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
...
Subject: Request to mailing list "A Test List" rejected
From: ant-bounces@example.com
To: herb@example.org
...
Your request to the ant@example.com mailing list

    Unsubscription request

has been rejected by the list moderator.  The moderator gave the
following reason for rejecting your request:

"No can do"
...

The unsubscription request can also be accepted. This removes the member from the mailing list.

>>> req_id = hold_unsubscription(mlist, 'herb@example.org')
>>> mlist.send_goodbye_message = False
>>> handle_unsubscription(mlist, req_id, Action.accept)
>>> print(mlist.members.get_member('herb@example.org'))
None

Notifications¶

Membership change requests¶

Usually, the list administrators want to be notified when there are membership change requests they need to moderate. These notifications are sent when the list is configured to send them.

>>> mlist.admin_immed_notify = True

Iris tries to subscribe to the mailing list.

>>> req_id = hold_subscription(mlist,
...     RequestRecord('iris@example.org', 'Iris Person',
...                   DeliveryMode.regular, 'en'))

There’s now a message in the virgin queue, destined for the list owner.

>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
...
Subject: New subscription request to A Test List from iris@example.org
From: ant-owner@example.com
To: ant-owner@example.com
...
Your authorization is required for a mailing list subscription request
approval:
<BLANKLINE>
    For:  iris@example.org
    List: ant@example.com

Similarly, the administrator gets notifications on unsubscription requests. Jeff is a member of the mailing list, and chooses to unsubscribe.

>>> unsub_req_id = hold_unsubscription(mlist, 'jeff@example.org')
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
...
Subject: New unsubscription request from A Test List by jeff@example.org
From: ant-owner@example.com
To: ant-owner@example.com
...
Your authorization is required for a mailing list unsubscription
request approval:
<BLANKLINE>
    By:   jeff@example.org
    From: ant@example.com
...

Membership changes¶

When a new member request is accepted, the mailing list administrators can receive a membership change notice.

>>> mlist.admin_notify_mchanges = True
>>> mlist.admin_immed_notify = False
>>> handle_subscription(mlist, req_id, Action.accept)
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
...
Subject: A Test List subscription notification
From: noreply@example.com
To: ant-owner@example.com
...
Iris Person <iris@example.org> has been successfully subscribed to A
Test List.

Similarly when an unsubscription request is accepted, the administrators can get a notification.

>>> req_id = hold_unsubscription(mlist, 'iris@example.org')
>>> handle_unsubscription(mlist, req_id, Action.accept)
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
...
Subject: A Test List unsubscription notification
From: noreply@example.com
To: ant-owner@example.com
...
Iris Person <iris@example.org> has been removed from A Test List.

Welcome messages¶

When a member is subscribed to the mailing list via moderator approval, she can get a welcome message.

>>> mlist.admin_notify_mchanges = False
>>> mlist.send_welcome_message = True
>>> req_id = hold_subscription(mlist,
...     RequestRecord('kate@example.org', 'Kate Person',
...                   DeliveryMode.regular, 'en'))
>>> handle_subscription(mlist, req_id, Action.accept)
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
...
Subject: Welcome to the "A Test List" mailing list
From: ant-request@example.com
To: Kate Person <kate@example.org>
...
Welcome to the "A Test List" mailing list!
...

Goodbye messages¶

Similarly, when the member’s unsubscription request is approved, she’ll get a goodbye message.

>>> mlist.send_goodbye_message = True
>>> req_id = hold_unsubscription(mlist, 'kate@example.org')
>>> handle_unsubscription(mlist, req_id, Action.accept)
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1
>>> print(messages[0].msg.as_string())
MIME-Version: 1.0
...
Subject: You have been unsubscribed from the A Test List mailing list
From: ant-bounces@example.com
To: kate@example.org
...

Pipelines¶

Pipelines process messages that have been accepted for posting, applying any modifications and also sending copies of the message to the archives, digests, NNTP, and outgoing queues. Pipelines are named and consist of a sequence of handlers, each of which is applied in turn. Unlike rules and chains, there is no way to stop a pipeline from processing the message once it’s started.

>>> mlist = create_list('test@example.com')
>>> print(mlist.posting_pipeline)
default-posting-pipeline
>>> from mailman.core.pipelines import process

For the purposes of these examples, we’ll enable just one archiver.

>>> from mailman.interfaces.mailinglist import IListArchiverSet
>>> for archiver in IListArchiverSet(mlist).archivers:
...     archiver.is_enabled = (archiver.name == 'mhonarc')

Processing a message¶

Messages hit the pipeline after they’ve been accepted for posting.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: My first post
... Message-ID: <first>
... X-Message-ID-Hash: 4CMWUN6BHVCMHMDAOSJZ2Q72G5M32MWB
...
... First post!
... """)
>>> msgdata = {}
>>> process(mlist, msg, msgdata, mlist.posting_pipeline)

The message has been modified with additional headers, footer decorations, etc.

>>> print(msg.as_string())
From: aperson@example.com
To: test@example.com
Message-ID: <first>
X-Message-ID-Hash: 4CMWUN6BHVCMHMDAOSJZ2Q72G5M32MWB
X-Mailman-Version: ...
Precedence: list
Subject: [Test] My first post
List-Id: <test.example.com>
Archived-At: <http://lists.example.com/.../4CMWUN6BHVCMHMDAOSJZ2Q72G5M32MWB>
List-Archive: <http://lists.example.com/archives/test@example.com>
List-Help: <mailto:test-request@example.com?subject=help>
List-Post: <mailto:test@example.com>
List-Subscribe: <http://lists.example.com/listinfo/test@example.com>,
 <mailto:test-join@example.com>
List-Unsubscribe: <http://lists.example.com/listinfo/test@example.com>,
 <mailto:test-leave@example.com>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
<BLANKLINE>
First post!
_______________________________________________
Test mailing list
test@example.com
http://lists.example.com/listinfo/test@example.com
<BLANKLINE>

The message metadata has information about recipients and other stuff. However there are currently no recipients for this message.

>>> dump_msgdata(msgdata)
original_sender : aperson@example.com
original_subject: My first post
recipients      : set()
stripped_subject: My first post

After pipeline processing, the message is now sitting in various other processing queues.

>>> from mailman.testing.helpers import get_queue_messages
>>> messages = get_queue_messages('archive')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
From: aperson@example.com
To: test@example.com
Message-ID: <first>
X-Message-ID-Hash: 4CMWUN6BHVCMHMDAOSJZ2Q72G5M32MWB
X-Mailman-Version: ...
Precedence: list
Subject: [Test] My first post
List-Id: <test.example.com>
...

First post!


>>> dump_msgdata(messages[0].msgdata)
_parsemsg       : False
original_sender : aperson@example.com
original_subject: My first post
recipients      : set()
stripped_subject: My first post
version         : 3

This mailing list is not linked to an NNTP newsgroup, so there’s nothing in the outgoing nntp queue.

>>> messages = get_queue_messages('nntp')
>>> len(messages)
0

The outgoing queue will hold the copy of the message that will actually get delivered to end recipients.

>>> messages = get_queue_messages('out')
>>> len(messages)
1

>>> print(messages[0].msg.as_string())
From: aperson@example.com
To: test@example.com
Message-ID: <first>
X-Message-ID-Hash: 4CMWUN6BHVCMHMDAOSJZ2Q72G5M32MWB
X-Mailman-Version: ...
Precedence: list
Subject: [Test] My first post
List-Id: <test.example.com>
...

First post!

_______________________________________________
Test mailing list
test@example.com
http://lists.example.com/listinfo/test@example.com

>>> dump_msgdata(messages[0].msgdata)
_parsemsg       : False
listid          : test.example.com
original_sender : aperson@example.com
original_subject: My first post
recipients      : set()
stripped_subject: My first post
version         : 3

There’s now one message in the digest mailbox, getting ready to be sent.

>>> from mailman.testing.helpers import digest_mbox
>>> digest = digest_mbox(mlist)
>>> sum(1 for mboxmsg in digest)
1

>>> print(list(digest)[0].as_string())
From: aperson@example.com
To: test@example.com
Message-ID: <first>
X-Message-ID-Hash: 4CMWUN6BHVCMHMDAOSJZ2Q72G5M32MWB
X-Mailman-Version: ...
Precedence: list
Subject: [Test] My first post
List-Id: <test.example.com>
...

First post!

Subscription services¶

The ISubscriptionService utility provides higher level convenience methods useful for searching, retrieving, iterating, and removing memberships across all mailing lists on th esystem. Adding new users is handled by the IRegistrar interface.

>>> from mailman.interfaces.subscriptions import ISubscriptionService
>>> from zope.component import getUtility
>>> service = getUtility(ISubscriptionService)

You can use the service to get all members of all mailing lists, for any membership role. At first, there are no memberships.

>>> service.get_members()
[]
>>> sum(1 for member in service)
0
>>> from uuid import UUID
>>> print(service.get_member(UUID(int=801)))
None

Listing members¶

When there are some members, of any role on any mailing list, they can be retrieved through the subscription service.

>>> from mailman.app.lifecycle import create_list
>>> ant = create_list('ant@example.com')
>>> bee = create_list('bee@example.com')
>>> cat = create_list('cat@example.com')

Some people become members.

>>> from mailman.interfaces.member import MemberRole
>>> from mailman.testing.helpers import subscribe
>>> anne_1 = subscribe(ant, 'Anne')
>>> anne_2 = subscribe(ant, 'Anne', MemberRole.owner)
>>> bart_1 = subscribe(ant, 'Bart', MemberRole.moderator)
>>> bart_2 = subscribe(bee, 'Bart', MemberRole.owner)
>>> anne_3 = subscribe(cat, 'Anne', email='anne@example.com')
>>> cris_1 = subscribe(cat, 'Cris')

The service can be used to iterate over them.

>>> for member in service.get_members():
...     print(member)
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.owner>
<Member: Bart Person <bperson@example.com>
    on ant@example.com as MemberRole.moderator>
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.member>
<Member: Bart Person <bperson@example.com>
    on bee@example.com as MemberRole.owner>
<Member: Anne Person <anne@example.com>
    on cat@example.com as MemberRole.member>
<Member: Cris Person <cperson@example.com>
    on cat@example.com as MemberRole.member>

The service can also be used to get the information about a single member.

>>> print(service.get_member(bart_2.member_id))
<Member: Bart Person <bperson@example.com>
    on bee@example.com as MemberRole.owner>

There is an iteration shorthand for getting all the members.

>>> for member in service:
...     print(member)
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.owner>
<Member: Bart Person <bperson@example.com>
    on ant@example.com as MemberRole.moderator>
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.member>
<Member: Bart Person <bperson@example.com>
    on bee@example.com as MemberRole.owner>
<Member: Anne Person <anne@example.com>
    on cat@example.com as MemberRole.member>
<Member: Cris Person <cperson@example.com>
    on cat@example.com as MemberRole.member>

Finding members¶

The subscription service can be used to find memberships based on specific search criteria. For example, we can find all the mailing lists that Anne is a member of with her aperson@example.com address.

>>> for member in service.find_members('aperson@example.com'):
...     print(member)
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.member>
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.owner>

There may be no matching memberships.

>>> service.find_members('dave@example.com')
[]

Memberships can also be searched for by user id.

>>> for member in service.find_members(anne_1.user.user_id):
...     print(member)
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.member>
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.owner>

You can find all the memberships for a specific mailing list.

>>> for member in service.find_members(list_id='ant.example.com'):
...     print(member)
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.member>
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.owner>
<Member: Bart Person <bperson@example.com>
    on ant@example.com as MemberRole.moderator>

You can find all the memberships for an address on a specific mailing list, but you have to give it the list id, not the fqdn listname since the former is stable but the latter could change if the list is moved.

>>> for member in service.find_members(
...         'bperson@example.com', 'ant.example.com'):
...     print(member)
<Member: Bart Person <bperson@example.com>
    on ant@example.com as MemberRole.moderator>

You can find all the memberships for an address with a specific role.

>>> for member in service.find_members(
...         list_id='ant.example.com', role=MemberRole.owner):
...     print(member)
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.owner>

You can also find a specific membership by all three criteria.

>>> for member in service.find_members(
...         'bperson@example.com', 'bee.example.com', MemberRole.owner):
...     print(member)
<Member: Bart Person <bperson@example.com>
    on bee@example.com as MemberRole.owner>

Removing members¶

Members can be removed via this service.

>>> len(service.get_members())
6
>>> service.leave('cat.example.com', 'cperson@example.com')
>>> len(service.get_members())
5
>>> for member in service:
...     print(member)
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.owner>
<Member: Bart Person <bperson@example.com>
    on ant@example.com as MemberRole.moderator>
<Member: Anne Person <aperson@example.com>
    on ant@example.com as MemberRole.member>
<Member: Bart Person <bperson@example.com>
    on bee@example.com as MemberRole.owner>
<Member: Anne Person <anne@example.com>
    on cat@example.com as MemberRole.member>

System versions¶

Mailman system information is available through the system object, which implements the ISystem interface.

>>> from mailman.interfaces.system import ISystem
>>> from mailman.core.system import system
>>> from zope.interface.verify import verifyObject

>>> verifyObject(ISystem, system)
True

The Mailman version is also available via the system object.

>>> print(system.mailman_version)
GNU Mailman ...

The Python version running underneath is also available via the system object.

# The entire python_version string is variable, so this is the best test
# we can do.
>>> import sys
>>> system.python_version == sys.version
True

List styles¶

List styles are a way to name and apply a template of attribute settings to new mailing lists. Every style has a name, which must be unique.

Styles are generally only applied when a mailing list is created, although there is no reason why styles can’t be applied to an existing mailing list. However, when a style changes, the mailing lists using that style are not automatically updated. Instead, think of styles as the initial set of defaults for just about any mailing list attribute. In fact, application of a style to a mailing list can really modify the mailing list in any way.

To start with, there are a few legacy styles.

>>> from zope.component import getUtility
>>> from mailman.interfaces.styles import IStyleManager
>>> manager = getUtility(IStyleManager)
>>> for style in manager.styles:
...     print(style.name)
legacy-announce
legacy-default

When you create a mailing list through the low-level IListManager API, no style is applied.

>>> from mailman.interfaces.listmanager import IListManager
>>> mlist = getUtility(IListManager).create('ant@example.com')
>>> print(mlist.display_name)
None

The legacy default style sets the list’s display name.

>>> manager.get('legacy-default').apply(mlist)
>>> print(mlist.display_name)
Ant

Registering styles¶

New styles must implement the IStyle interface.

>>> from zope.interface import implementer
>>> from mailman.interfaces.styles import IStyle
>>> @implementer(IStyle)
... class TestStyle:
...     name = 'a-test-style'
...     def apply(self, mailing_list):
...         # Just does something very simple.
...         mailing_list.display_name = 'TEST STYLE LIST'

You can register a new style with the style manager.

>>> manager.register(TestStyle())

All registered styles are returned in alphabetical order by style name.

>>> for style in manager.styles:
...     print(style.name)
a-test-style
legacy-announce
legacy-default

You can also ask the style manager for the style, by name.

>>> test_style = manager.get('a-test-style')
>>> print(test_style.name)
a-test-style

Unregistering styles¶

You can unregister a style, making it unavailable in the future.

>>> manager.unregister(test_style)
>>> for style in manager.styles:
...     print(style.name)
legacy-announce
legacy-default

Asking for a missing style returns None.

>>> print(manager.get('a-test-style'))
None

Apply styles at list creation¶

You can specify a style to apply when creating a list through the high-level API. Let’s start by registering the test style.

>>> manager.register(test_style)

Now, when we use the high level API, we can ask for the style to be applied.

>>> from mailman.app.lifecycle import create_list
>>> mlist = create_list('bee@example.com', style_name=test_style.name)

The style has been applied.

>>> print(mlist.display_name)
TEST STYLE LIST

If no style name is provided when creating the list, the system default style (taken from the configuration file) is applied.

>>> @implementer(IStyle)
... class AnotherStyle:
...     name = 'another-style'
...     def apply(self, mailing_list):
...         # Just does something very simple.
...         mailing_list.display_name = 'ANOTHER STYLE LIST'
>>> another_style = AnotherStyle()

We’ll set up the system default to apply this newly registered style if no other style is explicitly given.

>>> from mailman.testing.helpers import configuration
>>> with configuration('styles', default=another_style.name):
...     manager.register(another_style)
...     mlist = create_list('cat@example.com')
>>> print(mlist.display_name)
ANOTHER STYLE LIST

Archivers¶

Mailman supports pluggable archivers, and it comes with several default archivers.

>>> mlist = create_list('test@example.com')
>>> msg = message_from_string("""\
... From: aperson@example.org
... To: test@example.com
... Subject: An archived message
... Message-ID: <12345>
... X-Message-ID-Hash: RSZCG7IGPHFIRW3EMTVMMDNJMNCVCOLE
...
... Here is an archived message.
... """)

Archivers support an interface which provides the RFC 2369 List-Archive: header, and one that provides a permalink to the specific message object in the archive. This latter is appropriate for the message footer or for the RFC 5064 Archived-At: header.

If the archiver is not network-accessible, it will return None and the headers will not be added.

Mailman defines a draft spec for how list servers and archivers can interoperate.

>>> archivers = {}
>>> from operator import attrgetter
>>> for archiver in sorted(config.archivers, key=attrgetter('name')):
...     print(archiver.name)
...     print('   ', archiver.list_url(mlist))
...     print('   ', archiver.permalink(mlist, msg))
...     archivers[archiver.name] = archiver
mail-archive
    http://go.mail-archive.dev/test%40example.com
    http://go.mail-archive.dev/RSZCG7IGPHFIRW3EMTVMMDNJMNCVCOLE
mhonarc
    http://lists.example.com/.../test@example.com
    http://lists.example.com/.../RSZCG7IGPHFIRW3EMTVMMDNJMNCVCOLE
prototype
    None
    None

Sending the message to the archiver¶

The prototype archiver archives messages to a maildir.

>>> import os
>>> archivers['prototype'].archive_message(mlist, msg)
>>> archive_path = os.path.join(
...     config.ARCHIVE_DIR, 'prototype', mlist.fqdn_listname, 'new')
>>> len(os.listdir(archive_path))
1

The Mail-Archive.com¶

The Mail Archive is a public archiver that can be used to archive message for free. Mailman comes with a plugin for this archiver; by enabling it messages to public lists will get sent there automatically.

>>> archiver = archivers['mail-archive']
>>> print(archiver.list_url(mlist))
http://go.mail-archive.dev/test%40example.com
>>> print(archiver.permalink(mlist, msg))
http://go.mail-archive.dev/RSZCG7IGPHFIRW3EMTVMMDNJMNCVCOLE

To archive the message, the archiver actually mails the message to a special address at The Mail Archive. The message gets no header or footer decoration.

>>> from mailman.interfaces.archiver import ArchivePolicy
>>> mlist.archive_policy = ArchivePolicy.public
>>> archiver.archive_message(mlist, msg)

>>> from mailman.runners.outgoing import OutgoingRunner
>>> from mailman.testing.helpers import make_testable_runner
>>> outgoing = make_testable_runner(OutgoingRunner, 'out')
>>> outgoing.run()

>>> from operator import itemgetter
>>> messages = list(smtpd.messages)
>>> len(messages)
1

>>> print(messages[0].as_string())
From: aperson@example.org
To: test@example.com
Subject: An archived message
Message-ID: <12345>
X-Message-ID-Hash: RSZCG7IGPHFIRW3EMTVMMDNJMNCVCOLE
X-Peer: 127.0.0.1:...
X-MailFrom: test-bounces@example.com
X-RcptTo: archive@mail-archive.dev

Here is an archived message.

>>> smtpd.clear()

However, if the mailing list is not public, the message will never be archived at this service.

>>> mlist.archive_policy = ArchivePolicy.private
>>> print(archiver.list_url(mlist))
None
>>> print(archiver.permalink(mlist, msg))
None
>>> archiver.archive_message(mlist, msg)
>>> list(smtpd.messages)
[]

Additionally, this archiver can handle malformed Message-IDs.

>>> from mailman.utilities.email import add_message_hash
>>> mlist.archive_policy = ArchivePolicy.public
>>> del msg['message-id']
>>> del msg['x-message-id-hash']
>>> msg['Message-ID'] = '12345>'
>>> add_message_hash(msg)
>>> print(archiver.permalink(mlist, msg))
http://go.mail-archive.dev/YJIGBYRWZFG5LZEBQ7NR25B5HBR2BVD6

>>> del msg['message-id']
>>> del msg['x-message-id-hash']
>>> msg['Message-ID'] = '<12345'
>>> add_message_hash(msg)
>>> print(archiver.permalink(mlist, msg))
http://go.mail-archive.dev/XUFFJNJ2P2WC4NDPQRZFDJMV24POP64B

>>> del msg['message-id']
>>> del msg['x-message-id-hash']
>>> msg['Message-ID'] = '12345'
>>> add_message_hash(msg)
>>> print(archiver.permalink(mlist, msg))
http://go.mail-archive.dev/RSZCG7IGPHFIRW3EMTVMMDNJMNCVCOLE

>>> del msg['message-id']
>>> del msg['x-message-id-hash']
>>> add_message_hash(msg)
>>> msg['Message-ID'] = '    12345    '
>>> add_message_hash(msg)
>>> print(archiver.permalink(mlist, msg))
http://go.mail-archive.dev/RSZCG7IGPHFIRW3EMTVMMDNJMNCVCOLE

MHonArc¶

A MHonArc archiver is also available.

>>> archiver = archivers['mhonarc']
>>> print(archiver.name)
mhonarc

Messages sent to a local MHonArc instance are added to its archive via a subprocess call.

>>> from mailman.testing.helpers import LogFileMark
>>> mark = LogFileMark('mailman.archiver')
>>> archiver.archive_message(mlist, msg)
>>> print('LOG:', mark.readline())
LOG: ... /usr/bin/mhonarc
     -add
     -dbfile .../test@example.com.mbox/mhonarc.db
     -outdir .../mhonarc/test@example.com
     -stderr .../logs/mhonarc
     -stdout .../logs/mhonarc -spammode -umask 022

SMTP authentication¶

The SMTP server may require authentication. Mailman supports setting the SMTP user name and password. The actual authentication mechanism used is determined by Python’s smtplib module, which tries the more secure CRAM-MD5 method first, followed by the less secure mechanisms PLAIN and LOGIN.

When sending authentication data between Mailman and the MTA over an unsecured network, the submission (mail) server should offer CRAM-MD5 as mechanism to have Python’s smtplib module automatically choose the more secure mechanism.

When the user name and password match what’s expected by the server, everything is a-okay.

>>> mlist = create_list('test@example.com')

By default there is no user name and password, but this matches what’s expected by the test server.

>>> config.push('auth', """
... [mta]
... smtp_user: testuser
... smtp_pass: testpass
... """)

Attempting delivery first must authorize with the mail server.

>>> from mailman.mta.bulk import BulkDelivery
>>> bulk = BulkDelivery()

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: My first post
... Message-ID: <first>
...
... First post!
... """)

>>> bulk.deliver(mlist, msg, dict(recipients=['bperson@example.com']))
{}

>>> print(smtpd.get_authentication_credentials())
PLAIN AHRlc3R1c2VyAHRlc3RwYXNz
>>> config.pop('auth')

But if the user name and password does not match, the connection will fail.

>>> config.push('auth', """
... [mta]
... smtp_user: baduser
... smtp_pass: badpass
... """)

>>> bulk = BulkDelivery()
>>> response = bulk.deliver(
...     mlist, msg, dict(recipients=['bperson@example.com']))
>>> dump_msgdata(response)
bperson@example.com: (571, b'Bad authentication')

>>> config.pop('auth')

Standard bulk delivery¶

Mailman has several built in strategies for completing the actual delivery of messages to the immediate upstream mail transport agent, which completes the actual final delivery to recipients.

Bulk delivery attempts to deliver as few copies of the identical message as possible to as many recipients as possible. By grouping recipients this way, bandwidth between Mailman and the MTA, and consequently between the MTA and remote mail servers, can be greatly reduced. The downside is the messages cannot be personalized. See verp.txt for an alternative strategy.

>>> from mailman.mta.bulk import BulkDelivery

The standard bulk deliverer takes as an argument the maximum number of recipients per session. The default is to deliver the message in one chunk, containing all recipients.

>>> bulk = BulkDelivery()

Delivery strategies must implement the proper interface.

>>> from mailman.interfaces.mta import IMailTransportAgentDelivery
>>> from zope.interface.verify import verifyObject
>>> verifyObject(IMailTransportAgentDelivery, bulk)
True

Chunking recipients¶

The set of final recipients is contained in the recipients key in the message metadata. When max_recipients is specified as zero, then the bulk deliverer puts all recipients into one big chunk.

>>> from string import ascii_letters
>>> recipients = set(letter + 'person@example.com'
...                  for letter in ascii_letters)

>>> chunks = list(bulk.chunkify(recipients))
>>> len(chunks)
1
>>> len(chunks[0])
52

Let say the maximum number of recipients allowed is 4, then no chunk will have more than 4 recipients, though they can have fewer (but still not zero).

>>> bulk = BulkDelivery(4)
>>> chunks = list(bulk.chunkify(recipients))
>>> len(chunks)
13
>>> all(0 < len(chunk) <= 4 for chunk in chunks)
True

The chunking algorithm sorts recipients by top level domain by length.

>>> recipients = set([
...     'anne@example.com',
...     'bart@example.org',
...     'cate@example.net',
...     'dave@example.com',
...     'elle@example.org',
...     'fred@example.net',
...     'gwen@example.com',
...     'herb@example.us',
...     'ione@example.net',
...     'john@example.com',
...     'kate@example.com',
...     'liam@example.ca',
...     'mary@example.us',
...     'neil@example.net',
...     'ocho@example.org',
...     'paco@example.xx',
...     'quaq@example.zz',
...     ])

>>> bulk = BulkDelivery(4)
>>> chunks = list(bulk.chunkify(recipients))
>>> len(chunks)
6

We can’t make any guarantees about sorting within each chunk, but we can tell a few things. For example, the first two chunks will be composed of .net (4) and .org (3) domains (for a total of 7).

>>> len(chunks[0])
4
>>> len(chunks[1])
3

>>> for address in sorted(chunks[0].union(chunks[1])):
...     print(address)
bart@example.org
cate@example.net
elle@example.org
fred@example.net
ione@example.net
neil@example.net
ocho@example.org

We also know that the next two chunks will contain .com (5) addresses.

>>> len(chunks[2])
4
>>> len(chunks[3])
1

>>> for address in sorted(chunks[2].union(chunks[3])):
...     print(address)
anne@example.com
dave@example.com
gwen@example.com
john@example.com
kate@example.com

The next chunk will contain the .us (2) and .ca (1) domains.

>>> len(chunks[4])
3
>>> for address in sorted(chunks[4]):
...     print(address)
herb@example.us
liam@example.ca
mary@example.us

The final chunk will contain the outliers, .xx (1) and .zz (2).

>>> len(chunks[5])
2
>>> for address in sorted(chunks[5]):
...     print(address)
paco@example.xx
quaq@example.zz

Bulk delivery¶

The set of recipients for bulk delivery comes from the message metadata. If there are no calculated recipients, nothing gets sent.

>>> mlist = create_list('test@example.com')
>>> msg = message_from_string("""\
... From: aperson@example.org
... To: test@example.com
... Subject: test one
... Message-ID: <aardvark>
...
... This is a test.
... """)

>>> bulk = BulkDelivery()
>>> bulk.deliver(mlist, msg, {})
{}
>>> len(list(smtpd.messages))
0

>>> bulk.deliver(mlist, msg, dict(recipients=set()))
{}
>>> len(list(smtpd.messages))
0

With bulk delivery and no maximum number of recipients, there will be just one message sent, with all the recipients packed into the envelope recipients (i.e. RCTP TO).

>>> recipients = set('person_{0:02d}'.format(i) for i in range(100))
>>> msgdata = dict(recipients=recipients)
>>> bulk.deliver(mlist, msg, msgdata)
{}

>>> messages = list(smtpd.messages)
>>> len(messages)
1
>>> print(messages[0].as_string())
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
...
X-RcptTo: person_...
    person_...
...

This is a test.

The X-RcptTo: header contains the set of recipients, in sorted order.

>>> len(messages[0]['x-rcptto'].split(','))
100

When the maximum number of recipients is set to 20, 5 messages will be sent, each with 20 addresses in the RCPT TO.

>>> bulk = BulkDelivery(20)
>>> bulk.deliver(mlist, msg, msgdata)
{}

>>> messages = list(smtpd.messages)
>>> len(messages)
5
>>> for message in messages:
...     x_rcptto = message['x-rcptto']
...     print('Number of recipients:', len(x_rcptto.split(',')))
Number of recipients: 20
Number of recipients: 20
Number of recipients: 20
Number of recipients: 20
Number of recipients: 20

Delivery headers¶

The sending agent shows up in the RFC 5321 MAIL FROM, which shows up in the X-MailFrom: header in the sample message.

The bulk delivery module calculates the sending agent address first from the message metadata…

>>> bulk = BulkDelivery()
>>> recipients = set(['aperson@example.com'])
>>> msgdata = dict(recipients=recipients,
...                sender='asender@example.org')
>>> bulk.deliver(mlist, msg, msgdata)
{}

>>> message = list(smtpd.messages)[0]
>>> print(message.as_string())
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: asender@example.org
X-RcptTo: aperson@example.com

This is a test.

…followed by the mailing list’s bounces robot address…

>>> del msgdata['sender']
>>> bulk.deliver(mlist, msg, msgdata)
{}

>>> message = list(smtpd.messages)[0]
>>> print(message.as_string())
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: aperson@example.com

This is a test.

…and finally the site owner, if there is no mailing list target for this message.

>>> config.push('site-owner', """\
... [mailman]
... site_owner: site-owner@example.com
... """)

>>> bulk.deliver(None, msg, msgdata)
{}

>>> message = list(smtpd.messages)[0]
>>> print(message.as_string())
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: site-owner@example.com
X-RcptTo: aperson@example.com

This is a test.

# Remove test configuration.
>>> config.pop('site-owner')

Delivery failures¶

Mailman does not do final delivery. Instead, it sends mail through a site local mail server which manages queuing and final delivery. However, even this local mail server can produce delivery failures visible to Mailman in certain situations.

For example, there could be a problem delivering to any of the specified recipients.

# Tell the mail server to fail on the next 3 RCPT TO commands, one for
# each recipient in the following message.
>>> smtpd.err_queue.put(('rcpt', 500))
>>> smtpd.err_queue.put(('rcpt', 500))
>>> smtpd.err_queue.put(('rcpt', 500))

>>> recipients = set([
...     'aperson@example.org',
...     'bperson@example.org',
...     'cperson@example.org',
...     ])
>>> msgdata = dict(recipients=recipients)

>>> msg = message_from_string("""\
... From: aperson@example.org
... To: test@example.com
... Subject: test three
... Message-ID: <camel>
...
... This is a test.
... """)

>>> failures = bulk.deliver(mlist, msg, msgdata)
>>> for address in sorted(failures):
...     print(address, failures[address][0],
...                    failures[address][1].decode('ascii'))
aperson@example.org 500 Error: SMTPRecipientsRefused
bperson@example.org 500 Error: SMTPRecipientsRefused
cperson@example.org 500 Error: SMTPRecipientsRefused

>>> messages = list(smtpd.messages)
>>> len(messages)
0

Or there could be some other problem causing an SMTP response failure.

# Tell the mail server to register a temporary failure on the next MAIL
# FROM command.
>>> smtpd.err_queue.put(('mail', 450))

>>> failures = bulk.deliver(mlist, msg, msgdata)
>>> for address in sorted(failures):
...     print(address, failures[address][0],
...                    failures[address][1].decode('ascii'))
aperson@example.org 450 Error: SMTPResponseException
bperson@example.org 450 Error: SMTPResponseException
cperson@example.org 450 Error: SMTPResponseException

# Tell the mail server to register a permanent failure on the next MAIL
# FROM command.
>>> smtpd.err_queue.put(('mail', 500))

>>> failures = bulk.deliver(mlist, msg, msgdata)
>>> for address in sorted(failures):
...     print(address, failures[address][0],
...                    failures[address][1].decode('ascii'))
aperson@example.org 500 Error: SMTPResponseException
bperson@example.org 500 Error: SMTPResponseException
cperson@example.org 500 Error: SMTPResponseException

XXX Untested: socket.error, IOError, smtplib.SMTPException.

MTA connections¶

Outgoing connections to the outgoing mail transport agent (MTA) are mitigated through a Connection class, which can transparently manage multiple sessions in a single connection.

>>> from mailman.mta.connection import Connection

The number of sessions per connections is specified when the Connection object is created, as is the host and port number of the SMTP server. Zero means there’s an unlimited number of sessions per connection.

>>> connection = Connection(
...     config.mta.smtp_host, int(config.mta.smtp_port), 0)

At the start, there have been no connections to the server.

>>> smtpd.get_connection_count()
0

By sending a message to the server, a connection is opened.

>>> connection.sendmail('anne@example.com', ['bart@example.com'], """\
... From: anne@example.com
... To: bart@example.com
... Subject: aardvarks
...
... """)
{}

>>> smtpd.get_connection_count()
1

We can reset the connection count back to zero.

>>> from smtplib import SMTP
>>> def reset():
...     smtpd = SMTP()
...     smtpd.connect(config.mta.smtp_host, int(config.mta.smtp_port))
...     smtpd.docmd('RSET')

>>> reset()
>>> smtpd.get_connection_count()
0

>>> connection.quit()

By providing an SMTP user name and password in the configuration file, Mailman will authenticate with the mail server after each new connection.

>>> config.push('auth', """
... [mta]
... smtp_user: testuser
... smtp_pass: testpass
... """)

>>> connection = Connection(
...     config.mta.smtp_host, int(config.mta.smtp_port), 0,
...     config.mta.smtp_user, config.mta.smtp_pass)
>>> connection.sendmail('anne@example.com', ['bart@example.com'], """\
... From: anne@example.com
... To: bart@example.com
... Subject: aardvarks
...
... """)
{}
>>> print(smtpd.get_authentication_credentials())
PLAIN AHRlc3R1c2VyAHRlc3RwYXNz

>>> reset()
>>> config.pop('auth')

Sessions per connection¶

Let’s say we specify a maximum number of sessions per connection of 2. When the third message is sent, the connection is torn down and a new one is created.

The connection count starts at zero.

>>> connection = Connection(
...     config.mta.smtp_host, int(config.mta.smtp_port), 2)

>>> smtpd.get_connection_count()
0

We send two messages through the Connection object. Only one connection is opened.

>>> connection.sendmail('anne@example.com', ['bart@example.com'], """\
... From: anne@example.com
... To: bart@example.com
... Subject: aardvarks
...
... """)
{}

>>> smtpd.get_connection_count()
1

>>> connection.sendmail('anne@example.com', ['bart@example.com'], """\
... From: anne@example.com
... To: bart@example.com
... Subject: aardvarks
...
... """)
{}

>>> smtpd.get_connection_count()
1

The third message would cause a third session, exceeding the maximum. So the current connection is closed and a new one opened.

>>> connection.sendmail('anne@example.com', ['bart@example.com'], """\
... From: anne@example.com
... To: bart@example.com
... Subject: aardvarks
...
... """)
{}

>>> smtpd.get_connection_count()
2

A fourth message does not cause a new connection to be made.

>>> connection.sendmail('anne@example.com', ['bart@example.com'], """\
... From: anne@example.com
... To: bart@example.com
... Subject: aardvarks
...
... """)
{}

>>> smtpd.get_connection_count()
2

But a fifth one does.

>>> connection.sendmail('anne@example.com', ['bart@example.com'], """\
... From: anne@example.com
... To: bart@example.com
... Subject: aardvarks
...
... """)
{}

>>> smtpd.get_connection_count()
3

No maximum¶

A value of zero means that there is an unlimited number of sessions per connection.

>>> connection = Connection(
...     config.mta.smtp_host, int(config.mta.smtp_port), 0)
>>> reset()

Even after ten messages are sent, there’s still been only one connection to the server.

>>> connection.debug = True
>>> for i in range(10):
...     # Ignore the results.
...     results = connection.sendmail(
...         'anne@example.com', ['bart@example.com'], """\
... From: anne@example.com
... To: bart@example.com
... Subject: aardvarks
...
... """)

>>> smtpd.get_connection_count()
1

Development mode¶

By putting Mailman into development mode, you can force the recipients to a given hard-coded address. This allows you to test Mailman without worrying about accidental deliveries to unintended recipients.

>>> config.push('devmode', """
... [devmode]
... enabled: yes
... recipient: zperson@example.com
... """)

>>> smtpd.clear()
>>> connection.sendmail(
...     'anne@example.com',
...     ['bart@example.com', 'cate@example.com'], """\
... From: anne@example.com
... To: bart@example.com
... Subject: aardvarks
...
... """)
{}

>>> messages = list(smtpd.messages)
>>> len(messages)
1
>>> print(messages[0].as_string())
From: anne@example.com
To: bart@example.com
Subject: aardvarks
X-Peer: ...
X-MailFrom: anne@example.com
X-RcptTo: zperson@example.com, zperson@example.com



>>> config.pop('devmode')

Personalized decoration¶

Personalized messages can be decorated by headers and footers containing information specific to the recipient.

>>> from mailman.mta.decorating import DecoratingDelivery
>>> decorating = DecoratingDelivery()

Delivery strategies must implement the proper interface.

>>> from mailman.interfaces.mta import IMailTransportAgentDelivery
>>> from zope.interface.verify import verifyObject
>>> verifyObject(IMailTransportAgentDelivery, decorating)
True

Decorations¶

Decorations are added when the mailing list had a header and/or footer defined, and the decoration handler is told to do personalized decorations. We start by writing the site-global header and footer template.

>>> import os, tempfile
>>> template_dir = tempfile.mkdtemp()
>>> site_dir = os.path.join(template_dir, 'site', 'en')
>>> os.makedirs(site_dir)
>>> config.push('templates', """
... [paths.testing]
... template_dir: {0}
... """.format(template_dir))

>>> myheader_path = os.path.join(site_dir, 'myheader.txt')
>>> with open(myheader_path, 'w') as fp:
...     print("""\
... Delivery address: $user_address
... Subscribed address: $user_delivered_to
... """, file=fp)
>>> myfooter_path = os.path.join(site_dir, 'myfooter.txt')
>>> with open(myfooter_path, 'w') as fp:
...     print("""\
... User name: $user_name
... Language: $user_language
... Options: $user_optionsurl
... """, file=fp)

Then create a mailing list which will use this header and footer. Because these are site-global templates, we can use a shorted URL.

>>> mlist = create_list('test@example.com')
>>> mlist.header_uri = 'mailman:///myheader.txt'
>>> mlist.footer_uri = 'mailman:///myfooter.txt'

>>> transaction.commit()

>>> msg = message_from_string("""\
... From: aperson@example.org
... To: test@example.com
... Subject: test one
... Message-ID: <aardvark>
...
... This is a test.
... """)

>>> recipients = set([
...     'aperson@example.com',
...     'bperson@example.com',
...     'cperson@example.com',
...     ])

>>> msgdata = dict(
...     recipients=recipients,
...     personalize=True,
...     )

More information is included when the recipient is a member of the mailing list.

>>> from zope.component import getUtility
>>> from mailman.interfaces.member import MemberRole
>>> from mailman.interfaces.usermanager import IUserManager
>>> user_manager = getUtility(IUserManager)

>>> anne = user_manager.create_user('aperson@example.com', 'Anne Person')
>>> mlist.subscribe(list(anne.addresses)[0], MemberRole.member)
<Member: Anne Person <aperson@example.com> ...

>>> bart = user_manager.create_user('bperson@example.com', 'Bart Person')
>>> mlist.subscribe(list(bart.addresses)[0], MemberRole.member)
<Member: Bart Person <bperson@example.com> ...

>>> cris = user_manager.create_user('cperson@example.com', 'Cris Person')
>>> mlist.subscribe(list(cris.addresses)[0], MemberRole.member)
<Member: Cris Person <cperson@example.com> ...

The decorations happen when the message is delivered.

>>> decorating.deliver(mlist, msg, msgdata)
{}
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> from operator import itemgetter
>>> for message in sorted(messages, key=itemgetter('x-rcptto')):
...     print(message.as_string())
...     print('----------')
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: aperson@example.com

Delivery address: aperson@example.com
Subscribed address: aperson@example.com
This is a test.
User name: Anne Person
Language: English (USA)
Options: http://example.com/aperson@example.com
----------
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: bperson@example.com

Delivery address: bperson@example.com
Subscribed address: bperson@example.com
This is a test.
User name: Bart Person
Language: English (USA)
Options: http://example.com/bperson@example.com
----------
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: cperson@example.com

Delivery address: cperson@example.com
Subscribed address: cperson@example.com
This is a test.
User name: Cris Person
Language: English (USA)
Options: http://example.com/cperson@example.com
----------

Decorate only once¶

Do not decorate a message twice. Decorators must insert the decorated key into the message metadata.

>>> msgdata['nodecorate'] = True
>>> decorating.deliver(mlist, msg, msgdata)
{}
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> for message in sorted(messages, key=itemgetter('x-rcptto')):
...     print(message.as_string())
...     print('----------')
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: aperson@example.com

This is a test.
----------
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: bperson@example.com

This is a test.
----------
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: cperson@example.com

This is a test.
----------

Fully personalized delivery¶

Fully personalized mail delivery is an enhancement over VERP delivery where the To: field of the message is replaced with the recipient’s address. A typical email message is sent to the mailing list’s posting address and copied to the list membership that way. Some people like the more personal address.

Personalized delivery still does VERP.

>>> from mailman.mta.personalized import PersonalizedDelivery
>>> personalized = PersonalizedDelivery()

Delivery strategies must implement the proper interface.

>>> from mailman.interfaces.mta import IMailTransportAgentDelivery
>>> from zope.interface.verify import verifyObject
>>> verifyObject(IMailTransportAgentDelivery, personalized)
True

No personalization¶

By default, the To: header is not personalized.

>>> mlist = create_list('test@example.com')
>>> msg = message_from_string("""\
... From: aperson@example.org
... To: test@example.com
... Subject: test one
... Message-ID: <aardvark>
...
... This is a test.
... """)

>>> recipients = set([
...     'aperson@example.com',
...     'bperson@example.com',
...     'cperson@example.com',
...     ])

>>> personalized.deliver(mlist, msg, dict(recipients=recipients))
{}
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> from operator import itemgetter
>>> for message in sorted(messages, key=itemgetter('x-rcptto')):
...     print(message.as_string())
...     print('----------')
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: aperson@example.com

This is a test.
----------
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: bperson@example.com

This is a test.
----------
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: cperson@example.com

This is a test.
----------

To header¶

When the mailing list requests personalization, the To: header is replaced with the recipient’s address and name.

>>> from mailman.interfaces.mailinglist import Personalization
>>> mlist.personalize = Personalization.full
>>> transaction.commit()

>>> personalized.deliver(mlist, msg, dict(recipients=recipients))
{}
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> for message in sorted(messages, key=itemgetter('to')):
...     print(message.as_string())
...     print('----------')
From: aperson@example.org
To: aperson@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: aperson@example.com

This is a test.
----------
From: aperson@example.org
To: bperson@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: bperson@example.com

This is a test.
----------
From: aperson@example.org
To: cperson@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: cperson@example.com

This is a test.
----------

If the recipient is a user registered with Mailman, and the user has an associated real name, then this name also shows up in the To: header.

>>> from zope.component import getUtility
>>> from mailman.interfaces.usermanager import IUserManager
>>> user_manager = getUtility(IUserManager)

>>> bill = user_manager.create_user('bperson@example.com', 'Bill Person')
>>> cate = user_manager.create_user('cperson@example.com', 'Cate Person')
>>> transaction.commit()

>>> personalized.deliver(mlist, msg, dict(recipients=recipients))
{}
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> from operator import itemgetter
>>> for message in sorted(messages, key=itemgetter('x-rcptto')):
...     print(message.as_string())
...     print('----------')
From: aperson@example.org
To: aperson@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: aperson@example.com

This is a test.
----------
From: aperson@example.org
To: Bill Person <bperson@example.com>
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: bperson@example.com

This is a test.
----------
From: aperson@example.org
To: Cate Person <cperson@example.com>
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces@example.com
X-RcptTo: cperson@example.com

This is a test.
----------

Standard VERP delivery¶

Variable Envelope Return Path (VERP) delivery is an alternative to bulk delivery, where an individual message is crafted uniquely for each recipient.

The cost of enabling VERP is that Mailman must send to the upstream MTA, one message per recipient. Under bulk delivery, an exact copy of one message can be sent to many recipients, greatly reducing the bandwidth for delivery.

In Mailman, enabling VERP delivery for bounce detection brings with it a side benefit: the message which must be crafted uniquely for each recipient, can be further personalized to include all kinds of information unique to that recipient. In the simplest case, the message can contain footer information, e.g. pointing the user to their account URL or including a user-specific unsubscription link. In theory, VERP delivery means we can do sophisticated mail merge operations.

Mailman’s use of the term VERP really means message personalization.

>>> from mailman.mta.verp import VERPDelivery
>>> verp = VERPDelivery()

Delivery strategies must implement the proper interface.

>>> from mailman.interfaces.mta import IMailTransportAgentDelivery
>>> from zope.interface.verify import verifyObject
>>> verifyObject(IMailTransportAgentDelivery, verp)
True

No recipients¶

The message metadata specifies the set of recipients to send this message to. If there are no recipients, there’s nothing to do.

>>> mlist = create_list('test@example.com')
>>> msg = message_from_string("""\
... From: aperson@example.org
... To: test@example.com
... Subject: test one
... Message-ID: <aardvark>
...
... This is a test.
... """)

>>> verp.deliver(mlist, msg, {})
{}
>>> len(list(smtpd.messages))
0

>>> verp.deliver(mlist, msg, dict(recipients=set()))
{}
>>> len(list(smtpd.messages))
0

Individual copy¶

Each recipient of the message gets an individual, personalized copy of the message, with their email address encoded into the envelope sender. This is so the return path will point back to Mailman but allow for decoding of the intended recipient’s delivery address.

>>> recipients = set([
...     'aperson@example.com',
...     'bperson@example.com',
...     'cperson@example.com',
...     ])

VERPing is only actually done if the metadata requests it.

>>> msgdata = dict(recipients=recipients, verp=True)
>>> verp.deliver(mlist, msg, msgdata)
{}
>>> messages = list(smtpd.messages)
>>> len(messages)
3

>>> from operator import itemgetter
>>> for message in sorted(messages, key=itemgetter('x-rcptto')):
...     print(message.as_string())
...     print('----------')
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces+aperson=example.com@example.com
X-RcptTo: aperson@example.com

This is a test.
----------
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces+bperson=example.com@example.com
X-RcptTo: bperson@example.com

This is a test.
----------
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
X-Peer: ...
X-MailFrom: test-bounces+cperson=example.com@example.com
X-RcptTo: cperson@example.com

This is a test.
----------

The deliverer made a copy of the original message, so it wasn’t changed.

>>> print(msg.as_string())
From: aperson@example.org
To: test@example.com
Subject: test one
Message-ID: <aardvark>
<BLANKLINE>
This is a test.
<BLANKLINE>

Mailman runner control¶

Mailman has a number of runner subprocesses which perform long-running tasks such as listening on an LMTP port, processing REST API requests, or processing messages in a queue directory. In normal operation, the mailman command is used to start, stop and manage the runners. This is just a wrapper around the real master watcher, which handles runner starting, stopping, exiting, and log file reopening.

>>> from mailman.testing.helpers import TestableMaster

Start the master in a sub-thread.

>>> master = TestableMaster()
>>> master.start()

There should be a process id for every runner that claims to be startable.

>>> from lazr.config import as_boolean
>>> startable_runners = [conf for conf in config.runner_configs
...                      if as_boolean(conf.start)]
>>> len(list(master.runner_pids)) == len(startable_runners)
True

Now verify that all the runners are running.

>>> import os

# This should produce no output.
>>> for pid in master.runner_pids:
...     os.kill(pid, 0)

Stop the master process, which should also kill (and not restart) the child runner processes.

>>> master.stop()

None of the children are running now.

>>> import errno
>>> for pid in master.runner_pids:
...     try:
...         os.kill(pid, 0)
...         print('Process did not exit:', pid)
...     except OSError as error:
...         if error.errno == errno.ESRCH:
...             # The child process exited.
...             pass
...         else:
...             raise

Generating aliases¶

For some mail servers, Mailman must generate data files that are used to hook Mailman up to the mail server. The details of this differ for each mail server. Generally these files are automatically kept up-to-date when mailing lists are created or removed, but you might occasionally need to manually regenerate the file. The mailman aliases command does this.

>>> class FakeArgs:
...     directory = None
>>> from mailman.commands.cli_aliases import Aliases
>>> command = Aliases()

For example, connecting Mailman to Postfix is generally done through the LMTP protocol. Mailman starts an LMTP server and Postfix delivers messages to Mailman as an LMTP client. By default this is done through Postfix transport maps.

Selecting Postfix as the source of incoming messages enables transport map generation.

>>> config.push('postfix', """
... [mta]
... incoming: mailman.mta.postfix.LMTP
... lmtp_host: lmtp.example.com
... lmtp_port: 24
... """)

Let’s create a mailing list and then display the transport map for it. We’ll write the appropriate files to a temporary directory.

>>> import os, shutil, tempfile
>>> output_directory = tempfile.mkdtemp()
>>> cleanups.append((shutil.rmtree, output_directory))

>>> FakeArgs.directory = output_directory
>>> mlist = create_list('test@example.com')
>>> command.process(FakeArgs)

For Postfix, there are two files in the output directory.

>>> files = sorted(os.listdir(output_directory))
>>> for file in files:
...     print(file)
postfix_domains
postfix_lmtp

The transport map file contains all the aliases for the mailing list.

>>> with open(os.path.join(output_directory, 'postfix_lmtp')) as fp:
...     print(fp.read())
# AUTOMATICALLY GENERATED BY MAILMAN ON ...
...
test@example.com               lmtp:[lmtp.example.com]:24
test-bounces@example.com       lmtp:[lmtp.example.com]:24
test-confirm@example.com       lmtp:[lmtp.example.com]:24
test-join@example.com          lmtp:[lmtp.example.com]:24
test-leave@example.com         lmtp:[lmtp.example.com]:24
test-owner@example.com         lmtp:[lmtp.example.com]:24
test-request@example.com       lmtp:[lmtp.example.com]:24
test-subscribe@example.com     lmtp:[lmtp.example.com]:24
test-unsubscribe@example.com   lmtp:[lmtp.example.com]:24
<BLANKLINE>

The relay domains file contains a list of all the domains.

>>> with open(os.path.join(output_directory, 'postfix_domains')) as fp:
...     print(fp.read())
# AUTOMATICALLY GENERATED BY MAILMAN ON ...
...
example.com example.com

Display configuration values¶

Just like the Postfix command postconf(1), the mailman conf command lets you dump one or more Mailman configuration variables to standard output or a file.

Mailman’s configuration is divided in multiple sections which contain multiple key-value pairs. The mailman conf command allows you to display a specific key-value pair, or several key-value pairs.

>>> class FakeArgs:
...     key = None
...     section = None
...     output = None
>>> from mailman.commands.cli_conf import Conf
>>> command = Conf()

To get a list of all key-value pairs of any section, you need to call the command without any options.

>>> command.process(FakeArgs)
[antispam] header_checks:
...
[logging.bounce] level: info
...
[mailman] site_owner: noreply@example.com
...

You can list all the key-value pairs of a specific section.

>>> FakeArgs.section = 'shell'
>>> command.process(FakeArgs)
[shell] banner: Welcome to the GNU Mailman shell
[shell] prompt: >>>
[shell] use_ipython: no

You can also pass a key and display all key-value pairs matching the given key, along with the names of the corresponding sections.

>>> FakeArgs.section = None
>>> FakeArgs.key = 'path'
>>> command.process(FakeArgs)
[logging.archiver] path: mailman.log
[logging.bounce] path: bounce.log
[logging.config] path: mailman.log
[logging.database] path: mailman.log
[logging.debug] path: debug.log
[logging.error] path: mailman.log
[logging.fromusenet] path: mailman.log
[logging.http] path: mailman.log
[logging.locks] path: mailman.log
[logging.mischief] path: mailman.log
[logging.root] path: mailman.log
[logging.runner] path: mailman.log
[logging.smtp] path: smtp.log
[logging.subscribe] path: mailman.log
[logging.vette] path: mailman.log

If you specify both a section and a key, you will get the corresponding value.

>>> FakeArgs.section = 'mailman'
>>> FakeArgs.key = 'site_owner'
>>> command.process(FakeArgs)
noreply@example.com

Starting and stopping Mailman¶

The Mailman daemon processes can be started and stopped from the command line.

Set up¶

All we care about is the master process; normally it starts a bunch of runners, but we don’t care about any of them, so write a test configuration file for the master that disables all the runners.

>>> from mailman.commands.tests.test_control import make_config

Starting¶

>>> from mailman.commands.cli_control import Start
>>> start = Start()

>>> class FakeArgs:
...     force = False
...     run_as_user = True
...     quiet = False
...     config = make_config()
>>> args = FakeArgs()

Starting the daemons prints a useful message and starts the master watcher process in the background.

>>> start.process(args)
Starting Mailman's master runner

>>> from mailman.commands.tests.test_control import find_master

The process exists, and its pid is available in a run time file.

>>> pid = find_master()
>>> pid is not None
True

Stopping¶

You can also stop the master watcher process from the command line, which stops all the child processes too.

>>> from mailman.commands.cli_control import Stop
>>> stop = Stop()
>>> stop.process(args)
Shutting down Mailman's master runner

>>> from datetime import datetime, timedelta
>>> import os
>>> import time
>>> import errno
>>> def bury_master():
...     until = timedelta(seconds=2) + datetime.now()
...     while datetime.now() < until:
...         time.sleep(0.1)
...         try:
...             os.kill(pid, 0)
...             os.waitpid(pid, os.WNOHANG)
...         except OSError as error:
...             if error.errno == errno.ESRCH:
...                 # The process has exited.
...                 print('Master process went bye bye')
...                 return
...             else:
...                 raise
...     else:
...         raise AssertionError('Master process lingered')

>>> bury_master()
Master process went bye bye

XXX We need tests for restart (SIGUSR1) and reopen (SIGHUP).

Command line list creation¶

A system administrator can create mailing lists by the command line.

>>> class FakeArgs:
...     language = None
...     owners = []
...     quiet = False
...     domain = None
...     listname = None
...     notify = False

You cannot create a mailing list in an unknown domain.

>>> from mailman.commands.cli_lists import Create
>>> command = Create()

>>> class FakeParser:
...     def error(self, message):
...         print(message)
>>> command.parser = FakeParser()

>>> FakeArgs.listname = ['test@example.xx']
>>> command.process(FakeArgs)
Undefined domain: example.xx

The -d or --domain option is used to tell Mailman to auto-register the domain. Both the mailing list and domain will be created.

>>> FakeArgs.domain = True
>>> command.process(FakeArgs)
Created mailing list: test@example.xx

Now both the domain and the mailing list exist in the database.

>>> from mailman.interfaces.listmanager import IListManager
>>> from zope.component import getUtility
>>> list_manager = getUtility(IListManager)
>>> list_manager.get('test@example.xx')
<mailing list "test@example.xx" at ...>

>>> from mailman.interfaces.domain import IDomainManager
>>> getUtility(IDomainManager).get('example.xx')
<Domain example.xx, base_url: http://example.xx>

You can also create mailing lists in existing domains without the auto-creation flag.

>>> FakeArgs.domain = False
>>> FakeArgs.listname = ['test1@example.com']
>>> command.process(FakeArgs)
Created mailing list: test1@example.com

>>> list_manager.get('test1@example.com')
<mailing list "test1@example.com" at ...>

The command can also operate quietly.

>>> FakeArgs.quiet = True
>>> FakeArgs.listname = ['test2@example.com']
>>> command.process(FakeArgs)

>>> mlist = list_manager.get('test2@example.com')
>>> mlist
<mailing list "test2@example.com" at ...>

Setting the owner¶

By default, no list owners are specified.

>>> dump_list(mlist.owners.addresses)
*Empty*

But you can specify an owner address on the command line when you create the mailing list.

>>> FakeArgs.quiet = False
>>> FakeArgs.listname = ['test4@example.com']
>>> FakeArgs.owners = ['foo@example.org']
>>> command.process(FakeArgs)
Created mailing list: test4@example.com

>>> mlist = list_manager.get('test4@example.com')
>>> dump_list(repr(address) for address in mlist.owners.addresses)
<Address: foo@example.org [not verified] at ...>

You can even specify more than one address for the owners.

>>> FakeArgs.owners = ['foo@example.net',
...                    'bar@example.net',
...                    'baz@example.net']
>>> FakeArgs.listname = ['test5@example.com']
>>> command.process(FakeArgs)
Created mailing list: test5@example.com

>>> mlist = list_manager.get('test5@example.com')
>>> from operator import attrgetter
>>> dump_list(repr(address) for address in mlist.owners.addresses)
<Address: bar@example.net [not verified] at ...>
<Address: baz@example.net [not verified] at ...>
<Address: foo@example.net [not verified] at ...>

Setting the language¶

You can set the default language for the new mailing list when you create it. The language must be known to Mailman.

>>> FakeArgs.listname = ['test3@example.com']
>>> FakeArgs.language = 'ee'
>>> command.process(FakeArgs)
Invalid language code: ee

>>> from mailman.interfaces.languages import ILanguageManager
>>> getUtility(ILanguageManager).add('ee', 'iso-8859-1', 'Freedonian')
<Language [ee] Freedonian>

>>> FakeArgs.quiet = False
>>> FakeArgs.listname = ['test3@example.com']
>>> FakeArgs.language = 'fr'
>>> command.process(FakeArgs)
Created mailing list: test3@example.com

>>> mlist = list_manager.get('test3@example.com')
>>> print(mlist.preferred_language)
<Language [fr] French>
>>> FakeArgs.language = None

Notifications¶

When told to, Mailman will notify the list owners of their new mailing list.

>>> FakeArgs.listname = ['test6@example.com']
>>> FakeArgs.notify = True
>>> command.process(FakeArgs)
Created mailing list: test6@example.com

The notification message is in the virgin queue.

>>> from mailman.testing.helpers import get_queue_messages
>>> messages = get_queue_messages('virgin')
>>> len(messages)
1

>>> for message in messages:
...     print(message.msg.as_string())
MIME-Version: 1.0
...
Subject: Your new mailing list: test6@example.com
From: noreply@example.com
To: foo@example.net, bar@example.net, baz@example.net
...

The mailing list 'test6@example.com' has just been created for you.
The following is some basic information about your mailing list.

You can configure your mailing list at the following web page:

    http://lists.example.com/admin/test6@example.com

The web page for users of your mailing list is:

    http://lists.example.com/listinfo/test6@example.com

There is also an email-based interface for users (not administrators)
of your list; you can get info about using it by sending a message
with just the word 'help' as subject or in the body, to:

    test6-request@example.com

Please address all questions to noreply@example.com.

The ‘echo’ command¶

The mail command ‘echo’ simply replies with the original command and arguments to the sender.

>>> command = config.commands['echo']
>>> print(command.name)
echo
>>> print(command.argument_description)
[args]
>>> print(command.description)
Echo back your arguments.

The original message is ignored, but the results receive the echoed command.

>>> mlist = create_list('test@example.com')

>>> from mailman.runners.command import Results
>>> results = Results()

>>> from mailman.email.message import Message
>>> print(command.process(mlist, Message(), {}, ('foo', 'bar'), results))
ContinueProcessing.yes
>>> print(str(results))
The results of your email command are provided below.

echo foo bar

The ‘end’ command¶

The mail command processor recognized an ‘end’ command which tells it to stop processing email messages.

>>> command = config.commands['end']
>>> print(command.name)
end
>>> print(command.description)
Stop processing commands.

The ‘end’ command takes no arguments.

>>> print('DESCRIPTION:', command.argument_description)
DESCRIPTION:

The command itself is fairly simple; it just stops command processing, and the message isn’t even looked at.

>>> mlist = create_list('test@example.com')
>>> from mailman.email.message import Message
>>> print(command.process(mlist, Message(), {}, (), None))
ContinueProcessing.no

The ‘stop’ command is a synonym for ‘end’.

>>> command = config.commands['stop']
>>> print(command.name)
stop
>>> print(command.description)
An alias for 'end'.
>>> print('DESCRIPTION:', command.argument_description)
DESCRIPTION:
>>> print(command.process(mlist, Message(), {}, (), None))
ContinueProcessing.no

Email command help¶

You can get some help about the various email commands that are available by sending the word help to a mailing list’s -request address.

>>> mlist = create_list('test@example.com')
>>> from mailman.commands.eml_help import Help
>>> help = Help()
>>> print(help.name)
help
>>> print(help.description)
Get help about available email commands.
>>> print(help.argument_description)
[command]

With no arguments, help provides a list of the available commands and a short description of each of them.

>>> from mailman.runners.command import Results
>>> results = Results()

>>> from mailman.email.message import Message
>>> print(help.process(mlist, Message(), {}, (), results))
ContinueProcessing.yes
>>> print(results)
The results of your email command are provided below.

confirm     - Confirm a subscription request.
echo        - Echo back your arguments.
end         - Stop processing commands.
help        - Get help about available email commands.
join        - Join this mailing list.
leave       - Leave this mailing list.
stop        - An alias for 'end'.
subscribe   - An alias for 'join'.
unsubscribe - An alias for 'leave'.

With an argument, you can get more detailed help about a specific command.

>>> results = Results()
>>> print(help.process(mlist, Message(), {}, ('help',), results))
ContinueProcessing.yes
>>> print(results)
The results of your email command are provided below.
<BLANKLINE>
help [command]
Get help about available email commands.
<BLANKLINE>

Some commands have even more detailed help.

>>> results = Results()
>>> print(help.process(mlist, Message(), {}, ('join',), results))
ContinueProcessing.yes
>>> print(results)
The results of your email command are provided below.
<BLANKLINE>
join [digest=<no|mime|plain>]
Join this mailing list.
<BLANKLINE>
You will be asked to confirm your subscription request and you may be
issued a provisional password.
<BLANKLINE>
By using the 'digest' option, you can specify whether you want digest
delivery or not.  If not specified, the mailing list's default delivery
mode will be used.
<BLANKLINE>

Importing list data¶

If you have the config.pck file for a version 2.1 mailing list, you can import that into an existing mailing list in Mailman 3.0.

>>> from mailman.commands.cli_import import Import21
>>> command = Import21()

>>> class FakeArgs:
...     listname = None
...     pickle_file = None

>>> class FakeParser:
...     def error(self, message):
...         print(message)
>>> command.parser = FakeParser()

You must specify the mailing list you are importing into, and it must exist.

>>> command.process(FakeArgs)
List name is required

>>> FakeArgs.listname = ['import@example.com']
>>> command.process(FakeArgs)
No such list: import@example.com

When the mailing list exists, you must specify a real pickle file to import from.

>>> mlist = create_list('import@example.com')
>>> command.process(FakeArgs)
config.pck file is required

>>> FakeArgs.pickle_file = [__file__]
>>> command.process(FakeArgs)
Not a Mailman 2.1 configuration file: .../import.rst

Now we can import the test pickle file. As a simple illustration of the import, the mailing list’s ‘real name’ has changed.

>>> from pkg_resources import resource_filename
>>> FakeArgs.pickle_file = [
...     resource_filename('mailman.testing', 'config.pck')]

>>> print(mlist.display_name)
Import

>>> command.process(FakeArgs)
>>> print(mlist.display_name)
Test

Getting information¶

You can get information about Mailman’s environment by using the command line script mailman info. By default, the info is printed to standard output.

>>> from mailman.commands.cli_info import Info
>>> command = Info()

>>> class FakeArgs:
...     output = None
...     verbose = None
>>> args = FakeArgs()

>>> command.process(args)
GNU Mailman 3...
Python ...
...
config file: .../test.cfg
db url: ...
REST root url: http://localhost:9001/3.0/
REST credentials: restadmin:restpass

By passing in the -o/--output option, you can print the info to a file.

>>> from mailman.config import config
>>> import os
>>> output_path = os.path.join(config.VAR_DIR, 'output.txt')
>>> args.output = output_path
>>> command.process(args)
>>> with open(output_path) as fp:
...     print(fp.read())
GNU Mailman 3...
Python ...
...
config file: .../test.cfg
db url: ...
devmode: DISABLED
REST root url: http://localhost:9001/3.0/
REST credentials: restadmin:restpass

You can also get more verbose information, which contains a list of the file system paths that Mailman is using.

>>> args.output = None
>>> args.verbose = True
>>> config.create_paths = False
>>> config.push('fhs', """
... [mailman]
... layout: fhs
... """)
>>> cleanups.append((config.pop, 'fhs'))
>>> config.create_paths = True

The Filesystem Hierarchy Standard layout is the same everywhere by definition.

>>> command.process(args)
GNU Mailman 3...
Python ...
...
File system paths:
    ARCHIVE_DIR     = /var/lib/mailman/archives
    BIN_DIR         = /sbin
    CFG_FILE        = .../test.cfg
    DATA_DIR        = /var/lib/mailman/data
    ETC_DIR         = /etc
    EXT_DIR         = /etc/mailman.d
    LIST_DATA_DIR   = /var/lib/mailman/lists
    LOCK_DIR        = /var/lock/mailman
    LOCK_FILE       = /var/lock/mailman/master.lck
    LOG_DIR         = /var/log/mailman
    MESSAGES_DIR    = /var/lib/mailman/messages
    PID_FILE        = /var/run/mailman/master.pid
    QUEUE_DIR       = /var/spool/mailman
    TEMPLATE_DIR    = .../mailman/templates
    VAR_DIR         = /var/lib/mailman

Command line message injection¶

You can inject a message directly into a queue directory via the command line.

>>> from mailman.commands.cli_inject import Inject
>>> command = Inject()

>>> class FakeArgs:
...     queue = None
...     show = False
...     filename = None
...     listname = None
...     keywords = []
>>> args = FakeArgs()

>>> class FakeParser:
...     def error(self, message):
...         print(message)
>>> command.parser = FakeParser()

It’s easy to find out which queues are available.

>>> args.show = True
>>> command.process(args)
Available queues:
    archive
    bad
    bounces
    command
    digest
    in
    nntp
    out
    pipeline
    retry
    shunt
    virgin

>>> args.show = False

Usually, the text of the message to inject is in a file.

>>> import os, tempfile
>>> fd, filename = tempfile.mkstemp()
>>> with os.fdopen(fd, 'w') as fp:
...     print("""\
... From: aperson@example.com
... To: test@example.com
... Subject: testing
... Message-ID: <aardvark>
...
... This is a test message.
... """, file=fp)

However, the mailing list name is always required.

>>> args.filename = filename
>>> command.process(args)
List name is required

Let’s provide a list name and try again.

>>> mlist = create_list('test@example.com')
>>> transaction.commit()
>>> from mailman.testing.helpers import get_queue_messages

>>> get_queue_messages('in')
[]
>>> args.listname = ['test@example.com']
>>> command.process(args)

By default, the incoming queue is used.

>>> items = get_queue_messages('in')
>>> len(items)
1
>>> print(items[0].msg.as_string())
From: aperson@example.com
To: test@example.com
Subject: testing
Message-ID: ...
Date: ...

This is a test message.



>>> dump_msgdata(items[0].msgdata)
_parsemsg    : False
listid       : test.example.com
original_size: 203
version      : 3

But a different queue can be specified on the command line.

>>> args.queue = 'virgin'
>>> command.process(args)

>>> get_queue_messages('in')
[]
>>> items = get_queue_messages('virgin')
>>> len(items)
1
>>> print(items[0].msg.as_string())
From: aperson@example.com
To: test@example.com
Subject: testing
Message-ID: ...
Date: ...

This is a test message.



>>> dump_msgdata(items[0].msgdata)
_parsemsg    : False
listid       : test.example.com
original_size: 203
version      : 3

Standard input¶

The message text can also be provided on standard input.

>>> from io import StringIO

>>> standard_in = StringIO(str("""\
... From: bperson@example.com
... To: test@example.com
... Subject: another test
... Message-ID: <badger>
...
... This is another test message.
... """))

>>> import sys
>>> sys.stdin = standard_in
>>> args.filename = '-'
>>> args.queue = None

>>> command.process(args)
>>> items = get_queue_messages('in')
>>> len(items)
1
>>> print(items[0].msg.as_string())
From: bperson@example.com
To: test@example.com
Subject: another test
Message-ID: ...
Date: ...

This is another test message.



>>> dump_msgdata(items[0].msgdata)
_parsemsg    : False
listid       : test.example.com
original_size: 211
version      : 3

Metadata¶

Additional metadata keys can be provided on the command line. These key/value pairs get added to the message metadata dictionary when the message is injected.

>>> args = FakeArgs()
>>> args.filename = filename
>>> args.listname = ['test@example.com']
>>> args.keywords = ['foo=one', 'bar=two']
>>> command.process(args)

>>> items = get_queue_messages('in')
>>> dump_msgdata(items[0].msgdata)
_parsemsg    : False
bar          : two
foo          : one
listid       : test.example.com
original_size: 203
version      : 3

Errors¶

It is an error to specify a queue that doesn’t exist.

>>> args.queue = 'xxbogusxx'
>>> command.process(args)
No such queue: xxbogusxx

It is also an error to specify a mailing list that doesn’t exist.

>>> args.queue = None
>>> args.listname = ['bogus']
>>> command.process(args)
No such list: bogus

Command line list display¶

A system administrator can display all the mailing lists via the command line. When there are no mailing lists, a helpful message is displayed.

>>> class FakeArgs:
...     advertised = False
...     names = False
...     descriptions = False
...     quiet = False
...     domains = None

>>> from mailman.commands.cli_lists import Lists
>>> command = Lists()
>>> command.process(FakeArgs)
No matching mailing lists found

When there are a few mailing lists, they are shown in alphabetical order by their fully qualified list names, with a description.

>>> from mailman.interfaces.domain import IDomainManager
>>> from zope.component import getUtility
>>> getUtility(IDomainManager).add('example.net')
<Domain example.net...>

>>> mlist_1 = create_list('list-one@example.com')
>>> mlist_1.description = 'List One'

>>> mlist_2 = create_list('list-two@example.com')
>>> mlist_2.description = 'List Two'

>>> mlist_3 = create_list('list-one@example.net')
>>> mlist_3.description = 'List One in Example.Net'

>>> command.process(FakeArgs)
3 matching mailing lists found:
list-one@example.com
list-one@example.net
list-two@example.com

Names¶

You can display the mailing list names with their posting addresses, using the --names/-n switch.

>>> FakeArgs.names = True
>>> command.process(FakeArgs)
3 matching mailing lists found:
list-one@example.com [List-one]
list-one@example.net [List-one]
list-two@example.com [List-two]

Descriptions¶

You can also display the mailing list descriptions, using the --descriptions/-d option.

>>> FakeArgs.descriptions = True
>>> command.process(FakeArgs)
3 matching mailing lists found:
list-one@example.com [List-one] - List One
list-one@example.net [List-one] - List One in Example.Net
list-two@example.com [List-two] - List Two

Maybe you want the descriptions but not the names.

>>> FakeArgs.names = False
>>> command.process(FakeArgs)
3 matching mailing lists found:
list-one@example.com - List One
list-one@example.net - List One in Example.Net
list-two@example.com - List Two

Less verbosity¶

There’s also a --quiet/-q switch which reduces the verbosity a bit.

>>> FakeArgs.quiet = True
>>> FakeArgs.descriptions = False
>>> command.process(FakeArgs)
list-one@example.com
list-one@example.net
list-two@example.com

Specific domain¶

You can narrow the search down to a specific domain with the –domain option. A helpful message is displayed if no matching domains are given.

>>> FakeArgs.quiet = False
>>> FakeArgs.domain = ['example.org']
>>> command.process(FakeArgs)
No matching mailing lists found

But if a matching domain is given, only mailing lists in that domain are shown.

>>> FakeArgs.domain = ['example.net']
>>> command.process(FakeArgs)
1 matching mailing lists found:
list-one@example.net

More than one –domain argument can be given; then all mailing lists in matching domains are shown.

>>> FakeArgs.domain = ['example.com', 'example.net']
>>> command.process(FakeArgs)
3 matching mailing lists found:
list-one@example.com
list-one@example.net
list-two@example.com

Advertised lists¶

Mailing lists can be ‘advertised’ meaning their existence is public knowledge. Non-advertised lists are considered private. Display through the command line can select on this attribute.

>>> FakeArgs.domain = []
>>> FakeArgs.advertised = True
>>> mlist_1.advertised = False

>>> command.process(FakeArgs)
2 matching mailing lists found:
list-one@example.net
list-two@example.com

Managing members¶

The mailman members command allows a site administrator to display, add, and remove members from a mailing list.

>>> mlist1 = create_list('test1@example.com')

>>> class FakeArgs:
...     input_filename = None
...     output_filename = None
...     listname = []
...     regular = False
...     digest = None
...     nomail = None
>>> args = FakeArgs()

>>> from mailman.commands.cli_members import Members
>>> command = Members()

Listing members¶

You can list all the members of a mailing list by calling the command with no options. To start with, there are no members of the mailing list.

>>> args.listname = [mlist1.fqdn_listname]
>>> command.process(args)
test1@example.com has no members

Once the mailing list add some members, they will be displayed.

>>> from mailman.testing.helpers import subscribe
>>> subscribe(mlist1, 'Anne', email='anne@example.com')
<Member: Anne Person <anne@example.com> on test1@example.com
         as MemberRole.member>
>>> subscribe(mlist1, 'Bart', email='bart@example.com')
<Member: Bart Person <bart@example.com> on test1@example.com
         as MemberRole.member>
>>> command.process(args)
Anne Person <anne@example.com>
Bart Person <bart@example.com>

Members are displayed in alphabetical order based on their address.

>>> subscribe(mlist1, 'Anne', email='anne@aaaxample.com')
<Member: Anne Person <anne@aaaxample.com> on test1@example.com
         as MemberRole.member>
>>> command.process(args)
Anne Person <anne@aaaxample.com>
Anne Person <anne@example.com>
Bart Person <bart@example.com>

You can also output this list to a file.

>>> from tempfile import mkstemp
>>> fd, args.output_filename = mkstemp()
>>> import os
>>> os.close(fd)
>>> command.process(args)
>>> with open(args.output_filename) as fp:
...     print(fp.read())
Anne Person <anne@aaaxample.com>
Anne Person <anne@example.com>
Bart Person <bart@example.com>
>>> os.remove(args.output_filename)
>>> args.output_filename = None

The output file can also be standard out.

>>> args.output_filename = '-'
>>> command.process(args)
Anne Person <anne@aaaxample.com>
Anne Person <anne@example.com>
Bart Person <bart@example.com>
>>> args.output_filename = None

Filtering on delivery mode¶

You can limit output to just the regular non-digest members…

>>> from mailman.interfaces.member import DeliveryMode
>>> args.regular = True
>>> member = mlist1.members.get_member('anne@example.com')
>>> member.preferences.delivery_mode = DeliveryMode.plaintext_digests
>>> command.process(args)
Anne Person <anne@aaaxample.com>
Bart Person <bart@example.com>

…or just the digest members. Furthermore, you can either display all digest members…

>>> member = mlist1.members.get_member('anne@aaaxample.com')
>>> member.preferences.delivery_mode = DeliveryMode.mime_digests
>>> args.regular = False
>>> args.digest = 'any'
>>> command.process(args)
Anne Person <anne@aaaxample.com>
Anne Person <anne@example.com>

…just plain text digest members…

>>> args.digest = 'plaintext'
>>> command.process(args)
Anne Person <anne@example.com>

…just MIME digest members.

>>> args.digest = 'mime'
>>> command.process(args)
Anne Person <anne@aaaxample.com>

# Reset for following tests.
>>> args.digest = None

Filtering on delivery status¶

You can also filter the display on the member’s delivery status. By default, all members are displayed, but you can filter out only those whose delivery status is enabled…

>>> from mailman.interfaces.member import DeliveryStatus

>>> member = mlist1.members.get_member('anne@aaaxample.com')
>>> member.preferences.delivery_status = DeliveryStatus.by_moderator
>>> member = mlist1.members.get_member('bart@example.com')
>>> member.preferences.delivery_status = DeliveryStatus.by_user

>>> member = subscribe(mlist1, 'Cris', email='cris@example.com')
>>> member.preferences.delivery_status = DeliveryStatus.unknown
>>> member = subscribe(mlist1, 'Dave', email='dave@example.com')
>>> member.preferences.delivery_status = DeliveryStatus.enabled
>>> member = subscribe(mlist1, 'Elle', email='elle@example.com')
>>> member.preferences.delivery_status = DeliveryStatus.by_bounces

>>> args.nomail = 'enabled'
>>> command.process(args)
Anne Person <anne@example.com>
Dave Person <dave@example.com>

…or disabled by the user…

>>> args.nomail = 'byuser'
>>> command.process(args)
Bart Person <bart@example.com>

…or disabled by the list administrator (or moderator)…

>>> args.nomail = 'byadmin'
>>> command.process(args)
Anne Person <anne@aaaxample.com>

…or by the bounce processor…

>>> args.nomail = 'bybounces'
>>> command.process(args)
Elle Person <elle@example.com>

…or for unknown (legacy) reasons.

>>> args.nomail = 'unknown'
>>> command.process(args)
Cris Person <cris@example.com>

You can also display all members who have delivery disabled for any reason.

>>> args.nomail = 'any'
>>> command.process(args)
Anne Person <anne@aaaxample.com>
Bart Person <bart@example.com>
Cris Person <cris@example.com>
Elle Person <elle@example.com>

# Reset for following tests.
>>> args.nomail = None

Adding members¶

You can add members to a mailing list from the command line. To do so, you need a file containing email addresses and full names that can be parsed by email.utils.parseaddr().

>>> mlist2 = create_list('test2@example.com')

>>> import os
>>> path = os.path.join(config.VAR_DIR, 'addresses.txt')
>>> with open(path, 'w') as fp:
...     for address in ('aperson@example.com',
...                     'Bart Person <bperson@example.com>',
...                     'cperson@example.com (Cate Person)',
...                     ):
...         print(address, file=fp)

>>> args.input_filename = path
>>> args.listname = [mlist2.fqdn_listname]
>>> command.process(args)

>>> from operator import attrgetter
>>> dump_list(mlist2.members.addresses, key=attrgetter('email'))
aperson@example.com
Bart Person <bperson@example.com>
Cate Person <cperson@example.com>

You can also specify - as the filename, in which case the addresses are taken from standard input.

>>> from io import StringIO
>>> fp = StringIO()
>>> for address in ('dperson@example.com',
...                 'Elly Person <eperson@example.com>',
...                 'fperson@example.com (Fred Person)',
...                 ):
...         print(address, file=fp)
>>> filepos = fp.seek(0)
>>> import sys
>>> sys.stdin = fp

>>> args.input_filename = '-'
>>> command.process(args)
>>> sys.stdin = sys.__stdin__

>>> dump_list(mlist2.members.addresses, key=attrgetter('email'))
aperson@example.com
Bart Person <bperson@example.com>
Cate Person <cperson@example.com>
dperson@example.com
Elly Person <eperson@example.com>
Fred Person <fperson@example.com>

Blank lines and lines that begin with ‘#’ are ignored.

>>> with open(path, 'w') as fp:
...     for address in ('gperson@example.com',
...                     '# hperson@example.com',
...                     '   ',
...                     '',
...                     'iperson@example.com',
...                     ):
...         print(address, file=fp)

>>> args.input_filename = path
>>> command.process(args)
>>> dump_list(mlist2.members.addresses, key=attrgetter('email'))
aperson@example.com
Bart Person <bperson@example.com>
Cate Person <cperson@example.com>
dperson@example.com
Elly Person <eperson@example.com>
Fred Person <fperson@example.com>
gperson@example.com
iperson@example.com

Addresses which are already subscribed are ignored, although a warning is printed.

>>> with open(path, 'w') as fp:
...     for address in ('gperson@example.com',
...                     'aperson@example.com',
...                     'jperson@example.com',
...                     ):
...         print(address, file=fp)

>>> command.process(args)
Already subscribed (skipping): gperson@example.com
Already subscribed (skipping): aperson@example.com

>>> dump_list(mlist2.members.addresses, key=attrgetter('email'))
aperson@example.com
Bart Person <bperson@example.com>
Cate Person <cperson@example.com>
dperson@example.com
Elly Person <eperson@example.com>
Fred Person <fperson@example.com>
gperson@example.com
iperson@example.com
jperson@example.com

Displaying members¶

With no arguments, the command displays all members of the list.

>>> args.input_filename = None
>>> command.process(args)
aperson@example.com
Bart Person <bperson@example.com>
Cate Person <cperson@example.com>
dperson@example.com
Elly Person <eperson@example.com>
Fred Person <fperson@example.com>
gperson@example.com
iperson@example.com
jperson@example.com

Membership changes via email¶

Membership changes such as joining and leaving a mailing list, can be effected via the email interface. The Mailman email commands join, leave, and confirm are used.

Joining a mailing list¶

The mail command join subscribes an email address to the mailing list. subscribe is an alias for join.

>>> from mailman.commands.eml_membership import Join
>>> from mailman.utilities.string import wrap
>>> join = Join()
>>> print(join.name)
join
>>> print(wrap(join.description))
You will be asked to confirm your subscription request and you may be
issued a provisional password.
<BLANKLINE>
By using the 'digest' option, you can specify whether you want digest
delivery or not.  If not specified, the mailing list's default
delivery mode will be used.
>>> print(join.argument_description)
[digest=<no|mime|plain>]

No address to join¶

>>> mlist = create_list('alpha@example.com')
>>> mlist.send_welcome_message = False

When no address argument is given, the message’s From address will be used. If that’s missing though, then an error is returned.

>>> from mailman.runners.command import Results
>>> results = Results()

>>> from mailman.email.message import Message
>>> print(join.process(mlist, Message(), {}, (), results))
ContinueProcessing.no
>>> print(results)
The results of your email command are provided below.

join: No valid address found to subscribe

The subscribe command is an alias.

>>> from mailman.commands.eml_membership import Subscribe
>>> subscribe = Subscribe()
>>> print(subscribe.name)
subscribe
>>> results = Results()
>>> print(subscribe.process(mlist, Message(), {}, (), results))
ContinueProcessing.no
>>> print(results)
The results of your email command are provided below.
<BLANKLINE>
subscribe: No valid address found to subscribe
<BLANKLINE>

Joining the sender¶

When the message has a From field, that address will be subscribed.

>>> msg = message_from_string("""\
... From: Anne Person <anne@example.com>
...
... """)
>>> results = Results()
>>> print(join.process(mlist, msg, {}, (), results))
ContinueProcessing.yes
>>> print(results)
The results of your email command are provided below.
<BLANKLINE>
Confirmation email sent to Anne Person <anne@example.com>
<BLANKLINE>

Anne is not yet a member of the mailing list because she must confirm her subscription request first.

>>> print(mlist.members.get_member('anne@example.com'))
None

Mailman has sent her the confirmation message.

>>> from mailman.testing.helpers import get_queue_messages
>>> items = get_queue_messages('virgin')
>>> len(items)
1
>>> print(items[0].msg.as_string())
MIME-Version: 1.0
...
Subject: confirm ...
From: alpha-confirm+...@example.com
To: anne@example.com
...
<BLANKLINE>
Email Address Registration Confirmation
<BLANKLINE>
Hello, this is the GNU Mailman server at example.com.
<BLANKLINE>
We have received a registration request for the email address
<BLANKLINE>
    anne@example.com
<BLANKLINE>
Before you can start using GNU Mailman at this site, you must first
confirm that this is your email address.  You can do this by replying to
this message, keeping the Subject header intact.
<BLANKLINE>
If you do not wish to register this email address simply disregard this
message.  If you think you are being maliciously subscribed to the list, or
have any other questions, you may contact
<BLANKLINE>
    alpha-owner@example.com
<BLANKLINE>

Anne confirms her registration.

>>> def extract_token(message):
...     return str(message['subject']).split()[1].strip()
>>> token = extract_token(items[0].msg)

>>> from mailman.commands.eml_confirm import Confirm
>>> confirm = Confirm()
>>> msg = message_from_string("""\
... To: alpha-confirm+{token}@example.com
... From: anne@example.com
... Subject: Re: confirm {token}
...
... """.format(token=token))

>>> results = Results()
>>> print(confirm.process(mlist, msg, {}, (token,), results))
ContinueProcessing.yes
>>> print(results)
The results of your email command are provided below.

Confirmed

Anne is now a member of the mailing list.

>>> mlist.members.get_member('anne@example.com')
<Member: Anne Person <anne@example.com>
         on alpha@example.com as MemberRole.member>

Joining a second list¶

>>> mlist_2 = create_list('baker@example.com')
>>> msg = message_from_string("""\
... From: Anne Person <anne@example.com>
...
... """)
>>> print(join.process(mlist_2, msg, {}, (), Results()))
ContinueProcessing.yes

Anne is not a member of the mailing list.

>>> print(mlist_2.members.get_member('anne@example.com'))
None

One Anne confirms this subscription, she becomes a member of the mailing list.

>>> items = get_queue_messages('virgin')
>>> len(items)
1
>>> token = extract_token(items[0].msg)
>>> msg = message_from_string("""\
... To: baker-confirm+{token}@example.com
... From: anne@example.com
... Subject: Re: confirm {token}
...
... """.format(token=token))

>>> results = Results()
>>> print(confirm.process(mlist_2, msg, {}, (token,), results))
ContinueProcessing.yes
>>> print(results)
The results of your email command are provided below.

Confirmed


>>> print(mlist_2.members.get_member('anne@example.com'))
<Member: Anne Person <anne@example.com>
         on baker@example.com as MemberRole.member>

Leaving a mailing list¶

The mail command leave unsubscribes an email address from the mailing list. unsubscribe is an alias for leave.

>>> from mailman.commands.eml_membership import Leave
>>> leave = Leave()
>>> print(leave.name)
leave
>>> print(leave.description)
Leave this mailing list.
<BLANKLINE>
You may be asked to confirm your request.

Anne is a member of the baker@example.com mailing list, when she decides to leave it. She sends a message to the -leave address for the list and is sent a confirmation message for her request.

>>> results = Results()
>>> print(leave.process(mlist_2, msg, {}, (), results))
ContinueProcessing.yes
>>> print(results)
The results of your email command are provided below.
<BLANKLINE>
Anne Person <anne@example.com> left baker@example.com
<BLANKLINE>

Anne is no longer a member of the mailing list.

>>> print(mlist_2.members.get_member('anne@example.com'))
None

Anne does not need to leave a mailing list with the same email address she’s subscribe with. Any of her registered, linked, and validated email addresses will do.

>>> from mailman.interfaces.usermanager import IUserManager
>>> from zope.component import getUtility
>>> anne = getUtility(IUserManager).get_user('anne@example.com')
>>> address = anne.register('anne.person@example.org')

>>> results = Results()
>>> print(mlist.members.get_member('anne@example.com'))
<Member: Anne Person <anne@example.com>
         on alpha@example.com as MemberRole.member>

>>> msg = message_from_string("""\
... To: alpha-leave@example.com
... From: anne.person@example.org
...
... """)

Since Anne’s alternative address has not yet been verified, it can’t be used to unsubscribe Anne from the alpha mailing list.

>>> print(leave.process(mlist, msg, {}, (), results))
ContinueProcessing.no

>>> print(results)
The results of your email command are provided below.

Invalid or unverified email address: anne.person@example.org


>>> print(mlist.members.get_member('anne@example.com'))
<Member: Anne Person <anne@example.com>
         on alpha@example.com as MemberRole.member>

Once Anne has verified her alternative address though, it can be used to unsubscribe her from the list.

>>> from mailman.utilities.datetime import now
>>> address.verified_on = now()

>>> results = Results()
>>> print(leave.process(mlist, msg, {}, (), results))
ContinueProcessing.yes

>>> print(results)
The results of your email command are provided below.

Anne Person <anne.person@example.org> left alpha@example.com


>>> print(mlist.members.get_member('anne@example.com'))
None

Confirmations¶

Bart wants to join the alpha list, so he sends his subscription request.

>>> msg = message_from_string("""\
... From: Bart Person <bart@example.com>
...
... """)

>>> print(join.process(mlist, msg, {}, (), Results()))
ContinueProcessing.yes

There are two messages in the virgin queue, one of which is the confirmation message.

>>> for item in get_queue_messages('virgin'):
...     if str(item.msg['subject']).startswith('confirm'):
...         break
... else:
...     raise AssertionError('No confirmation message')
>>> token = extract_token(item.msg)

Bart replies to the original message, specifically keeping the Subject header intact except for any prefix. Mailman matches the token and confirms Bart as a user of the system.

>>> msg = message_from_string("""\
... From: Bart Person <bart@example.com>
... To: alpha-confirm+{token}@example.com
... Subject: Re: confirm {token}
...
... """.format(token=token))

>>> results = Results()
>>> print(confirm.process(mlist, msg, {}, (token,), results))
ContinueProcessing.yes

>>> print(results)
The results of your email command are provided below.

Confirmed

Now Bart is now a member of the mailing list.

>>> print(mlist.members.get_member('bart@example.com'))
<Member: Bart Person <bart@example.com>
         on alpha@example.com as MemberRole.member>

Dumping queue files¶

The qfile command dumps the contents of a queue pickle file. This is especially useful when you have shunt files you want to inspect.

XXX Test the interactive operation of qfile

Pretty printing¶

By default, the qfile command pretty prints the contents of a queue pickle file to standard output.

>>> from mailman.commands.cli_qfile import QFile
>>> command = QFile()

>>> class FakeArgs:
...     interactive = False
...     doprint = True
...     qfile = []

Let’s say Mailman shunted a message file.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: Uh oh
...
... I borkeded Mailman.
... """)

>>> shuntq = config.switchboards['shunt']
>>> basename = shuntq.enqueue(msg, foo=7, bar='baz', bad='yes')

Once we’ve figured out the file name of the shunted message, we can print it.

>>> from os.path import join
>>> qfile = join(shuntq.queue_directory, basename + '.pck')

>>> FakeArgs.qfile = [qfile]
>>> command.process(FakeArgs)
[----- start pickle -----]
<----- start object 1 ----->
From: aperson@example.com
To: test@example.com
Subject: Uh oh

I borkeded Mailman.

<----- start object 2 ----->
{'_parsemsg': False, 'bad': 'yes', 'bar': 'baz', 'foo': 7, 'version': 3}
[----- end pickle -----]

Maybe we don’t want to print the contents of the file though, in case we want to enter the interactive prompt.

>>> FakeArgs.doprint = False
>>> command.process(FakeArgs)

Command line list removal¶

A system administrator can remove mailing lists by the command line.

>>> create_list('test@example.com')
<mailing list "test@example.com" at ...>

>>> from mailman.interfaces.listmanager import IListManager
>>> from zope.component import getUtility
>>> list_manager = getUtility(IListManager)
>>> list_manager.get('test@example.com')
<mailing list "test@example.com" at ...>

>>> class FakeArgs:
...     quiet = False
...     archives = False
...     listname = ['test@example.com']
>>> args = FakeArgs()

>>> from mailman.commands.cli_lists import Remove
>>> command = Remove()
>>> command.process(args)
Removed list: test@example.com

>>> print(list_manager.get('test@example.com'))
None

You can also remove lists quietly.

>>> create_list('test@example.com')
<mailing list "test@example.com" at ...>

>>> args.quiet = True
>>> command.process(args)

>>> print(list_manager.get('test@example.com'))
None

Getting status¶

The status of the Mailman master process can be queried from the command line. It’s clear at this point that nothing is running.

>>> from mailman.commands.cli_status import Status
>>> status = Status()

>>> class FakeArgs:
...     pass

The status is printed to stdout and a status code is returned.

>>> status.process(FakeArgs)
GNU Mailman is not running
0

We can simulate the master starting up by acquiring its lock.

>>> from flufl.lock import Lock
>>> lock = Lock(config.LOCK_FILE)
>>> lock.lock()

Getting the status confirms that the master is running.

>>> status.process(FakeArgs)
GNU Mailman is running (master pid: ...

We shut down the master and confirm the status.

>>> lock.unlock()
>>> status.process(FakeArgs)
GNU Mailman is not running
0

Unshunt¶

When errors occur while processing email messages, the messages will end up in the shunt queue. The unshunt command allows system administrators to manage the shunt queue.

>>> from mailman.commands.cli_unshunt import Unshunt
>>> command = Unshunt()

>>> class FakeArgs:
...     discard = False

Let’s say there is a message in the shunt queue.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: A broken message
... Message-ID: <aardvark>
...
... """)

>>> shuntq = config.switchboards['shunt']
>>> len(list(shuntq.files))
0
>>> base_name = shuntq.enqueue(msg, {})
>>> len(list(shuntq.files))
1

The unshunt command by default moves the message back to the incoming queue.

>>> inq = config.switchboards['in']
>>> len(list(inq.files))
0

>>> command.process(FakeArgs)

>>> from mailman.testing.helpers import get_queue_messages
>>> items = get_queue_messages('in')
>>> len(items)
1
>>> print(items[0].msg.as_string())
From: aperson@example.com
To: test@example.com
Subject: A broken message
Message-ID: <aardvark>

unshunt moves all shunt queue messages.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: A broken message
... Message-ID: <badgers>
...
... """)
>>> base_name = shuntq.enqueue(msg, {})

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: A broken message
... Message-ID: <crow>
...
... """)
>>> base_name = shuntq.enqueue(msg, {})

>>> len(list(shuntq.files))
2

>>> command.process(FakeArgs)
>>> items = get_queue_messages('in')
>>> len(items)
2

>>> sorted(item.msg['message-id'] for item in items)
['<badgers>', '<crow>']

Return to the original queue¶

While the messages in the shunt queue are generally returned to the incoming queue, if the error occurred while the message was being processed from a different queue, it will be returned to the queue it came from.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: A broken message
... Message-ID: <dingo>
...
... """)

The queue that the message comes from is in message metadata.

>>> base_name = shuntq.enqueue(msg, {}, whichq='bounces')

>>> len(list(shuntq.files))
1
>>> len(list(config.switchboards['bounces'].files))
0

The message is automatically re-queued to the bounces queue.

>>> command.process(FakeArgs)
>>> len(list(shuntq.files))
0
>>> items = get_queue_messages('bounces')
>>> len(items)
1

>>> print(items[0].msg.as_string())
From: aperson@example.com
To: test@example.com
Subject: A broken message
Message-ID: <dingo>

Discarding all shunted messages¶

If you don’t care about the shunted messages, just discard them.

>>> msg = message_from_string("""\
... From: aperson@example.com
... To: test@example.com
... Subject: A broken message
... Message-ID: <elephant>
...
... """)
>>> base_name = shuntq.enqueue(msg, {})

>>> FakeArgs.discard = True
>>> command.process(FakeArgs)

The messages are now gone.

>>> items = get_queue_messages('in')
>>> len(items)
0

Printing the version¶

You can print the Mailman version number.

>>> from mailman.commands.cli_version import Version
>>> command = Version()

>>> command.process(None)
GNU Mailman 3...

Operating on mailing lists¶

The withlist command is a pretty powerful way to operate on mailing lists from the command line. This command allows you to interact with a list at a Python prompt, or process one or more mailing lists through custom made Python functions.

XXX Test the interactive operation of withlist

Getting detailed help¶

Because withlist is so complex, you need to request detailed help.

>>> from mailman.commands.cli_withlist import Withlist
>>> command = Withlist()

>>> class FakeArgs:
...     interactive = False
...     run = None
...     details = True
...     listname = []

>>> class FakeParser:
...     def error(self, message):
...         print(message)
>>> command.parser = FakeParser()

>>> args = FakeArgs()
>>> command.process(args)
This script provides you with a general framework for interacting with a
mailing list.
...

Running a command¶

By putting a Python function somewhere on your sys.path, you can have withlist call that function on a given mailing list. The function takes a single argument, the mailing list.

>>> import os, sys
>>> old_path = sys.path[:]
>>> sys.path.insert(0, config.VAR_DIR)

>>> with open(os.path.join(config.VAR_DIR, 'showme.py'), 'w') as fp:
...     print("""\
... def showme(mailing_list):
...     print("The list's name is", mailing_list.fqdn_listname)
...
... def displayname(mailing_list):
...     print("The list's display name is", mailing_list.display_name)
... """, file=fp)

If the name of the function is the same as the module, then you only need to name the function once.

>>> mlist = create_list('aardvark@example.com')
>>> args.details = False
>>> args.run = 'showme'
>>> args.listname = 'aardvark@example.com'
>>> command.process(args)
The list's name is aardvark@example.com

The function’s name can also be different than the modules name. In that case, just give the full module path name to the function you want to call.

>>> args.run = 'showme.displayname'
>>> command.process(args)
The list's display name is Aardvark

Multiple lists¶

You can run a command over more than one list by using a regular expression in the listname argument. To indicate a regular expression is used, the string must start with a caret.

>>> mlist_2 = create_list('badger@example.com')
>>> mlist_3 = create_list('badboys@example.com')

>>> args.listname = '^.*example.com'
>>> command.process(args)
The list's display name is Aardvark
The list's display name is Badboys
The list's display name is Badger

>>> args.listname = '^bad.*'
>>> command.process(args)
The list's display name is Badboys
The list's display name is Badger

>>> args.listname = '^foo'
>>> command.process(args)

Error handling¶

You get an error if you try to run a function over a non-existent mailing list.

>>> args.listname = 'mystery@example.com'
>>> command.process(args)
No such list: mystery@example.com

You also get an error if no mailing list is named.

>>> args.listname = None
>>> command.process(args)
--run requires a mailing list name

IPython¶

You can use IPython as the interactive shell by changing certain configuration variables in the [shell] section of your mailman.cfg file. Set use_ipython to “yes” to switch to IPython, which must be installed on your system.

Other configuration variables in the [shell] section can be used to configure other aspects of the interactive shell. You can change both the prompt and the banner.

Mailman - The GNU Mailing List Management System¶

Here is a history of user visible changes to Mailman.

3.0.4 – “Show Don’t Tell”¶

(2016-XX-XX)

3.0.3 – “Show Don’t Tell”¶

(2016-03-29)

Bugs¶

When approving a subscription request via the REST API, for a user who is already a member, return an HTTP 409 Conflict code instead of the previous server traceback (and resulting HTTP 500 code). (Closes: #193)

Trying to subscribe an address as a list owner (or moderator or nonmember) which is already subscribed with that role produces a server error. Originally given by Anirudh Dahiya. (Closes #198)

Cross-posting messages held on both lists no longer fails. (Closes #176)

Don’t let unknown charsets crash the “approved” rule. Given by Aurélien Bompard. (Closes #203)

Don’t let crashes in IArchiver plugins break handlers or runners. (Closes #208)

Fix “None” as display name in welcome message. Given by Aditya Divekar. (Closes #194)

Pin the Falcon version to < 1.0.

Fix mailman shell processing of $PYTHONSTARTUP. (Closes #224)

3.0.2 – “Show Don’t Tell”¶

(2016-02-08)

Bugs¶

Fix IntegrityErrors raised under PostreSQL when deleting users and addresses. Given by Aurélien Bompard.

Fix traceback in approved handler when the moderator password is None. Given by Aurélien Bompard.

Allow mailing lists to have localhost names with a suffix matching the subcommand extensions. Given by Aurélien Bompard. (Closes: #168)

Don’t traceback if a nonexistent message-id is deleted from the message store. Given by Aurélien Bompard, tweaked by Barry Warsaw. (Closes: #167)

Fix membership query when multiple users are subscribed to a mailing list. Reported by Darrell Kresge. (Closes: #190)

Prevent moderation of messages held for a different list. (Closes: #161)

Interfaces¶

IMessageStore.delete_message() no longer raises a LookupError when you attempt to delete a nonexistent message from the message store.

REST¶

When [devmode]enabled is set, the JSON output is sorted. Given by Aurélien Bompard.

Fixed a number of corner cases for the return codes when PUTing or PATCHing list configuration variables. (Closes: #182)

Don’t crash the REST server when trying to subscribe a user without a preferred address. (Closes: #185)

3.0.1 – “Show Don’t Tell”¶

(2015-11-13)

Bugs¶

When the mailing list’s admin_notify_mchanges is True, the list owners now get the subscription notification. (Closes: #1)

Fix the traceback that occurred when trying to convert a text/html subpart to plaintext via the mimedel handler. Now, a configuration variable [mailman]html_to_plain_text_command in the mailman.cfg file defines the command to use. It defaults to lynx. (Closes: #109)

Confirmation messages should not be Precedence: bulk. (Closes #75)

The prototype archiver is not web accessible so it does not have a list_url or permalink. Given by Aurélien Bompard.

The REST API incorrectly parsed is_server_owner values when given explicitly in the POST that creates a user. (Closes #136)

By POSTing to a user resource with an existing unlinked address, you can link the address to the user. Given by Abhilash Raj.

Fix constraint violations on mailing list deletes affecting PostgreSQL. Given by Abhilash Raj. (Closes #115)

mailman command with no subcommand now prints the help text. Given by Abhilash Raj. (Closes #137)

The MHonArc archiver must set stdin=PIPE when calling the subprocess. Given by Walter Doekes.

For now, treat DeliveryMode.summary_digests the same as .mime_digests. (Closes #141). Also, don’t enqueue a particular digest if there are no recipients for that digest.

For Python versions earlier than 3.5, use a compatibility layer for a backported smtpd module which can accept non-UTF-8 data. (Closes #140)

Bulk emails are now decorated with headers and footers. Given by Aurélien Bompard. (Closes #145)

Core no longer depends on the standalone mock module. (Closes: #146)

Fix the logging of moderation reasons. Given by Aurélien Bompard. Also, update the postauth.txt and postheld.txt templates to not include the bogus URLs, and to include the translated moderation reasons.

Collapse multiple Re: in Subject headers. Given by Mark Sapiro. (Closes: #147)

Added Trove classifiers to setup.py. (Closes: #152)

Fix the processing of subscription confirmation messages when the mailing list is set to confirm-then-moderate. (Closes #114)

Fix pagination values start and total_size in the REST API. Given by Aurélien Bompard. (Closes: #154)

Fix UnicodeEncodeError in the hold chain when sending the authorization email to the mailing list moderators. (Closes: #144)

3.0.0 – “Show Don’t Tell”¶

(2015-04-28)

Architecture¶

Domains now have a list of owners, which are IUser objects, instead of the single contact_address they used to have. IUser objects now also have a is_server_owner flag (defaulting to False) to indicate whether they have superuser privileges. Give by Abhliash Raj, with fixes and refinements by Barry Warsaw. (LP: #1423756)

Mailing list subscription policy work flow has been completely rewritten. It now properly supports email verification and subscription confirmation by the user, and approval by the moderator using unique tokens. IMailingList objects now have a subscription_policy attribute. (LP: #1095552)

Port the REST machinery to Falcon 0.3. (LP: #1446881)

Bugs¶

Fix calculation of default configuration file to use when the $var_dir is created by mailman start. (LP: #1411435)

When creating a user with an email address, do not create the user record if the email address already exists. Given by Andrew Stuart. (LP: #1418280)

When deleting a user via REST, make sure all linked addresses are deleted. Found by Andrew Stuart. (LP: #1419519)

When trying to subscribe an address to a mailing list through the REST API where a case-differing version of the address is already subscribed, return a 409 error instead of a 500 error. Found by Ankush Sharma. (LP: #1425359)

mailman lists --domain was not properly handling its arguments. Given by Manish Gill. (LP: #1166911)

When deleting a user object, make sure their preferences are also deleted. Given by Abhishek. (LP: #1418276)

Be sure a mailing list’s acceptable aliases are deleted when the mailing list itself is deleted. (LP: #1432239)

The built-in example IArchiver implementations now explicitly return None. (LP: #1203359)

The test suite now runs successfully again with PostgreSQL. Given by Aurélien Bompard. (LP: #1435941)

Configuration¶

When specifying a file system path in the [paths.*] section, $cfg_file can be used to expand into the path of the -C option if given. In the default [paths.dev] section, $var_dir is now specified relative to $cfg_file so that it won’t accidentally be relative to the current working directory, if -C is given.

$cwd is now an additional substitution variable for the mailman.cfg file’s [paths.*] sections. A new [paths.here] section is added, which puts the var_dir in $cwd. It is made the default layout.

Documentation¶

Improve the documentation describing how to run Alembic to add new schema migrations. Given by Abhilash Raj.

REST¶

Backward incompatible change: The JSON representation for pending mailing list subscription hold now no longer includes the password key. Also, the address key has been renamed email for consistent terminology and other usage.

You can now view the contents of, inject messages into, and delete messages from the various queue directories via the <api>/queues resource.

You can now DELETE an address. If the address is linked to a user, the user is not delete, it is just unlinked.

A new API is provided to support non-production testing infrastructures, allowing a client to cull all orphaned UIDs via DELETE on <api>/reserved/uids/orphans. Note that no guarantees of API stability will ever be made for resources under reserved. (LP: #1420083)

Domains can now optionally be created with owners; domain owners can be added after the fact; domain owners can be deleted. Also, users now have an is_server_owner flag as part of their representation, which defaults to False, and can be PUT and PATCH’d. Given by Abhilash Raj, with fixes and refinements by Barry Warsaw. (LP: #1423756)

3.0 beta 5 – “Carve Away The Stone”¶

(2014-12-29)

Bugs¶

Fixed Unicode errors in the digest runner and when sending messages to the site owner as a fallback. Given by Aurélien Bompard. (LP: #1130957).

Fixed Unicode errors when a message being added to the digest has non-ascii characters in its payload, but no Content-Type header defining a charset. Given by Aurélien Bompard. (LP: #1170347)

Fixed messages without a text/plain part crashing the Approved rule. Given by Aurélien Bompard. (LP: #1158721)

Fixed getting non-ASCII filenames from RFC 2231 i18n’d messages. Given by Aurélien Bompard. (LP: #1060951)

Fixed AttributeError on MIME digest messages. Given by Aurélien Bompard. (LP: #1130696)

Commands¶

The mailman conf command no longer takes the -t/–sort option; the output is always sorted.

Configuration¶

The [database]migrations_path setting is removed.

Database¶

The ORM layer, previously implemented with Storm, has been replaced by SQLAlchemy, thanks to the fantastic work by Abhilash Raj and Aurélien Bompard. Alembic is now used for all database schema migrations.

The new logger mailman.database logs any errors at the database layer.

Development¶

Python 3.4 is now the minimum requirement.

You no longer have to create a virtual environment separately when running the test suite. Just use tox.

You no longer have to edit src/mailman/testing/testing.cfg to run the test suite against PostgreSQL. See src/mailman/docs/START.rst for details.

Interfaces¶

The RFC 2369 headers added to outgoing messages are now added in sorted order.

Several changes to the internal API:

IListManager.mailing_lists is guaranteed to be sorted in List-ID order.

IDomains.mailing_lists is guaranteed to be sorted in List-ID order.

Iteration over domains via the IDomainManager is guaranteed to be sorted by IDomain.mail_host order.

ITemporaryDatabase interface and all implementations are removed.

REST¶

The Falcon Framework has replaced restish as the REST layer. This is an internal change only.

The JSON representation http_etag key uses an algorithm that is insensitive to Python’s dictionary sort order.

The address resource now has an additional ‘/user’ sub-resource which can be used to GET the address’s linked user if there is one. This sub-resource also supports POST to link an unlinked address (with an optional ‘auto_create’ flag), and PUT to link the address to a different user. It also supports DELETE to unlink the address. (LP: #1312884) Given by Aurélien Bompard based on work by Nicolas Karageuzian.

The /3.0/system path is deprecated; use /3.0/system/versions to get the system version information.

You can access the system configuration via the resource path /3.0/system/configuration/<section>. This returns a dictionary with the keys being the section’s variables and the values being their value from mailman.cfg as verbatim strings. You can get a list of all section names via /3.0/system/configuration which returns a dictionary containing the http_etag and the section names as a sorted list under the sections key. The system configuration resource is read-only.

Member resource JSON now include the member_id as a separate key.

3.0 beta 4 – “Time and Motion”¶

(2014-04-22)

Development¶

Mailman 3 no longer uses zc.buildout and tests are now run by the nose2 test runner. See src/mailman/docs/START.rst for details on how to build Mailman and run the test suite. Also, use -P to select a test pattern and -E to enable stderr debugging in runners.

Use the enum34 package instead of flufl.enum.

Use setuptools instead of distribute, since the latter is defunct.

REST¶

Add reply_to_address and first_strip_reply_to as writable attributes of a mailing list’s configuration. (LP: #1157881)

Support pagination of some large collections (lists, users, members). [Florian Fuchs] (LP: #1156529)

Expose hide_address to the .../preferences REST API. [Sneha Priscilla.] (LP: #1203519)

Mailing lists can now individually enable or disable any archiver available site-wide. [Joanna Skrzeszewska] (LP: #1158040)

Addresses can be added to existing users, including display names, via the REST API. [Florian Fuchs]

Fixed a crash in the REST server when searching for nonmembers via /find which we’ve never seen before, because those members only have an address record, not a user record. This requires a small change in the API where the JSON response’s address key now contains the URL to the address resource, the new email key contains the email address as a string, and the user key is optional.

Commands¶

mailman conf now has a -t/–sort flag which sorts the output by section and then key. [Karl-Aksel Puulmann and David Soto] (LP: 1162492)

Greatly improve the fidelity of the Mailman 2.1 list importer functionality (i.e. mailman import21). [Aurélien Bompard].

Configuration¶

Add support for the Exim 4 MTA. [Stephen Turnbull]

When creating the initial file system layout in var, e.g. via bin/mailman info, add an var/etc/mailman.cfg file if one does not already exist. Also, when initializing the system, look for that file as the configuration file, just after ./mailman.cfg and before ~/.mailman.cfg. (LP: #1157861)

Database¶

The bounceevent table now uses list-ids to cross-reference the mailing list, to match other tables. Similarly for the IBounceEvent interface.

Added a listarchiver table to support list-specific archivers.

Bugs¶

Non-queue runners should not create var/queue subdirectories. [Sandesh Kumar Agrawal] (LP: #1095422)

Creation of lists with upper case names should be coerced to lower case. (LP: #1117176)

Fix REST server crash on mailman reopen due to no interception of signals. (LP: #1184376)

Add subject_prefix to the IMailingList interface, and clarify the docstring for display_name. (LP: #1181498)

Fix importation from MM2.1 to MM3 of the archive policy. [Aurélien Bompard] (LP: #1227658)

Fix non-member moderation rule to prefer a member sender if both members and non-members are in the message’s sender list. [Aurélien Bompard] (LP: #1291452)

Fix IntegrityError (against PostgreSQL) when deleting a list with content filters. [Aurélien Bompard] (LP: #1117174)

Fix test isolation bug in languages.rst. [Piotr Kasprzyk] (LP: #1308769)

3.0 beta 3 – “Here Again”¶

(2012-12-31)

Compatibility¶

Python 2.7 is now required. Python 2.6 is no longer officially supported. The code base is now also python2.7 -3 clean, although there are still some warnings in 3rd party dependencies. (LP: #1073506)

REST¶

API change: The JSON representation for held messages no longer includes the data key. The values in this dictionary are flatted into the top-level JSON representation. The key key is remove since it’s redundant. Use message_id for held messages, and address for held subscriptions/unsubscriptions. The following _mod_* keys are inserted without the _mod_ prefix:

_mod_subject -> subject

_mod_hold_date -> hold_date

_mod_reason -> reason

_mod_sender -> sender

_mod_message_id -> message_id

List styles are supported through the REST API. Get the list of available styles (by name) via …/lists/styles. Create a list in a specific style by using POST data style_name=<style>. (LP: #975692)

Allow the getting/setting of IMailingList.subject_prefix via the REST API (given by Terri Oda). (LP: #1062893)

Expose a REST API for membership change (subscriptions and unsubscriptions) moderation. (LP: #1090753)

Add list_id to JSON representation for a mailing list (given by Jimmy Bergman).

The canonical resource for a mailing list (and thus its self_link) is now the URL with the list-id. To reference a mailing list, the list-id url is preferred, but for backward compatibility, the posting address is still accepted.

You can now PUT and PATCH on user resources to change the user’s display name or password. For passwords, you pass in the clear text password and Mailman will hash it before storing.

You can now verify and unverify an email address through the REST API. POST to …/addresses/<email>/verify and …/addresses/<email>/unverify respectively. The POST data is ignored. It is not an error to verify or unverify an address more than once, but verifying an already verified address does not change its .verified_on date. (LP: #1054730)

Deleting a user through the REST API also deletes all the user’s linked addresses and memberships. (LP: #1074374)

A user’s password can be verified by POSTing to …/user/<id>/login. The data must contain a single parameter cleartext_password and if this matches, a 204 (No Content) will be returned, otherwise a 403 (Forbidden) is returned. (LP: #1065447)

Configuration¶

[passlib]path configuration variable renamed to [passlib]configuration.

Postfix-specific configurations in the [mta] section are moved to a separate file, named by the [mta]configuration variable.

In the new postfix.cfg file, postfix_map_cmd is renamed to postmap_command.

The default list style is renamed to legacy-default and a new legacy-announce style is added. This is similar to the legacy-default except set up for announce-only lists.

Database¶

The ban table now uses list-ids to cross-reference the mailing list, since these cannot change even if the mailing list is moved or renamed.

The following columns were unused and have been removed:

mailinglist.new_member_options

mailinglist.send_reminders

mailinglist.subscribe_policy

mailinglist.unsubscribe_policy

mailinglist.subscribe_auto_approval

mailinglist.private_roster

mailinglist.admin_member_chunksize

Interfaces¶

The IBanManager is no longer a global utility. Instead, you adapt an IMailingList to an IBanManager to manage the bans for a specific mailing list. To manage the global bans, adapt None.

Commands¶

bin/mailman aliases loses the –output, –format, and –simple arguments, and adds a –directory argument. This is necessary to support the Postfix relay_domains support.

bin/mailman start was passing the wrong relative path to its runner subprocesses when -C was given. (LP: #982551)

bin/runner command has been simplified and its command line options reduced. Now, only one -r/–runner option may be provided and the round-robin feature has been removed.

Other¶

Added support for Postfix relay_domains setting for better virtual domain support. [Jimmy Bergman].

Two new events are triggered on membership changes: SubscriptionEvent when a new member joins a mailing list, and an UnsubscriptionEvent when a member leaves a mailing list. (LP: #1047286)

Improve the –help text for the start, stop, restart, and reopen subcommands. (LP: #1035033)

Bugs¶

Fixed send_goodbye_message(). (LP: #1091321)

Fixed REST server crash on reopen command. Identification and test provided by Aurélien Bompard. (LP: #1184376)

3.0 beta 2 – “Freeze”¶

(2012-09-05)

Architecture¶

The link between members and the mailing lists they are subscribed to, is now via the RFC 2369 list_id instead of the fqdn listname (i.e. posting address). This is because while the posting address can change if the mailing list is moved to a new server, the list id is fixed. (LP: #1024509)

IListManager.get_by_list_id() added.

IListManager.list_ids added.

IMailingList.list_id added.

Several internal APIs that accepted fqdn list names now require list ids, e.g. ISubscriptionService.join() and .find_members().

IMember.list_id attribute added; .mailing_list is now an alias that retrieves and returns the IMailingList.

passlib is now used for all password hashing instead of flufl.password. The default hash is sha512_crypt. (LP: #1015758)

Internally, all datetimes are kept in the UTC timezone, however because of LP: #280708, they are stored in the database in naive format.

received_time is now added to the message metadata by the LMTP runner instead of by Switchboard.enqueue(). This latter no longer depends on received_time in the metadata.

The ArchiveRunner no longer acquires a lock before it calls the individual archiver implementations, since not all of them need a lock. If they do, the implementations must acquire said lock themselves.

The news runner and queue has been renamed to the more accurate nntp. The runner has also been ported to Mailman 3 (LP: #967409). Beta testers can safely remove $var_dir/queue/news.

A mailing list’s moderator password is no longer stored in the clear; it is hashed with the currently selected scheme.

An AddressVerificationEvent is triggered when an IAddress is verified or unverified. (LP: #975698)

A PasswordChangeEvent is triggered when an IUser’s password changes. (LP: #975700)

When a queue runner gets an exception in its _dispose() method, a RunnerCrashEvent is triggered, which contains references to the queue runner, mailing list, message, metadata, and exception. Interested parties can subscribe to that zope.event for notification.

Events renamed and moved: * mailman.chains.accept.AcceptNotification * mailman.chains.base.ChainNotification * mailman.chains.discard.DiscardNotification * mailman.chains.hold.HoldNotification * mailman.chains.owner.OwnerNotification * mailman.chains.reject.RejectNotification changed to (respectively): * mailman.interfaces.chains.AcceptEvent * mailman.interfaces.chains.ChainEvent * mailman.interfaces.chains.DiscardEvent * mailman.interfaces.chains.HoldEvent * mailman.interfaces.chains.AcceptOwnerEvent * mailman.interfaces.chains.RejectEvent

A ConfigurationUpdatedEvent is triggered when the system-wide global configuration stack is pushed or popped.

The policy for archiving has now been collapsed into a single enum, called ArchivePolicy. This describes the three states of never archive, archive privately, and archive_publicly. (LP: #967238)

Database¶

Schema migrations (LP: #971013)

mailinglist.include_list_post_header -> allow_list_posts

mailinglist.news_prefix_subject_too -> nntp_prefix_subject_too

mailinglist.news_moderation -> newsgroup_moderation

mailinglist.archive and mailinglist.archive_private have been collapsed into archive_policy.

mailinglist.nntp_host has been removed.

mailinglist.generic_nonmember_action has been removed (LP: #975696)

Schema migrations (LP: #1024509) - member.mailing_list -> list_id

The PostgreSQL port of the schema accidentally added a moderation_callback column to the mailinglist table. Since this is unused in Mailman, it was simply commented out of the base schema for PostgreSQL.

REST¶

Expose archive_policy in the REST API. Contributed by Alexander Sulfrian. (LP: #1039129)

Configuration¶

New configuration variables clobber_date and clobber_skew supported in every [archiver.<name>] section. These are used to determine under what circumstances a message destined for a specific archiver should have its Date: header clobbered. (LP: #963612)

With the switch to passlib, [passwords]password_scheme has been removed. Instead use [passwords]path to specify where to find the passlib.cfg file. See the comments in schema.cfg for details.

Configuration schema variable changes: * [nntp]username -> [nntp]user * [nntp]port (added)

Header check specifications in the mailman.cfg file have changed quite bit. The previous [spam.header.foo] sections have been removed. Instead, there’s a new [antispam] section that contains a header_checks variable. This variable takes multiple lines of Header: regexp values, one per line. There is also a new jump_chain variable which names the chain to jump to should any of the header checks (including the list-specific, and programmatically added ones) match.

Documentation¶

Some additional documentation on related components such as Postorius and hyperkitty have been added, given by Stephen J Turnbull.

Bug fixes¶

Fixed the RFC 1153 digest footer to be compliant. (LP: #887610)

Fixed a UnicodeError with non-ascii message bodies in the approved rule, given by Mark Sapiro. (LP: #949924)

Fixed a typo when returning the configuration file’s header match checks. (LP: #953497)

List-Post should be NO when posting is not allowed. (LP: #987563)

Non-unicode values in msgdata broke pending requests. (LP: #1031391)

Show devmode in bin/mailman info output. (LP: #1035028)

Fix residual references to the old IMailingList archive variables. (LP: #1031393)

3.0 beta 1 – “The Twilight Zone”¶

(2012-03-23)

Architecture¶

Schema migrations have been implemented.

Implement the style manager as a utility instead of an attribute hanging off the mailman.config.config object.

PostgreSQL support contributed by Stephen A. Goss. (LP: #860159)

Separate out the RFC 2369 header adding handler.

Dynamically calculate the List-Id header instead of storing it in the database. This means it cannot be changed.

Major redesign of the template search system, fixing LP: #788309. $var_dir is now used when search for all template overrides, site, domain, or mailing list. The in-tree English templates are used only as a last fallback.

Support downloading templates by URI, including mailman:// URIs. This is used in welcome and goodbye messages, as well as regular and digest headers and footers, and supports both language and mailing list specifications. E.g. mailman:///test@example.com/it/welcome.txt

$user_password is no longer supported as a placeholder in headers and footers.

Mailing lists get multiple chains and pipelines. For example, normal postings go through the posting_chain while messages to owners to through owners_chain. The default built-in chain is renamed to default-posting-chain while the built-in pipeline is renamed default-posting-pipeline.

The experimental maildir runner is removed. Use LMTP.

The LMTP server now requires that the incoming message have a Message-ID, otherwise it rejects the message with a 550 error. Also, the LMTP server adds the X-Message-ID-Hash header automatically. The inject cli command will also add the X-Message-ID-Hash header, but it will craft a Message-ID header first if one is missing from the injected text. Also, inject will always set the correct value for the original_size attribute on the message object, instead of trusting a possibly incorrect value if it’s already set. The individual IArchiver implementations no longer set the X-Message-ID-Hash header.

The Prototype archiver now stores its files in maildir format inside of $var_dir/archives/prototype, given by Toshio Kuratomi.

Improved “8 mile high” document distilled by Stephen J Turnbull from the Pycon 2012 Mailman 3 sprint. Also improvements to the Sphinx build given by Andrea Crotti (LP: #954718).

Pipermail has been eradicated.

Configuration variable [mailman]filtered_messages_are_preservable controls whether messages which have their top-level Content-Type filtered out can be preserved in the bad queue by list owners.

Configuration section [scrubber] removed, as is the scrubber handler. This handler was essentially incompatible with Mailman 3 since it required coordination with Pipermail to store attachments on disk.

Database¶

Schema changes: - welcome_msg -> welcome_message_uri - goodbye_msg -> goodbye_message_uri - send_welcome_msg -> send_welcome_message - send_goodbye_msg -> send_goodbye_message - msg_header -> header_uri - msg_footer -> footer_uri - digest_header -> digest_header_uri - digest_footer -> digest_footer_uri - start_chain -> posting_chain - pipeline -> posting_pipeline - real_name -> display_name (mailinglist, user, address)

Schema additions: - mailinglist.filter_action - mailinglist.owner_chain - mailinglist.owner_pipeline

REST¶

Held messages can now be moderated through the REST API. Mailing list resources now accept a held path component. GETing this returns all held messages for the mailing list. POSTing to a specific request id under this url can dispose of the message using Action enums.

Mailing list resources now have a member_count attribute which gives the number of subscribed members. Given by Toshio Kuratomi.

Interfaces¶

Add property IUserManager.members to return all IMembers in the system.

Add property IListmanager.name_components which returns 2-tuples for every mailing list as (list_name, mail_host).

Remove previously deprecated IListManager.get_mailing_lists().

IMailTransportAgentAliases now explicitly accepts duck-typed arguments.

IRequests interface is removed. Now just use adaptation from IListRequests directly (which takes an IMailingList object).

handle_message() now allows for Action.hold which is synonymous with Action.defer (since the message is already being held).

IListRequests.get_request() now takes an optional request_type argument to narrow the search for the given request.

New ITemplateLoader utility.

ILanguageManager.add() returns the ILanguage object just created.

IMailinglist.decorators removed; it was unused

IMailingList.real_name -> IMailingList.display_name

IUser.real_name -> IUser.display_name

IAddress.real_name -> IAddress.display_name

Add property IRoster.member_count.

Commands¶

IPython support in bin/mailman shell contributed by Andrea Crotti. (LP: #949926).

The mailman.cfg configuration file will now automatically be detected if it exists in an etc directory which is a sibling of argv0.

bin/mailman shell is an alias for withlist.

The confirm email command now properly handles Re:-like prefixes, even if they contain non-ASCII characters. (LP: #685261)

The join email command no longer accepts an address= argument. Its digest= argument now accepts the following values: no (for regular delivery), mime, or plain.

Added a help email command.

A welcome message is sent when the user confirms their subscription via email.

Global -C option now accepts an absolute path to the configuration file. Given by Andrea Crotti. (LP: #953707)

Bug fixes¶

Subscription disabled probe warning notification messages are now sent without a Precedence: header. Given by Mark Sapiro. (LP: #808821)

Fixed KeyError in retry runner, contributed by Stephen A. Goss. (LP: #872391)

Fixed bogus use of bounce_processing attribute (should have been process_bounces, with thanks to Vincent Fretin. (LP: #876774)

Fix test_moderation for timezones east of UTC+0000, given by blacktav. (LP: #890675)

3.0 alpha 8 – “Where’s My Thing?”¶

(2011-09-23)

Architecture¶

Factor out bounce detection to flufl.bounce.

Unrecognized bounces can now also be forwarded to the site owner.

mailman.qrunner log is renamed to mailman.runner

master-qrunner.lck -> master.lck

master-qrunner.pid -> master.pid

Four new events are created, and notifications are sent during mailing list lifecycle changes: - ListCreatingEvent - sent before the mailing list is created - ListCreatedEvent - sent after the mailing list is created - ListDeletingEvent - sent before the mailing list is deleted - ListDeletedEvent - sent after the mailing list is deleted

Four new events are created, and notifications are sent during domain lifecycle changes: - DomainCreatingEvent - sent before the domain is created - DomainCreatedEvent - sent after the domain is created - DomainDeletingEvent - sent before the domain is deleted - DomainDeletedEvent - sent after the domain is deleted

Using the above events, when a domain is deleted, associated mailing lists are deleted. (LP: #837526)

IDomain.email_host -> .mail_host (LP: #831660)

User and Member ids are now proper UUIDs.

Improved the way enums are stored in the database, so that they are more explicitly expressed in the code, and more database efficient.

REST¶

Preferences for addresses, users, and members can be accessed, changed, and deleted through the REST interface. Hierarchical, combined preferences for members, and system preferences can be read through the REST interface. (LP: #821438)

The IMailingList attribute host_name has been renamed to mail_host for consistency. This changes the REST API for mailing list resources. (LP: #787599)

New REST resource http://…/members/find can be POSTed to in order to find member records. Optional arguments are subscriber (email address to search for), fqdn_listname, and role (i.e. MemberRole). (LP: #799612)

You can now query or change a member’s delivery_mode attribute through the REST API (LP: #833132). Given by Stephen A. Goss.

New REST resource http://…/<domain>/lists can be GETed in order to find all the mailing lists in a specific domain (LP: #829765). Given by Stephen A. Goss.

Fixed /lists/<fqdn_listname>/<role>/<email> (LP: #825570)

Remove role plurals from /lists/<fqdn_listname/rosters/<role>

Fixed incorrect error code for /members/<bogus> (LP: #821020). Given by Stephen A. Goss.

DELETE users via the REST API. (LP: #820660)

Moderators and owners can be added via REST (LP: #834130). Given by Stephen A. Goss.

Getting the roster or configuration of a nonexistent list did not give a 404 error (LP: #837676). Given by Stephen A. Goss.

PATCHing an invalid attribute on a member did not give a 400 error (LP: #833376). Given by Stephen A. Goss.

Getting the memberships for a non-existent address did not give a 404 error (LP: #848103). Given by Stephen A. Goss.

Commands¶

bin/qrunner is renamed to bin/runner.

bin/mailman aliases gains -f and -s options.

bin/mailman create no longer allows a list to be created with bogus owner addresses. (LP: #778687)

bin/mailman start –force option is fixed. (LP: #869317)

Documentation¶

Update the COPYING file to contain the GPLv3. (LP: #790994)

Major terminology change: ban the terms “queue runners” and “qrunners” since not all runners manage queue directories. Just call them “runners”. Also, the master is now just called “the master runner”.

Testing¶

New configuration variable in [devmode] section, called wait which sets the timeout value used in the test suite for starting up subprocesses.

Handle SIGTERM in the REST server so that the test suite always shuts down correctly. (LP: #770328)

Other bugs and changes¶

Moderating a message with Action.accept now sends the message. (LP: #827697)

Fix AttributeError triggered by i18n call in autorespond_to_sender() (LP: #827060)

Local timezone in X-Mailman-Approved-At caused test failure. (LP: #832404)

InvalidEmailAddressError no longer repr()’s its value.

Rewrote a test for compatibility between Python 2.6 and 2.7. (LP: #833208)

Fixed Postfix alias file generation when more than one mailing list exists. (LP: #874929). Given by Vincent Fretin.

3.0 alpha 7 – “Mission”¶

(2011-04-29)

Architecture¶

Significant updates to the subscription model. Members can now subscribe with a preferred address, and changes to that will be immediately reflected in mailing list subscriptions. Users who subscribe with an explicit address can easily change to a different address, as long as that address is verified. (LP: #643949)

IUsers and IMembers are now assigned a unique, random, immutable id.

IUsers now have created_on and .preferred_address properties.

IMembers now have a .user attribute for easy access to the subscribed user.

When created with add_member(), passwords are always stored encrypted.

In all interfaces, “email” refers to the textual email address while “address” refers to the IAddress object.

mailman.chains.base.Chain no longer self registers.

New member and nonmember moderation rules and chains. This effectively ports moderation rules from Mailman 2 and replaces attributes such as member_moderation_action, default_member_moderation, and generic_nonmember_action. Now, nonmembers exist as subscriptions on a mailing list and members have a moderation_action attribute which describes the disposition for postings from that address.

Member.is_moderated was removed because of the above change.

default_member_action and default_nonmember_action were added to mailing lists.

All sender addresses are registered (unverified) with the user manager by the incoming queue runner. This way, nonmember moderation rules will always have an IAddress that they can subscribe to the list (as MemberRole.nonmember).

Support for SMTP AUTH added via smtp_user and smtp_pass configuration variables in the [mta] section. (LP: #490044)

IEmailValidator interface for pluggable validation of email addresses.

.subscribe() is moved from the IAddress to the IMailingList

IAddresses get their registered_on attribute set when the object is created.

Configuration¶

[devmode] section gets a new ‘testing’ variable.

Added password_scheme and password_length settings for defining the default password encryption scheme.

creator_pw_file and site_pw_file are removed.

Commands¶

‘bin/mailman start’ does a better job of producing an error when Mailman is already running.

‘bin/mailman status’ added for providing command line status on the master queue runner watcher process.

‘bin/mailman info’ now prints the REST root url and credentials.

mmsitepass removed; there is no more site password.

REST¶

Add Basic Auth support for REST API security. (Jimmy Bergman)

Include the fqdn_listname and email address in the member JSON representation.

Added reply_goes_to_list, send_welcome_msg, welcome_msg, default_member_moderation to the mailing list’s writable attributes in the REST service. (Jimmy Bergman)

Expose the new membership model to the REST API. Canonical member resource URLs are now much shorter and live in their own top-level namespace instead of within the mailing list’s namespace.

/addresses/<email>/memberships gets all the memberships for a given email address.

/users is a new top-level URL under which user information can be accessed. Posting to this creates new users.

Users can subscribe to mailing lists through the REST API.

Domains can be deleted via the REST API.

PUT and PATCH to a list configuration now returns a 204 (No Content).

Build¶

Support Python 2.7. (LP: #667472)

Disable site-packages in buildout.cfg because of LP: #659231.

Don’t include eggs/ or parts/ in the source tarball. (LP: #656946)

flufl.lock is now required instead of locknix.

Bugs fixed¶

Typo in scan_message(). (LP: #645897)

Typo in add_member(). (LP: #710182) (Florian Fuchs)

Re-enable bounce detectors. (LP: #756943)

Clean up many pyflakes problems; ditching pylint.

3.0 alpha 6 – “Cut to the Chase”¶

(2010-09-20)

Commands¶

The functionality of ‘bin/list_members’ has been moved to ‘bin/mailman members’.

‘bin/mailman info’ -v/–verbose output displays the file system layout paths Mailman is currently configured to use.

Configuration¶

You can now configure the paths Mailman uses for queue files, lock files, data files, etc. via the configuration file. Define a file system ‘layout’ and then select that layout in the [mailman] section. Default layouts include ‘local’ for putting everything in /var/tmp/mailman, ‘dev’ for local development, and ‘fhs’ for Filesystem Hierarchy Standard 2.3 (LP #490144).

Queue file directories now live in $var_dir/queues.

REST¶

lazr.restful has been replaced by restish as the REST publishing technology used by Mailman.

New REST API for getting all the members of a roster for a specific mailing list.

New REST API for getting and setting a mailing list’s configuration. GET and PUT are supported to retrieve the current configuration, and set all the list’s writable attributes in one request. PATCH is supported to partially update a mailing list’s configuration. Individual options can be set and retrieved by using subpaths.

Subscribing an already subscribed member via REST now returns a 409 HTTP error. LP: #552917

Fixed a bug when deleting a list via the REST API. LP: #601899

Architecture¶

X-BeenThere header is removed.

Mailman no longer touches the Sender or Errors-To headers.

Chain actions can now fire Zope events in their _process() implementations.

Environment variable $MAILMAN_VAR_DIR can be used to control the var/ directory for Mailman’s runtime files. New environment variable $MAILMAN_UNDER_MASTER_CONTROL is used instead of the qrunner’s –subproc/-s option.

Miscellaneous¶

Allow X-Approved and X-Approve headers, equivalent to Approved and Approve. LP: #557750

Various test failure fixes. LP: #543618, LP: #544477

List-Post header is retained in MIME digest messages. LP: #526143

Importing from a Mailman 2.1.x list is partially supported.

3.0 alpha 5 – “Distant Early Warning”¶

(2010-01-18)

REST¶

Add REST API for subscription services. You can now:

list all members in all mailing lists

subscribe (and possibly register) an address to a mailing list

unsubscribe an address from mailing list

Commands¶

‘bin/dumpdb’ is now ‘bin/mailman qfile’

‘bin/unshunt’ is now ‘bin/mailman unshunt’

Mailman now properly handles the ‘-join’, ‘-leave’, and ‘-confirm’ email commands and sub-addresses. ‘-subscribe’ and ‘-unsubscribe’ are aliases for ‘-join’ and ‘-leave’ respectively.

Configuration¶

devmode settings now live in their own [devmode] section.

Mailman now searches for a configuration file using this search order. The first file that exists is used.

-C config command line argument

$MAILMAN_CONFIG_FILE environment variable

./mailman.cfg

~/.mailman.cfg

/etc/mailman.cfg

3.0 alpha 4 – “Vital Signs”¶

(2009-11-28)

Commands¶

‘bin/inject’ is now ‘bin/mailman inject’, with some changes

‘bin/mailmanctl’ is now ‘bin/mailman start|stop|reopen|restart’

‘bin/mailman version’ is added (output same as ‘bin/mailman –version’)

‘bin/mailman members’ command line arguments have changed. It also now ignores blank lines and lines that start with #. It also no longer quits when it sees an address that’s already subscribed.

‘bin/withlist’ is now ‘bin/mailman withlist’, and its command line arguments have changed.

‘bin/mailman lists’ command line arguments have changed.

‘bin/genaliases’ is now ‘bin/mailman aliases’

Architecture¶

A near complete rewrite of the low-level SMTP delivery machinery. This greatly improves readability, testability, reuse and extensibility. Almost all the old functionality has been retained. The smtp_direct.py handler is gone.

Refactor model objects into the mailman.model subpackage.

Refactor most of the i18n infrastructure into a separate flufl.i18n package.

Switch from setuptools to distribute.

Remove the dependency on setuptools_bzr

Do not create the .mo files during setup.

Configuration¶

All log files now have a ‘.log’ suffix by default.

The substitution placeholders in the verp_format configuration variable have been renamed.

Add a devmode configuration variable that changes some basic behavior. Most importantly, it allows you to set a low-level SMTP recipient for all mail for testing purposes. See also devmode_recipient.

3.0 alpha 3 – “Working Man”¶

(2009-08-21)

Configuration¶

Configuration is now done through lazr.config. Defaults.py is dead. lazr.config files are essentially hierarchical ini files.

Domains are now stored in the database instead of in the configuration file.

pre- and post- initialization hooks are now available to plugins. Specify additional hooks to run in the configuration file.

Add the environment variable $MAILMAN_CONFIG_FILE which overrides the -C command line option.

Make LMTP more compliant with Postfix docs (Patrick Koetter)

Added a NullMTA for mail servers like Exim which just work automatically.

Architecture¶

‘bin/mailman’ is a new super-command for managing Mailman from the command line. Some older bin scripts have been converted, with more to come.

Mailman now has an administrative REST interface which can be used to get information from and manage Mailman remotely.

Back port of Mailman 2.1’s limit on .bak file restoration. After 3 restores, the file is moved to the bad queue, with a .psv extension. (Mark Sapiro)

Digest creation is moved into a new queue runner so it doesn’t block main message processing.

Other changes¶

bin/make_instance is no longer necessary, and removed

The debug log is turned up to info by default to reduce log file spam.

Building and installation¶

All doc tests can now be turned into documentation, via Sphinx. Just run bin/docs after bin/buildout.

3.0 alpha 2 – “Grand Designs”¶

(03-Jan-2009)

Licensing¶

Mailman 3 is now licensed under the GPLv3.

Bug fixes¶

Changed bin/arch to attempt to open the mbox before wiping the old archive. Launchpad bug #280418.

Added digest.mbox and pending.pck to the ‘list’ files checked by check_perms. Launchpad bug #284802.

Architecture¶

Converted to using zope.testing as the test infrastructure. Use bin/test now to run the full test suite. <http://pypi.python.org/pypi/zope.testing/3.7.1>

Partially converted to using lazr.config as the new configuration regime. Not everything has been converted yet, so some manual editing of mailman/Defaults.py is required. This will be rectified in future versions. <http://launchpad.net/lazr.config>

All web-related stuff is moved to its own directory, effectively moving it out of the way for now.

The email command infrastructure has been reworked to play more nicely with the plug-in architecture. Not all commands have yet been converted.

Other changes¶

The LMTP server now properly calculates the message’s original size.

For command line scripts, -C names the configuration file to use. For convenient testing, if -C is not given, then the environment variable MAILMAN_CONFIG_FILE is consulted.

Support added for a local MHonArc archiver, as well as archiving automatically in the remote Mail-Archive.com service.

The permalink proposal for supporting RFC 5064 has been adopted.

Mailing lists no longer have a .web_page_url attribute; this is taken from the mailing list’s domain’s base_url attribute.

Incoming MTA selection is now taken from the config file instead of plugins. An MTA for Postfix+LMTP is added. bin/genaliases works again.

If a message has no Message-ID, the stock archivers will return None for the permalink now instead of raising an assertion.

IArchiver no longer has an is_enabled property; this is taken from the configuration file now.

Installation¶

Python 2.6 is the minimal requirement.

Converted to using zc.buildout as the build infrastructure. See docs/ALPHA.txt for details. <http://pypi.python.org/pypi/zc.buildout/1.1.1>

3.0 alpha 1 – “Leave That Thing Alone”¶

(08-Apr-2008)

User visible changes¶

So called ‘new style’ subject prefixing is the default now, and the only option. When a list’s subject prefix is added, it’s always done so before any Re: tag, not after. E.g. ‘[My List] Re: The subject’.

RFC 2369 headers List-Subscribe and List-Unsubscribe now use the preferred -join and -leave addresses instead of the -request address with a subject value.

Configuration¶

There is no more separate configure; make; make install step. Mailman 3.0 is a setuptools package.

Mailman can now be configured via a ‘mailman.cfg’ file which lives in $VAR_PREFIX/etc. This is used to separate the configuration from the source directory. Alternative configuration files can be specified via -C/–config for most command line scripts. mailman.cfg contains Python code. mm_cfg.py is no more. You do not need to import Defaults.py in etc/mailman.cfg. You should still consult Defaults.py for the list of site configuration variables available to you.

See the etc/mailman.cfg.sample file.

PUBLIC_ARCHIVE_URL and DEFAULT_SUBJECT_PREFIX now takes $-string substitutions instead of %-string substitutions. See documentation in Defaults.py.in for details.

Message headers and footers now only accept $-string substitutions; %-strings are no longer supported. The substitution variable ‘_internal_name’ has been removed; use $list_name or $real_name instead. The substitution variable $fqdn_listname has been added. DEFAULT_MSG_FOOTER in Defaults.py.in has been updated accordingly.

The KNOWN_SPAMMERS global variable is replaced with HEADER_MATCHES. The mailing list’s header_filter_rules variable is replaced with header_matches which has the same semantics as HEADER_MATCHES, but is list-specific.

DEFAULT_MAIL_COMMANDS_MAX_LINES -> EMAIL_COMMANDS_MAX_LINES

All SMTP_LOG_* templates use $-strings and all consistently write the Message-ID as the first item in the log entry.

DELIVERY_MODULE now names a handler, not a module (yes, this is a misnomer, but it will likely change again before the final release).

Architecture¶

Internally, all strings are Unicodes.

Implementation of a chain-of-rules based approach for deciding whether a message should initially be accepted, held for approval, rejected/bounced, or discarded. This replaces most of the disposition handlers in the pipeline. The IncomingRunner now only processes message through the rule chains, and once accepted, places the message in a new queue processed by the PipelineRunner.

Substantially reworked the entire queue runner process management, including mailmanctl, a new master script, and the qrunners. This should be much more robust and reliable now.

The Storm ORM is used for data storage, with the SQLite backend as the default relational database.

Zope interfaces are used to describe the major components.

Users are now stored in a unified database, and shared across all mailing lists.

Mailman’s web interface is now WSGI compliant. WSGI is a Python standard (PEP 333) allowing web applications to be (more) easily integrated with any number of existing Python web application frameworks. For more information see:

http://www.wsgi.org/wsgi http://www.python.org/dev/peps/pep-0333/

Mailman can still be run as a traditional CGI program of course.

Mailman now provides an LMTP server for more efficient integration with supporting mail servers (e.g. Postfix, Sendmail). The Local Mail Transport Protocol is defined in RFC 2033:

http://www.faqs.org/rfcs/rfc2033.html

Virtual domains are now fully supported in that mailing lists of the same name can exist in more than one domain. This is accomplished by renaming the lists/ and archives/ subdirectories after the list’s posting address. For example, data for list foo in example.com and list foo in example.org will be stored in lists/foo@example.com and lists/foo@example.org.

For Postfix or manual MTA users, you will need to regenerate your mail aliases. Use bin/genaliases.

VIRTUAL_HOST_OVERVIEW has been removed, effectively Mailman now operates as if it were always enabled. If your site has more than one domain, you must configure all domains by using add_domain() in your etc/mailman.cfg flie (see below – add_virtual() has been removed).

If you had customizations based on Site.py, you will need to re-implement them. Site.py has been removed.

The site list is no more. You can remove your ‘mailman’ site list unless you want to retain it for other purposes, but it is no longer used (or required) by Mailman. You should set NO_REPLY_ADDRESS to an address that throws away replies, and you should set SITE_OWNER_ADDRESS to an email address that reaches the person ultimately responsible for the Mailman installation. The MAILMAN_SITE_LIST variable has been removed.

qrunners no longer restart on SIGINT; SIGUSR1 is used for that now.

Internationalization Big Changes¶

Translators should work only on messages/<lang>/LC_MESSAGES/mailman.po. Templates files are generated from mailman.po during the build process.

New Features¶

Confirmed member change of address is logged in the ‘subscribe’ log, and if admin_notify_mchanges is true, a notice is sent to the list owner using a new adminaddrchgack.txt template.

There is a new list attribute ‘subscribe_auto_approval’ which is a list of email addresses and regular expressions matching email addresses whose subscriptions are exempt from admin approval. RFE 403066.

Command line scripts¶

Most scripts have grown a -C/–config flag to allow you to specify a different configuration file. Without this, the default etc/mailman.cfg file will be used.

the -V/–virtual-host-overview switch in list_lists has been removed, while -d/–domain and -f/–full have been added.

bin/newlist is renamed bin/create_list and bin/rmlist is renamed bin/remove_list. Both take fully-qualified list names now (i.e. the list’s posting address), but also accept short names, in which case the default domain is used. newlist’s -u/–urlhost and -e/–emailhost switches have been removed. The domain that the list is being added to must already exist.

Backport the ability to specify additional footer interpolation variables by the message metadata ‘decoration-data’ key.

Bug fixes and other patches¶

Removal of DomainKey/DKIM signatures is now controlled by Defaults.py mm_cfg.py variable REMOVE_DKIM_HEADERS (default = No).

Queue runner processing is improved to log and preserve for analysis in the shunt queue certain bad queue entries that were previously logged but lost. Also, entries are preserved when an attempt to shunt throws an exception (1656289).

The processing of Topics regular expressions has changed. Previously the Topics regexp was compiled in verbose mode but not documented as such which caused some confusion. Also, the documentation indicated that topic keywords could be entered one per line, but these entries were not handled properly. Topics regexps are now compiled in non-verbose mode and multi- line entries are ‘ored’. Existing Topics regexps will be converted when the list is updated so they will continue to work.

The List-Help, List-Subscribe, and List-Unsubscribe headers were incorrectly suppressed in messages that Mailman sends directly to users.

The ‘adminapproved’ metadata key is renamed ‘moderator_approved’.

GNU Mailman Coding Style Guide¶

Python coding style guide for GNU Mailman¶

NOTE: The canonical version of this style guide can be found at:

http://barry.warsaw.us/software/STYLEGUIDE.txt

This document contains a style guide for Python programming, as used in GNU Mailman. PEP 8 is the basis for this style guide so it’s recommendations should be followed except for the differences outlined here. This document assumes the use of Python 3.

After file comments (e.g. license block), add an __all__ section that names, one-per-line, all the public names exported by this module. See the GNU Mailman Python template as an example.
Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants, but after any __all__ definitions.

Imports should be grouped, with the order being:
1. non-from imports, grouped from shorted module name to longest module name, with ties being broken by alphabetical order.
2. from-imports grouped alphabetically.
In general, there should be one class per module. This is not a hard-and-fast rule. Keep files small, but it’s okay to group related code together. List everything exported from the module in the __all__.
Right hanging comments are discouraged, in favor of preceding comments. E.g. bad:
```
foo = blarzigop(bar)  # if you don't blarzigop it, it'll shlorp
```
Good:
```
# If you don't blarzigop it, it'll shlorp.
foo = blarzigop(bar)
```
Comments should always be complete sentences, with proper capitalization and full stops at the end.
Major sections of code in a module should be separated by form feed characters (e.g. ^L – that’s a single character control-L not two characters). This helps with Emacs navigation.

Put a ^L before module-level functions, before class definitions, before big blocks of constants which follow imports, and any place else that would be convenient to jump to. Always put two blank lines before a ^L.
Put two blank lines between any top level construct or block of code (e.g. after import blocks). Put only one blank line between methods in a class. No blank lines between the class definition and the first method in the class. No blank lines between a class/method and its docstrings.
Try to minimize the vertical whitespace in a class or function. If you’re inclined to separate stanzas of code for readability, consider putting a comment in describing what the next stanza’s purpose is. Don’t put stupid or obvious comments in just to avoid vertical whitespace though.

Unless internal quote characters would mess things up, the general rule is that single quotes should be used for short strings, double quotes for triple-quoted multi-line strings and docstrings. E.g.:

foo = 'a foo thing'
warn = "Don't mess things up"
notice = """Our three chief weapons are:
         - surprise
         - deception
         - an almost fanatical devotion to the pope
         """

Write docstrings for modules, functions, classes, and methods. Docstrings can be omitted for special methods (e.g. __init__() or __str__()) where the meaning is obvious.
PEP 257 describes good docstrings conventions. Note that most importantly, the “”” that ends a multiline docstring should be on a line by itself, e.g.:
```
"""Return a foobang

Optional plotz says to frobnicate the bizbaz first.
"""
```
For one liner docstrings, keep the closing “”” on the same line.
fill-column for docstrings should be 78.
When testing the emptiness of sequences, use if len(seq) == 0 instead of relying on the falseness of empty sequences. However, if a variable can be one of several false values, it’s okay to just use if seq, though a preceding comment is usually in order.
Always decide whether a class’s methods and instance variables should be public or non-public.

Single leading underscores are generally preferred for non-public attributes. Use double leading underscores only in classes designed for inheritance to ensure that truly private attributes will never name clash. These should be rare.

Public attributes should have no leading or trailing underscores unless they conflict with reserved words, in which case, a single trailing underscore is preferable to a leading one, or a corrupted spelling, e.g. class_ rather than klass.

GNU Mailman Acknowledgments¶

Governance¶

GNU Mailman was invented by John Viega. Barry Warsaw is the current project leader. Aurélien Bompard leads HyperKitty and bundler development. Florian Fuchs leads Postorius development. Development of mailman.client is a group effort.

All project decisions are made by consensus via the Mailman Cabal, er, Steering Committee which can be contacted directly via mailman-cabal@python.org

Core Developers¶

The following folks are or have been core developers of Mailman (in reverse alphabetical order):

Abhilash Raj, Mailman’s Youngest Core Dev
Aurélien Bompard, Pizzaman
Barry Warsaw, Mailman’s yappy guard dog
Florian Fuchs
Harald Meland, Norse Mailman
John Viega, Mailman’s inventor
Ken Manheimer, Mailman’s savior
Mark Sapiro, Mailman’s compulsive responder
Scott Cotton, Cookie-Monster
Stephen J. Turnbull, Standards Otaku
Terri Oda
Thomas Wouters, Mailman’s Dutch treat
Tokio Kikuchi (RIP), Mailman’s weatherman

Copyright Assignees¶

Here is the list of other contributors who have donated large bits of code, and have assigned copyright for contributions to the FSF:

Juan Carlos Rey Anaya
Andrija Arsic
Richard Barrett
Jimmy Bergman
Stephan Berndts
Norbert Bollow
Joe Dugan
Ethan Mindlace Fremen
Ben Gertzfield
Victoriano Giralt
Stephen Goss
Mads Kiilerich
Cedric Knight
Patrick Ben Koetter
The Dragon De Monsyne
Les Niles
Reed O’Brien
Terri Oda
Simone Piunno
Claudia Schmidt
Andreas Schosser
Richard Wackerbarth

Special Thanks¶

Very special thanks to Andrija Arsic for his winning new GNU Mailman logos.

Thanks also go to the following people for their important contributions in other aspects of the Mailman project:

Brad Knowles
JC Dill
Clytie Siddall

Thanks also to Dragon for his original Mailman logo contribution, and to Terri Oda for the neat shortcut icon and the member documentation.

Control.com sponsored development of several Mailman 2.1 features, including topics filters, external membership sources, and initial virtual mailing list support. My thanks especially to Dan Pierson and Ken Crater from Control.com.

Here is the list of other people who have contributed useful ideas, suggestions, bug fixes, testing, etc., or who have been very helpful in answering questions on mailman-users. Please let me know if you have been left off the list!

David Abrahams
William Ahern
Terry Allen
Jose Paulo Moitinho de Almeida
Sven Anderson
Matthias Andree
Anton Antonov
Mike Avery
Stonewall Ballard
Moreno Baricevic
Jimmy Bergman
Jeff Berliner
Stuart Bishop
David Blomquist
Bojan
Søren Bondrup
Grant Bowman
Alessio Bragadini
J. D. Bronson
Stan Bubrouski
Daniel Buchmann
Ben Burnett
Ted Cabeen
Mentor Cana
John Carnes
Julio A. Cartaya
Claudio Cattazzo
Donn Cave
David Champion
Hye-Shik Chang
Eric D. Christensen
Tom G. Christensen
Paul Cox
Stefaniu Criste
Robert Daeley
Ned Dawes
Emilio Delgado
John Dennis
Stefan Divjak
Maximillian Dornseif
Fred Drake
Maxim Dzumanenko
Piarres Beobide Egaña
Rob Ellis
Kerem Erkan
Fil
Patrick Finnerty
Bob Fleck
Erik Forsberg
Darrell Fuhriman
Robert Garrigós
Carson Gaspar
Pascal GEORGE
Vadim Getmanshchuk
David Gibbs
Dmitri I GOULIAEV
Terry Grace
Federico Grau
Pekka Haavisto
David Habben
Stig Hackvan
Jeff Hahn
Terry Hardie
Paul Hebble
Tollef Fog Heen
Peer Heinlein
James Henstridge
Walter Hop
Bert Hubert
Henny Huisman
Jeremy Hylton
Ikeda Soji
Rostyk Ivantsiv
Ron Jarrell
Matthias Juchem
Tamito KAJIYAMA
Nino Katic
SHIGENO Kazutaka
Ashley M. Kirchner
Matthias Klose
Harald Koch
Patrick Koetter
Eddie Kohler
Chris Kolar
Uros Kositer
Andrew Kuchling
Ricardo Kustner
L’homme Moderne
Sylvain Langlade
Ed Lau
J C Lawrence
Greg Lindahl
Christopher P. Lindsey
Martin von Loewis
Dario Lopez-Kästen
Tanner Lovelace
Jay Luker
Gergely Madarasz
Luca Maranzano
John A. Martin
Andrew Martynov
Jason R. Mastaler
Michael Mclay
Michael Meltzer
Marc MERLIN
Nigel Metheringham
Dan Mick
Garey Mills
Martin Mokrejs
Michael Fischer v. Mollard
David Martínez Moreno
Dirk Mueller
Jonas Muerer
Erik Myllymaki
Balazs Nagy
Moritz Naumann
Dale Newfield
Hrvoje Niksic
Les Niles
Mike Noyes
David B. O’Donnell
Timothy O’Malley
“office”
Dan Ohnesorg
Gerald Oskoboiny
Eva Österlind
Toni Panadès
Jon Parise
Chris Pepper
Tim Peters
Joe Peterson
PieterB
Rodolfo Pilas
Skye Poier
Martin Pool
Don Porter
Francesco Potortì
Bob Puff
Abhilash Raj
Michael Ranner
John Read
Sean Reifschneider
Christian Reis
Ademar de Souza Reis, Jr.
Bernhard Reiter
Stephan Richter
Tristan Roddis
Heiko Rommel
Luigi Rosa
Guido van Rossum
Nicholas Russo
Chris Ryan
Cabel Sasser
Bartosz Sawicki
Kai Schaetzl
Karoly Segesdi
Gleydson Mazioli da Silva
Pasi Sjöholm
Chris Snell
Mikhail Sobolev
David Soto
Greg Stein
Dale Stimson
Students of HIT <mailman-cn@mail.cs.hit.edu.cn>
Alexander Sulfrian
Szabolcs Szigeti
Vizi Szilard
David T-G
Owen Taylor
Danny Terweij
Jim Tittsler
Todd (Freedom Lover)
Roger Tsang
Chuq Von Rospach
Jens Vagelpohl
Valia V. Vaneeva
Anti Veeranna
Todd Vierling
Bill Wagner
Greg Ward
Mark Weaver
Kathleen Webb
Florian Weimer
Ousmane Wilane
Dan Wilder
Seb Wills
Dai Xiaoguang
Ping Yeh
YASUDA Yukihiro
Michael Yount
Blair Zajac
Mikhail Zabaluev
Noam Zeilberger
Daniel Zeiss
Todd Zullinger

And everyone else on mailman-developers@python.org and mailman-users@python.org! Thank you, all.