Content filtering

Mailman can filter the content of messages posted to a mailing list by stripping MIME subparts, and possibly reorganizing the MIME structure of a message.

>>> from mailman.app.lifecycle import create_list
>>> mlist = create_list('test@example.com')

Several mailing list options control content filtering. First, the feature must be enabled, then there are two options that control which MIME types get filtered and which get passed. Finally, there is an option to control whether text/html parts will get converted to plain text. Let’s set up some defaults for these variables, then we’ll explain them in more detail below.

>>> mlist.filter_content = True
>>> mlist.filter_types = []
>>> mlist.pass_types = []
>>> mlist.filter_extensions = []
>>> mlist.pass_extensions = []
>>> mlist.convert_html_to_plaintext = False

Filtering the outer content type

A simple filtering setting will just search the content types of the messages parts, discarding all parts with a matching MIME type. If the message’s outer content type matches the filter, the entire message will be discarded. However, if we turn off content filtering altogether, then the handler short-circuits.

>>> from mailman.interfaces.mime import FilterAction

>>> mlist.filter_types = ['image/jpeg']
>>> mlist.filter_action = FilterAction.discard
>>> from mailman.testing.helpers import (specialized_message_from_string
...   as message_from_string)
>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: image/jpeg
... MIME-Version: 1.0
...
... xxxxx
... """)

>>> from mailman.config import config
>>> process = config.handlers['mime-delete'].process
>>> mlist.filter_content = False
>>> msgdata = {}
>>> process(mlist, msg, msgdata)
>>> print(msg.as_string())
From: aperson@example.com
Content-Type: image/jpeg
MIME-Version: 1.0

xxxxx
>>> msgdata
{}

Similarly, no content filtering is performed on digest messages, which are crafted internally by Mailman.

>>> mlist.filter_content = True
>>> msgdata = {'isdigest': True}
>>> process(mlist, msg, msgdata)
>>> print(msg.as_string())
From: aperson@example.com
Content-Type: image/jpeg
MIME-Version: 1.0

xxxxx
>>> from mailman.testing.documentation import dump_msgdata
>>> dump_msgdata(msgdata)
isdigest: True

Simple multipart filtering

If one of the subparts in a multipart message matches the filter type, then just that subpart will be stripped. If that leaves just a single subpart, the multipart will be replaced by the subpart.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=BOUNDARY
... MIME-Version: 1.0
...
... --BOUNDARY
... Content-Type: image/jpeg
... MIME-Version: 1.0
...
... xxx
...
... --BOUNDARY
... Content-Type: image/gif
... MIME-Version: 1.0
...
... yyy
... --BOUNDARY--
... """)

>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: image/gif
X-Content-Filtered-By: Mailman/MimeDel ...

yyy

Collapsing multipart/alternative messages

When content filtering encounters a multipart/alternative part, and the results of filtering leave only one of the subparts, then the multipart/alternative may be collapsed. For example, in the following message, the outer content type is a multipart/mixed. Inside this part is just a single subpart that has a content type of multipart/alternative. This inner multipart has two subparts, a jpeg and a gif.

Content filtering will remove the jpeg part, leaving the multipart/alternative with only a single gif subpart. Because there’s only one subpart left, the MIME structure of the message will be reorganized, removing the inner multipart/alternative so that the outer multipart/mixed has just a single gif subpart, and then the multipart is recast as just the subpart.

>>> mlist.collapse_alternatives = True
>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=BOUNDARY
... MIME-Version: 1.0
...
... --BOUNDARY
... Content-Type: multipart/alternative; boundary=BOUND2
... MIME-Version: 1.0
...
... --BOUND2
... Content-Type: image/jpeg
... MIME-Version: 1.0
...
... xxx
...
... --BOUND2
... Content-Type: image/gif
... MIME-Version: 1.0
...
... yyy
... --BOUND2--
...
... --BOUNDARY--
... """)
>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: image/gif
X-Content-Filtered-By: Mailman/MimeDel ...

yyy

When the outer part is a multipart/alternative and filtering leaves this outer part with just one subpart, the entire message is converted to the left over part’s content type. In other words, the left over inner part is promoted to being the outer part.

>>> mlist.filter_types = ['image/jpeg', 'text/html']
>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/alternative; boundary=AAA
...
... --AAA
... Content-Type: text/html
...
... <b>This is some html</b>
... --AAA
... Content-Type: text/plain
...
... This is plain text
... --AAA--
... """)

>>> process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain
X-Content-Filtered-By: Mailman/MimeDel ...

This is plain text

Clean up.

>>> mlist.filter_types = ['image/jpeg']

Conversion to plain text

Some mailing lists prohibit HTML email, and in fact, such email can be a phishing or spam vector. However, many mail readers will send HTML email by default because users think it looks pretty. One approach to handling this would be to filter out text/html parts and rely on multipart/alternative collapsing to leave just a plain text part. This works because many mail readers that send HTML email actually send a plain text part in the second subpart of such multipart/alternatives.

While this is a good suggestion for plain text-only mailing lists, often a mail reader will send only a text/html part with no plain text alternative. in this case, the site administer can enable text/html to text/plain conversion by defining a conversion command. A list administrator still needs to enable such conversion for their list though.

>>> mlist.convert_html_to_plaintext = True

By default, Mailman sends the message through lynx, but since this program is not guaranteed to exist, we’ll craft a simple, but stupid script to simulate the conversion process. The script expects a single argument, which is the name of the file containing the message payload to filter.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: text/html
... MIME-Version: 1.0
...
... <html><head></head>
... <body></body></html>
... """)

>>> from mailman.handlers.tests.test_mimedel import dummy_script
>>> with dummy_script():
...     process(mlist, msg, {})
>>> print(msg.as_string())
From: aperson@example.com
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
X-Content-Filtered-By: Mailman/MimeDel ...

Converted text/html to text/plain
Filename: ...

Discarding empty parts

Similarly, if after filtering a multipart section ends up empty, then the entire multipart is discarded. For example, here’s a message where an inner multipart/mixed contains two jpeg subparts. Both jpegs are filtered out, so the entire inner multipart/mixed is discarded.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=AAA
...
... --AAA
... Content-Type: multipart/mixed; boundary=BBB
...
... --BBB
... Content-Type: image/jpeg
...
... xxx
... --BBB
... Content-Type: image/jpeg
...
... yyy
... --BBB---
... --AAA
... Content-Type: multipart/alternative; boundary=CCC
...
... --CCC
... Content-Type: text/html
...
... <h2>This is a header</h2>
...
... --CCC
... Content-Type: text/plain
...
... A different message
... --CCC--
... --AAA
... Content-Type: image/gif
...
... zzz
... --AAA
... Content-Type: image/gif
...
... aaa
... --AAA--
... """)

>>> with dummy_script():
...     process(mlist, msg, {})

>>> print(msg.as_string())
From: aperson@example.com
Content-Type: multipart/mixed; boundary=AAA
X-Content-Filtered-By: Mailman/MimeDel ...

--AAA
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"

Converted text/html to text/plain
Filename: ...

--AAA
Content-Type: image/gif

zzz
--AAA
Content-Type: image/gif

aaa
--AAA--

Passing MIME types

All of the above examples set only filter_types which will cause parts with matching MIME types to be removed. There is also a pass_types setting which if non-empty, causes parts with non-matching MIME types to be removed. I.e., it lists only those MIME types that will be kept. Also, note that if we want to keep elemental parts including those that might be sub-parts of some multipart or message/rfc822 part, we also need to include those types in pass_types. Here we use the example above to illustrate pass_types instead. Note that we want to keep both multipart/mixed and multipart/alternative to be able to keep some of their sub-parts so we just keep all multipart types. Likewise, we’ll keep all text parts even though in this case the original text/plain part will be removed by collapse_alternatives. Finally, we’ll keep only the image/gif image parts.

>>> mlist.filter_types = []
>>> mlist.pass_types = ['multipart', 'text', 'image/gif']
>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=AAA
...
... --AAA
... Content-Type: multipart/mixed; boundary=BBB
...
... --BBB
... Content-Type: image/jpeg
...
... xxx
... --BBB
... Content-Type: image/jpeg
...
... yyy
... --BBB---
... --AAA
... Content-Type: multipart/alternative; boundary=CCC
...
... --CCC
... Content-Type: text/html
...
... <h2>This is a header</h2>
...
... --CCC
... Content-Type: text/plain
...
... A different message
... --CCC--
... --AAA
... Content-Type: image/gif
...
... zzz
... --AAA
... Content-Type: image/gif
...
... aaa
... --AAA--
... """)

>>> with dummy_script():
...     process(mlist, msg, {})

>>> print(msg.as_string())
From: aperson@example.com
Content-Type: multipart/mixed; boundary=AAA
X-Content-Filtered-By: Mailman/MimeDel ...

--AAA
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"

Converted text/html to text/plain
Filename: ...

--AAA
Content-Type: image/gif

zzz
--AAA
Content-Type: image/gif

aaa
--AAA--

Interaction between passing and filtering

If both pass_types and filter_types are non-empty, any parts with types in filter_types are removed and any remaining parts with types not in pass_types are removed as well. Here’s an example.

>>> mlist.filter_types = ['text/rtf']
>>> mlist.pass_types = ['multipart/mixed', 'text']
>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=AAA
...
... --AAA
... Content-Type: text/plain
...
... plain text
... --AAA
... Content-Type: text/html
...
...
... <h2>A header</h2>
...
... --AAA
... Content-Type: text/rtf
...
... will be removed
... --AAA
... Content-Type: text/plain
...
... another plain text
... --AAA
... Content-Type: image/gif
...
... image will be removed
... --AAA
... Content-Type: multipart/alternative; boundary="BBB"
...
... --BBB
... Content-Type: text/plain
...
... plain text
... --BBB
... Content-Type: text/html
...
...  <h2>Another header</h2>
...
... --BBB--
... --AAA--
...
... """)

>>> with dummy_script():
...     process(mlist, msg, {})

>>> print(msg.as_string())
From: aperson@example.com
Content-Type: multipart/mixed; boundary=AAA
X-Content-Filtered-By: Mailman/MimeDel ...

--AAA
Content-Type: text/plain

plain text
--AAA
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"

Converted text/html to text/plain
Filename: ...



--AAA
Content-Type: text/plain

another plain text
--AAA--

Note that the html is still converted to plain text because we didn’t change that, the text/rtf part is removed due to filter_types and the image and the entire multipart/alternative parts are removed as they aren’t in pass_types

Passing and filtering extensions

pass_extensions and filter_extensions work in exactly the same way. These filters are applied following filter_types and pass_types and filter based on the filename extension of any part that has an associated filename with extension. Parts that don’t have an associated filename with an extension are not affected. Here’s a simple example.

>>> mlist.filter_types = []
>>> mlist.pass_types = []
>>> mlist.filter_extensions = ['pdf']
>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=AAA
...
... --AAA
... Content-Type: text/plain
...
... plain text
... --AAA
... Content-Type: application/pdf; name="file.pdf"
... Content-Disposition: attachment; filename="file.pdf"
...
... pdf to remove
... --AAA--
... """)

>>> with dummy_script():
...     process(mlist, msg, {})

>>> print(msg.as_string())
From: aperson@example.com
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain
X-Content-Filtered-By: Mailman/MimeDel ...

plain text

Reporting Option

Finally, there is a global mailman.cfg setting to append a report of what was filtered to the message. Here’s a simple example.

>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=AAA
...
... --AAA
... Content-Type: text/plain
...
... plain text
... --AAA
... Content-Type: application/pdf; name="file.pdf"
... Content-Disposition: attachment; filename="file.pdf"
...
... pdf to remove
... --AAA--
... """)

>>> with dummy_script('report'):
...     process(mlist, msg, {})

>>> print(msg.as_string())
From: aperson@example.com
MIME-Version: 1.0
X-Content-Filtered-By: Mailman/MimeDel ...
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

plain text

___________________________________________
Mailman's content filtering has removed the
following MIME parts from this message.

Content-Type: application/pdf
    Name: file.pdf

And here’s a more complex one. Note that if the filtered message is multipart, the report is attached as a separate MIME part.

>>> mlist.filter_types = ['text/rtf']
>>> mlist.pass_types = ['multipart/mixed', 'text']
>>> msg = message_from_string("""\
... From: aperson@example.com
... Content-Type: multipart/mixed; boundary=AAA
...
... --AAA
... Content-Type: text/plain
...
... plain text
... --AAA
... Content-Type: text/html
...
...
... <h2>A header</h2>
...
... --AAA
... Content-Type: text/rtf
...
... will be removed
... --AAA
... Content-Type: text/plain
...
... another plain text
... --AAA
... Content-Type: image/gif
...
... image will be removed
... --AAA
... Content-Type: multipart/alternative; boundary="BBB"
...
... --BBB
... Content-Type: text/plain
...
... plain text
... --BBB
... Content-Type: text/html
...
...  <h2>Another header</h2>
...
... --BBB--
... --AAA--
...
... """)

>>> with dummy_script('report'):
...     process(mlist, msg, {})

>>> print(msg.as_string())
From: aperson@example.com
Content-Type: multipart/mixed; boundary=AAA
X-Content-Filtered-By: Mailman/MimeDel ...

--AAA
Content-Type: text/plain

plain text
--AAA
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"

Converted text/html to text/plain
Filename: ...



--AAA
Content-Type: text/plain

another plain text
--AAA
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit


___________________________________________
Mailman's content filtering has removed the
following MIME parts from this message.

Content-Type: text/rtf

Content-Type: image/gif

Content-Type: multipart/alternative

--AAA--