wiki:LancelotModules

Project Lancelot Modules

This is a brief overview of the various processing modules that can be used in Project Lancelot workflows. A workflow in Project Lancelot is a chain of steps to be taken when a message arrives to a specific address, such as LIST+subscribe@DOMAIN. In general, mail to LIST+foo@DOMAIN is handled by a workflow defined by the mail.workflow.foo configuration parameter. You can change workflows by suitably redefining these configuration parameters so as to include or omit different processing modules, and you can configure the processing modules themselves in a list's configuration database. This makes Project Lancelot highly configurable for all sorts of mailing list applications.

Here are some exemplary workflows as used for real-world mailing lists:

mail.workflow.admin = authenticate(level=1) admin
mail.workflow.help = send_help
mail.workflow.owner = forward
mail.workflow.submit = policy_addresses policy_mime list_headers archive_store post_message
mail.workflow.subscribe = policy_subscribe ask_confirm(cmd=subscribe) subscribe_user
mail.workflow.unsubscribe = ask_confirm(cmd=unsubscribe) unsubscribe_user
mail.workflow.config = prep_config ask_confirm(cmd=config) do_config

Writing processing modules

Project Lancelot processing modules live in the Lancelot::Module namespace and are simple Perl modules. A very simple processing module is the following:

package Lancelot::Module::simple;

sub execute {
    my ($db, $msg, $argsref) = @_;

    $msg->set_flag('simple', $msg->get_flag('simple') + 1);
    return '';
}

1;

(which should go into a file called Lancelot/Module?/simple.pm). This illustrates several interesting points:

  • Every Project Lancelot processing module must define a subroutine called execute. Other than that it may contain arbitrary subroutines or references to other Perl modules, objects, etc.
  • The execute subroutine takes two mandatory parameters, namely a reference to the Project Lancelot configuration database object (of class Lancelot::DB) and a reference to the message under consideration (of class Lancelot::Message). The third, optional, argument is a reference to a hash containing any parameters mentioned for the processing module in the workflow. E.g., for simple(foo=bar,baz=quux), $argsref would point to a hash containing the foo and baz keys with the values of 'bar' and 'quux', respectively. (Note: For now, the workflow specification parser is exceedingly simple-minded. It does not support spaces or any kind of escaping of commas, so I would recommend you stick to alphanumeric keys and values only.)
  • A processing module returns either an empty string, in which case processing will continue with the next processing module in the list, or the name of a message template to be returned to the sender of the message being considered, with various substitutions. There is a number of predefined substitutions, but the processing module may also return a reference to a hash giving extra substitutions in addition to the template name. The template name may also be DROP, which will discard the message and stop processing the workflow. (This will probably be extended in the future.)
  • Processing modules may add flags to a message for the benefit of other processing modules later in the workflow. This is supported by the get_flag and set_flag methods of Lancelot::Message.
  • Being Perl modules, processing modules must finish with a 1;.

Module admin

Description

This module allows remote administration of mailing lists by designated »administrators«, using specially formatted e-mail messages. The possible operations include

  • subscribing and unsubscribing people
  • changing people's subscription options
  • retrieving lists of subscribers
  • retrieving and setting list configuration parameters

It is up to the list administrator (who has shell access to the list) to enable or disable these operations. Remote administration is explained in more detail in the RemoteAdminHowto.

The admin module does not deal with the authentication of administrative requests. This should be handled separately, e.g., through the authenticate framework. The only policy enforced by the admin module is that administrative requests must have been sent from addresses that are marked as »administrators« in the list's subscriber database.

Configuration Parameters

  • admin.allowed (whitespace-separated list of strings): Lists those commands that can be used in administrative requests. (Default: all commands)
  • admin.setconfig.allow (regular expression): If set, only those list configuration parameters that match this regular expression can be changed remotely. admin.setconfig.refuse takes precedence over this parameter. (Default: undefined)
  • admin.setconfig.refuse (regular expression): If set, list configuration parameters that match this regular expression cannot be changed remotely. This parameter takes precedence over admin.setconfig.allow. (Default: undefined)

Module archive_lurker

Description

This module passes the current message to the Lurker mail archive package, which you must obtain and install separately. (It's a cool piece of software so check it out.)

Configuration Parameters

  • archive.lurker.command (Unix command): The command used to invoke the Lurker archiver. This will be passed the message to be archived on its standard input channel. A %s in the value of this configuration parameter will be replaced by the Lurker list name (see below) before the command is invoked. (Default: lurker-index -c /etc/lurker/lurker.conf -l %s -m)
  • archive.lurker.name (string): The name of the list from Lurker's point of view. (Default: value of list.name)

Module archive_store

Description

This module stores the current message in a »Project Lancelot-style archive«. Each message is numbered, and the archive filename is derived from this number by dividing it by 100 to determine a directory name, and taking the remainder modulo 100 to determine the name of a file in that directory. By default, messages are archived in a subdirectory archive of the list directory, so message 12345 to list list@… will be archived as

$HOME/.ll/list@example.com/archive/123/45

It is up to you to come up with a method of actually making the archive accessible.

Before archiving, this module copies the message and removes the following headers from the copy:

Delivered-To List-Help List-ID List-Owner List-Post List-Subscribe
List-Unsubscribe Received Return-Path X-Virus-Scanned

The resulting message is then written to the archive.

Configuration Parameters

  • archive.directory (directory name): The name of the directory that holds the archive. If this is an absolute path name (starting with a slash) it is used as is; if it is a relative path name it is taken relative to the list directory. The directory (and any non-existing parent directories) will be created if it does not exist. (Default: archive)
  • archive.dirmode (Unix file mode): The mode (set of permission bits) to be used if the archive directory is created by this module. Should be given as an octal number with a leading zero, as in
    archive.dirmode = 0750
    
    If you forget the leading zero, the mode will be interpreted as a decimal number, which will lead to unexpected and interesting results. -- Make sure that the mode you specify actually works, i.e., results in a directory that is write-accessible to the user Project Lancelot is running as. (Default: 0755)

Module ask_confirm

Description

This module can be used to implement workflows that need to be confirmed by the user, such as »double-opt-in« subscription schemes. With a workflow definition like

mail.workflow.subscribe = ask_confirm(cmd=subscribe) subscribe_user

when a message to list+subscribe@… is received, the module logs a subscribe action on behalf of the sender of the message as pending, sends back a message with a reply-to address of the form

list+subscribe-confirm-LONGRANDOMSTRING@example.com

and aborts the workflow at this point.

If, later on, a message is received from the address to be subscribed that contains the correct long random string, processing of that message continues with the next workflow module (in this case, subscribe_user).

This module is potentially useful for subscription, unsubscription and changes of subscription options. At some point it will probably be replaced by the general authentication framework as in authenticate (which hadn't been implemented yet at the time that ask_confirm was needed).

Configuration Parameters

  • list.confirmdelay (integer): The number of seconds that a confirmation request remains pending. (Default: 3 days)

Module authenticate

Description

This module can be used to allow certain list operations only after suitable authentication. It supports an extensible set of authentication methods including clear-text passwords (Mailman style) and GPG signatures. This is explained in more detail in the LancelotAuthenticationHowto?.

The module is used by putting it in a workflow before the operation you want to protect. For example,

mail.workflow.submit = authenticate(level=1) list_headers post_message

could be used for an announcements-only mailing list to refuse postings from everyone except the people in charge. Authentication is more usually employed to enable remote list administration and moderation by e-mail.

Configuration Parameters

This module does not use configuration parameters.

Module do_config

See the explanation of prep_config below.

Module forward

Description

This module forwards an incoming message to a different address as a MIME attachment. The address can be given as an argument to the module, as in

forward(addr=alfred@example.com)

or, if no address is given, it defaults to the list owner's address as specified in list.owneraddress. This can be used, for example, in a workflow like

mail.workflow.owner = forward

so mail sent to list+owner@… will be forwarded to the actual list owner as per the list's configuration.

The forwarding message's From: address is the list's owner address, like list+owner@….

The forwarding mechanism uses the same SMTP setup as the bulk sending process documented for the post_message module.

Configuration Parameters

  • forward.notice (string): Message used for the MIME body of the forwarding message. A %s is replaced by (in descending order of preference) the value of the Delivered-To header if it exists, the value of the To header if it exists, and the value of the list.name configuration parameter. (Default: This just came in to %s.)

Module list_headers

Description

This module performs various header manipulations:

  • Insertion of RFC2919 and RFC2369 headers
  • Subject-line tagging
  • Reply-To: munging
  • Addition and removal of arbitrary other headers

Configuration Parameters

  • mail.addheaders (whitespace-separated list of strings): Gives the names of headers to be added to the message. The new headers themselves are contained in mail.header parameters. For example, the
    mail.addheaders = foo bar
    mail.header.foo = baz
    mail.header.bar = quux
    
    configuration adds the headers Foo: baz and Bar: quux.
  • mail.forcereplyto (boolean): If true, and if mail.replyto is set, adds a Reply-To: header to the message using the content of mail.replyto. Any previous Reply-To: header is overwritten. (Note: The issue of whether using this is actually a good idea is highly contentious. See Chip Rosenthal's article on the subject.) (Default: false)
  • mail.listheaders (boolean): If true, RFC2369 headers (List-Post, List-Subscribe, and friends) will be generated for the list. (Default: true)
  • mail.listid (string): If set, this will be used as the complete RFC2919 List-ID: header, i.e., it should include the angle brackets and a comment (if desired). If not set, the List-ID: header will be derived from general.listaddress by replacing the @ in that address by .list-id.. (Note: There is no way of not generating a List-ID header.) (Default: unset)
  • mail.removeheaders (whitespace-separated list of strings): The headers listed in this parameter will be removed from the message. Don't include colons.
  • mail.replyto (string): If set, this string will be used as the value of a Reply-To: header if the message does not already contain one. This can be used to direct replies to the list if the author of the original message didn't supply their own reply-to address (but see the caveat about reply-to munging above). (Default: unset)
  • mail.subjecttag (string): If set, this string will be inserted at the beginning of a message's Subject: header if it is not already present. Note that this does not include trailing spaces, so mail.subjecttag = [List] will result in something like Subject: [List]My Subject. Use double quotes to add trailing spaces, like mail.subjecttag = "[List] ". (A possibly empty sequence of Re: tags and spaces in front of the subject tag will be ignored, to keep the header from growing without bounds.) (Default: unset)

Module policy_addresses

Description

This module can check whether the list's submission address shows up in the current message's To: or Cc: headers, and reject the message otherwise.

It can also check whether the sender of the current message is a subscriber to the list, and set the from_subscriber flag on the message if this is the case.

Configuration Parameters

  • mail.policy.explicitaddress (boolean): If true, the list's submission address (see general.listaddress) must occur in the To: or Cc: headers for the message to be considered for further processing, otherwise the message is bounced using the policy-addresses-explicit template. (Default: true)
  • mail.policy.subscribersonly (boolean): Determines whether message senders should be tested for subscribership. If this is set and the message sender is not a subscriber, set the nonsubscriber flag on the message. (Default: true)

Module policy_mime

Description

This module checks the MIME type of a message and possibly its subparts and does the following:

  • Messages of content type text/plain are accepted for further processing.
  • Two-part messages of content type multipart/signed are also accepted for further processing.
  • Messages of content type multipart/alternative are accepted for further processing in the following cases:
    • They contain exactly one subpart, which is of type text/plain. (This part will be coerced to be the main body before continuing.)
    • They contain exactly two subparts, one of which is of type text/plain and the other is of type text/html. (The text/html part will be removed, and the text/plain part will be coerced to be the main message body.) All other multipart/alternative messages are either dropped (if the message contains two subparts or if the sender is not a list subscriber) or bounced back to the sender using the policy-mime template.
  • Messages of content type multipart/mixed are accepted for further processing if they contain exactly one subpart and that subpart is of type text/plain. (This subpart will be made the main body as above.)
  • All other messages are dropped (if the sender is not a list subscriber) or bounced back to the sender using the policy-mime template.

Configuration Parameters

This module does not support configuration parameters -- you either include it in the workflow or you don't. It is intentionally simplistic, but we will entertain suggestions for a beefed-up MIME policy module that could, for example, pass attachments if they have specific (configurable) content types and/or pass a virus check etc.

Module policy_subscribe

Description

This module enforces policy for new subscribers. For the time being, it disallows subscription requests for closed lists, but it could in theory be enhanced to, e.g., allow only addresses in certain domains to subscribe.

Use this as the first module in mail.workflow.subscribe.

Configuration Parameters

  • subscribe.type (open or closed): If set to closed, subscription requests will be refused with an error message. (Default: open)

Module post_digest

Description

This module handles digests (collections of messages posted during a certain time period). It works from a Project Lancelot-style message archive, so it should be added to mail.workflow.submit somewhere after archive_store.

Project Lancelot allows for various types of digest, which are constructed using digest-generating modules (»digest providers«) in Lancelot::Digest. This is documented in more detail in the LancelotDigestHowto?.

This module is responsible for checking, when a new message is processed, whether there is enough material for a digest, and sending that digest out. For low-traffic lists, it may be the case that »almost enough« material is hanging around for ages waiting for the final message to arrive, so there is also a provision for sending a digest with whatever is there after a certain amount of time has passed. This is orchestrated by the ll-janitor(1) cron job rather than the mail.workflow.submit workflow.

Configuration Parameters

  • digest.headers (whitespace-separated list of strings): The headers from the original message to be included in the digest. This parameter is advisory only, as it can be overridden by the digest provider.
  • digest.maxsize (integer): If the amount of currently accumulated older material plus the size of the current message exceeds this limit, post_digest sends out the older material as a digest and keeps the current message as the first message of the next digest. (Default: 33000)
  • digest.minsize (integer): If the amount of currently accumulated older material plus the size of the current message exceeds this limit but does not exceed digest.maxsize, all the accumulated material and the current message are sent out as a digest, and the next digest starts out empty. (Default: 30000 bytes)
  • digest.name (string): The »name« of a digest, as in »The Foobar Digest«. This parameter is advisory only, as it can be overridden by the digest provider.
  • digest.type (string): The type of digest to be produced for this list. This must be the name of a Perl module under Lancelot::Digest. For now, the only admissible value is plain, which produces plain-text digests according to RFC 1153. (Default: plain)
  • digest.volume (yearly or quarterly or monthly): Determines the period of time after which the digest »volume« number is incremented. Digests are counted sequentially within »volumes«, so a digest could be volume 10, number 3. This is largely cosmetic. (Default: yearly)

Module post_message

Description

This module posts the current message to list subscribers using SMTP to a designated server. It handles only those subscribers who do not elect to receive the list as a digest.

It performs the following operations before actually posting the message:

  • The message is given a unique message number if it does not already have one (e.g., because it was archived).
  • A Precedence: bulk header is added.
  • An X-Software: header is added to show off the Lancelot version number.
  • An X-Loop: header is added to avoid creating mail loops.
  • Any Return-Path: and Delivered-To: headers are removed from the message.

Configuration Parameters

The following parameters really apply to the Project Lancelot bulk mail sender, which can be found in Lancelot::SMTP::bulk_send().

  • smtp.debug (boolean): If true, the SMTP protocol dialogue will be echoed on standard error. This only makes sense when debugging. (Default: false)
  • smtp.maxrecipients (integer): The maximum number of recipient addresses passed to the SMTP server during a single SMTP message submission. The SMTP standard mandates that a server accept at least 100 recipients at one go, so if you know your server does better then feel free to up this. (Default: 100)
  • smtp.port (string): The port number used by the SMTP server (Default: 25)
  • smtp.server (string): The name or IP address of the SMTP server. (Default: localhost)
  • smtp.timeout (string): The SMTP timeout (in seconds). (Default: 120)
  • smtp.verp (boolean): Whether Project Lancelot should use VERP (variable envelope return paths) when sending out messages to subscribers. This makes it possible to figure out, from a bounce message, the address that the original list message was sent to, which is a desirable feature; however, without support from your MTA, this means that bulk-sending list messages to many recipients at a time does not work. If smtp.verp is set to true, Project Lancelot checks whether your MTA supports the ESMTP VERP extension and if so, uses that, which is efficient as the MTA puts in the variable bits. If your MTA does not support the ESMTP VERP extension, Project Lancelot does the VERPing itself, at a possibly big performance hit. (Default: false)

Module prep_config

Description

This module, together with do_config, allows subscribers to change their subscription options (such as digest or nomail) remotely via e-mail. This is usually implemented as

mail.workflow.change = prep_config ask_confirm(command=change) do_config

The prep_config module parses the change request(s) and marks them as pending using ask_confirm; when the request is confirmed do_config actually executes them.

This module expects change requests either in the message subject header:

Subject: set digest yes

or in the message body:

set digest yes
set nomail no

In the subject, only one option can be changed at a time.

Configuration Parameters

This module does not use any configuration parameters.

Module send_help

Description

This module sends an explanatory text to the originator of the message. It is used in workflows like

mail.workflow.help = send_help

It takes a parameter naming a template that the explanatory message is based on (addresses etc. will be adapted to the list in question). The standard text is called help-message, and there is another pre-cooked message, admin-help-message, which explains remote list administration. This can be used in

mail.workflow.helpadmin = send_help(message=admin-help-message)

Configuration Parameters

This module does not use configuration parameters (but the reply message templates may do so).

Module subscribe_user

Description

This module adds the sender of the current message (according to the From: header) to the subscriber database.

Configuration Parameters

  • subscribe.digestdefault (boolean): If true, a new subscriber is added as a digest subscriber, if false, as a receiver of individual messages. (Default: false)
  • subscribe.nomaildefault (boolean): If true, a new subscriber is added as one who will not receive messages but whose submissions will be accepted as legitimate. If false, they will be added as normal, »receiving« subscribers. (Default: false)
  • subscribe.moderateddefault (boolean): If true, a new subscriber is added as a »moderated« subscriber whose submissions must be vetted by a list moderator before they are posted to the list. If false, this does not apply. This mechanism is independent of »whole-list« moderation, so special hard cases may be put on probation while the rest of the list is unmoderated. Depending on your version of Project Lancelot, this may not actually be implemented. (Default: false)
  • subscribe.notices (boolean): If true, the list owner is notified by e-mail if a new subscriber is added to the list. (Default: false)
  • subscribe.welcome (boolean): If true, a new subscriber is sent a welcoming message (template welcome-message) after their subscription has been successfully processed. (Default: false)

Module unsubscribe_user

Description

This module removes the sender of the current message (according to the From: header) from the subscriber database.

Configuration Parameters

  • unsubscribe.goodbye (boolean): If true, the sender of the message is sent a farewell message (template goodbye-message) after their unsubscription has been successfully processed. (Default: false)
  • unsubscribe.notices (boolean): If true, the list owner is notified by e-mail if a person has unsubscribed from the list. (Default: false)
Last modified 6 years ago Last modified on Aug 4, 2011, 7:16:55 PM