Hypermail 2.0


Contents:


What is Hypermail?

Hypermail is a program that takes a file of mail messages in UNIX mailbox format and generates a set of cross-referenced HTML documents. Each file that is created represents a separate message in the mail archive and contains links to other articles, so that the entire archive can be browsed in a number of ways by following links. Archives generated by Hypermail can be incrementally updated, and Hypermail is set by default to only update archives when changes are detected.

Each HTML file that is generated for a message contains (where applicable):

In addition, Hypermail will convert references in each message to email addresses and URLs to hyperlinks so they can be selected. Email addresses can be converted to mailto: URLs or links to a CGI mail program.

To complement each set of HTML messages, four index files are created which sort the articles by date received, thread, subject, and author. Each entry in these index files are links to the individual articles and provide a bird's-eye view of every archived message.

Hypermail was originally developed and designed by Tom Gruber for Enterprise Integration Technologies (EIT) in Common Lisp. It was later rewritten in C by Kevin Hughes while at EIT. Hypermail is now being maintained by Kent Landfield <kent@landfield.com>.

To see what Hypermail can do, take a look at these Hypermail-produced archives:


Usage

Usage: hypermail [options]

Options:
  -a URL        : URL to other archives
  -b URL        : URL to archive information
  -c file       : Configuration file to read in
  -d dir        : The directory to save HTML files in
  -i            : Read messages from standard input
  -l label      : What to name the output archive
  -m mbox       : Mail archive to read in
  -M            : Use metadata
  -n listaddr   : The submission address of the list
  -o keyword=val: Set config item
  -p            : Show progress
  -s htmlsuffix : HTML file suffix (.html, .htm, ..)
  -t            : Use Tables
  -T            : Use index tables
  -u            : Update archive by one article
  -v            : Show configuration variables only
  -V            : Show version information and exit
  -x            : Overwrite previous messages
  -1            : Read only one mail from input
  -L lang       : Specify language to use (de en es se fi )

Using the flags -h, or -? with Hypermail will display this usage summary.


Command-Line Options

Input and Output Options:
-i,
-m  "mailbox",
-d  "directory",
-c  "file"

To tell Hypermail what mailbox to read in, use the -m option. If articles will be sent to Hypermail through standard input, use the -i option. Note that the -m and -i options can't be used together! By default, Hypermail will look for a file called mbox to read its articles in from.

The -d option specifies the directory to put the HTML files and index files that are created into. If the directory doesn't exist, a new one will be created with the name that is specified. If the -d option isn't used, Hypermail will look for a directory with the same name as the mailbox or will create one if needed.

   example 1: hypermail -m "wu-ftpd" -d "/wu-ftpd"
   example 2: cat "/var/spool/mail/wu-ftpd" | hypermail -i

  1. This example reads the articles in wu-ftpd and will save the output in the /wu-ftpd directory.

  2. This reads the file /var/spool/mail/wu-ftpd from standard input and will save the output in a directory called wu-ftpd in the same directory Hypermail was run from.

Note that Hypermail can only read messages in the UNIX mailbox format! Such archives are typically RFC 822 mail messages appended to each other that look similar to this:

   From john@foo.com  Mon Jan  1 00:01:30 1994
   Date: Mon, 1 Jan 1994 00:01:15 PDT
   From: john@foo.com
   To: everyone@foo.com
   Subject: Hello, world!

   Hi, everyone, just saying hello!

   From someone.else@foo.com  Mon Jan  1 00:02:00 1994
   Date: Mon, 1 Jan 1994 00:01:45 PDT
   ...

The messages are typically separated by lines in this format:

   From wu-ftpd@wugate.wustl.edu  Fri Jul  1 00:18:20 1994

The -c option tells Hypermail to read in settings from a configuration file. By default, the program will attempt to read settings from a file called .hmrc in the user's home directory if it exists.

In the configuration file, variables are set in the following manner:

variable = number
variable = "string"
The complete set of variables that Hypermail recognizes is described in the Configuration Options section.

Archive Interface Options:

-l  "label",
-b  "About URL",
-a  "Other Archives URL"

The -l option tells Hypermail what to call the archive - the name that is specified will be in the title of the index pages so users know what sort of messages are being archived.

The -a option includes a link labelled "Other mail archives" in the index pages to any specified URL. This way users who are looking at the archive have the opportunity to go to pointers to other mail archives. By default, this will be a pointer to the parent directory in which the archive files reside.

The -b option includes a link labelled "About this archive" in the index pages to any specified URL. This way users who are looking at the archive have the opportunity to go to information about the archive.

   example: hypermail -l "WU-FTPD Development Archives"
            -a "http://www.landfield.com/wu-ftpd/"
            -b "http://www.landfield.com/wu-ftpd/mail-archive/"

In the index files for the archive, the above setting will produce something like this:

(top of page)

WU-FTPD Archives

(list of indexed articles below)

Updating Options:

-x,
-u

The -x option tells Hypermail to explicitly overwrite any previous HTML files that may exist. Use this option only when it is desirable to completely rewrite the entire archive.

The -u option tells Hypermail to update the archives by one message only. With this option, only one email message will be read in from a file or standard input. This message will be added to the end of the existing HTML file archive and will be integrated into it by links and cross-references. All archive index files will be regenerated to include the new message.

Important! - If the -u option is used and Hypermail is reading from a mailbox file, Hypermail will assume that the file contains only one article, no matter how many files the mailbox may actually contain. Make sure that Hypermail is given only one article when using this option.

If the mailbox that is being read from is an archive that new messages are always being added to, don't use either of these options. Hypermail will then read in all the messages given it but will only write new messages that have been appended to the mailbox.

   example 1: cat "one.letter" | hypermail -i -u -d "/wu-ftpd/mail-archives"
   example 2: hypermail -i -u -m "one.letter" -d "/wu-ftpd/mail-archives"
   example 3: hypermail -m "mailbox" -d "/wu-ftpd/mail-archives" -x
   example 4: hypermail -m "mailbox" -d "/wu-ftpd/mail-archives"

  1. This tells Hypermail to take the article it receives from standard input and integrate it with the archive under the wu-ftpd/mail-archives directory. If no archive exists, a new one will be created with the specified letter as the first file of the archive.

  2. This does the same thing, except that the letter is read in from a file that contains only that letter.

  3. With these options, Hypermail will read in the articles from mailbox and write over any existing files in the wu-ftpd/mail-archives directory if they exist. If no archive exists, a new one will be created.

  4. With these options, Hypermail will read in the articles from mailbox and only write new articles - it will not overwrite any existing archive files.

Note that no matter what options are specified, the index files are always rewritten. The date when Hypermail was last run is included in index pages, so it's easy to tell when the archive was last updated.

Miscellaneous Options

-p
-v
-V

The -p option shows a progress report as Hypermail reads in and writes out messages - the number of files that Hypermail is reading and writing and the file names of the directory and files created are shown. This information is written to standard output.

The -v option shows the configuration variables and their values that Hypermail would use if it was run with the same configuration file and command line options. This is useful when starting up a new list or modifying a list configuration file. Once the information is displayed, Hypermail terminates and not actual processing occurs.

The -V option prints the Hypermail version information. Once the information is displayed, Hypermail terminates and not actual processing occurs.


Configuration Options

Hypermail has many variables that can be set as environment variables or as variables in the specified configuration file. For instance, using the C shell, one could define variables in this manner:

   setenv HM_MBOX /home/john/my_mailbox
   setenv HM_FILEMODE 0600

In the configuration file, variables must be in lowercase and separated by their values with an equals (=) sign. Blank lines and lines beginning with the # character are skipped:

   mbox = "/home/john/my_mailbox"
   filemode = 0600
While the example uses quotes ("), they is not required when used in the configuration file.

Below is a list of configuration variables that can be defined. Boolean numbers can have the value of 0 or 1.

HM_CONFIGFILE "filename"
This is the default configuration file to read settings in from. This can only be specified as an environment variable. If the first character is "~", Hypermail will look for the file under the current user's home directory.

HM_MBOX "filename"
This is the default mailbox to read messages in from. Define this with a value of "NONE" to read from standard input as the default.

HM_LABEL "label name"
Define this as the default label to put in archives.

HM_HTMLSUFFIX "suffix"
Use this to specify the html file suffix (html, htm, shtml, ..) to be used when Hypermail generates the html files. This is dependent on local needs. Do not put a '.' in the value. It would result in "file..html", probably not what you want.

HM_LANGUAGE "language-id"
This is a two-letter string specifying the default language to use. Set this the value of the language table you wish to use when running and generating archives.

Current supported languages:
de - German
en - English
es - Spanish
fi - Finnish
fr - French
is - Icelandic
se - Swedish

HM_USELOCK [ 0 | 1 ]
Controls whether to use hypermail's built-in locking mechanism. By default, this option is set to 1. Set it to 0 if you have an external locking mechanism, like, for example, when using procmail or smartlist.

HM_LOCKTIME "number-of-seconds"
The number of seconds that a lock should be honored when processing inbound messages before it is overridden.

HM_USEMETA [ 0 | 1 ]
This option allows you to use metadata to store the content type of a MIME attachments and, later on, when a user browses the attachment, send back this information in the HTTP Content-Type header. When set to 1, the Content-Type header of a MIME attachment will be stored in a metadata file. Let us say that the MIME attachments for a message are stored in directory att-num. The metadata for those attachments will then be stored in directory att-num/.meta. If a MIME attachment is stored in file att-file, its metadata will be stored in file att-file.meta. This convention is directly compatible with the Apache server handling of metadata.

HM_ARCHIVES "URL"
This will create a link in the archived index pages to the specified URL. Define as "NONE" to omit such a link.

HM_ABOUT "URL"
This will create a link in the archived index pages to the specified URL. Define as "NONE" to omit such a link.

HM_DIR "directory"
This is the default directory that Hypermail will look for when creating and updating archives. If defined as "NONE", the directory name will be the same name as the mailbox read in.

HM_DEFAULTINDEX "type"
This is the index that is intended to be the one that users see first when entering the archive. Possible values are "date", "thread", "subject", and "author".

HM_REVERSE boolean_number
Defining this variable as 1 will reverse-sort the article entries in the date and thread index files by the date they were received. That is, the most recent messages will appear at the top of the index rather than the other way around.

HM_USETABLE boolean_number
Defining this will cause Hypermail to generate an index menu in HTML table format at the top and bottom of each page.

HM_INDEXTABLE boolean_number
Setting this variable to 1 will tell Hypermail to generate a message index Subject/Author/Date listings using a table format. Set to 0 if you want the standard Hypermail index page look and feel.

HM_PROGRESS [ 0 | 1 | 2 ]
Define as 1 or 2 to always show a progress report as Hypermail works. Defined as 2 shows more information about the attachment files created.

HM_ATTACHMENTLINK "attachment-link-format"

Format of the attachment links.
%p - full path to the attachment
%f - file name part only
%d - directory name only
%n - message number
%c - content type string

HM_SHOW_MSG_LINKS boolean_number
Define as 1 to put the individual message links at the top of the individual message pages. Define as 0 to produce pages without the Next, Previous, Reply, In-reply-to, etc. links.

HM_SHOWHEADERS boolean_number
Define this as 1 to show the article header lines in the archived HTML files. These lines typically include the "To:", "From:", and "Subject:" information found in most email messages.

HM_SHOWREPLIES boolean_number
Define as 1 to show all replies to a message as links in article files.

HM_SHOWHTML boolean_number
Define as 1 to show the articles in a proportionally-spaced font rather than a fixed-width (monospace) font.

HM_SHOWBR boolean_number
Define as 1 to place <br> tags at the end of article lines. Otherwise, all non-quoted article lines will word wrap. This only takes effect if HM_SHOWHTML is defined.

HM_IQUOTES boolean_number
Define as 1 to italicize quoted lines.

HM_SHOWHR boolean_number
Define as 1 to place horizontal rules before and after articles.

HM_OVERWRITE boolean_number
Define as 1 to make Hypermail overwrite existing archives by default.

HM_READONE boolean_number
Set this to 1 to specify there is only one message in the input.

HM_INCREMENT boolean_number
Define as 1 to read one article only and append it to existing archives by default.

HM_DISCARD_DUP_MSGIDS boolean_number
Set this to 0 to accept messages with a Message-ID matching that of a message already in this archive. By default such messages are discarded.

HM_REQUIRE_MSGIDS boolean_number
Set this to 0 to accept messages without a Message-ID header. Set this to 1 to discard messages without a Message-ID header. By default such messages are discarded.

HM_DATEFORMAT strftime-date-format
Format used in strftime(3) call for displaying dates. See strftime(3) for the valid conversion specifications.

HM_THRDLEVELS number
This specifies the number of thread levels to outline in the thread index. For instance, if HM_THRDLEVELS is 2, replies to messages will be indented once in the index, but replies to replies, etc., will only be indented once as well.

HM_EURODATE boolean_number
Define as 1 to display article received dates with days before months instead of months before days.

HM_DIRMODE octal_number
This is an octal number representing the permissions that new directories are set to when they are created. If the archives will be made publically available, it's a good idea to define this as 0755. If files will be updated incrementally with sendmail, you should make sure the owner of the archives is a sendmail trusted user. See sendmail documentation for more information.

HM_FILEMODE octal_number
This is an octal number representing the file permissions that new files are set to when they are created. If the archives will be made publically available, it's a good idea to define this as 0644. If files will be updated incrementally with sendmail, you should make sure the owner of the archives is a sendmail trusted user. See sendmail documentation for more information.

HM_MAILCOMMAND "command"
This specifies the mail command to use when converting email addresses to links. Variables can be used in constructing the command string:

If defined as "NONE", email addresses will not be converted to links in articles. A possible command one could use is "mailto:$TO", but this could easily be changed to specify a CGI program such as "/cgi-bin/mail?to=$TO". A CGI mail program is included with the source which can be used for this purpose.

HM_MAILTO address
The address of the contact point that is put in the HTML header line
<LINK REV=made HREF=mailto:MAILTO>
The <LINK...> header can be disabled by default by setting HM_MAILTO to "NONE".

HM_DOMAINADDR "domainname"
Set this to the domainname you want added to a mail address appearing in the RFC822 field which lack a hostname. When the list resides on the same host as the user sending the message, it is often not required of the MTA to domain-ize these addresses for delivery. In such cases, Hypermail will add the DOMAINADDR to the email address. If defined as NONE, this feature is turned off.

HM_IHTMLHEADERFILE "path"
Define path as the path to a file containing valid HTML formatting statements that you wish to included at the top of every index page. Hypermail will print this file as the header of the index so make sure it contains <HTML>, <HEAD>, and <BODY> and other statements that suit your local customized needs.

HM_IHTMLFOOTERFILE "path"
Define path as the path to a file containing valid HTML formatting statements that you wish to included at the bottom of every index page. Hypermail will print this file as the trailer of the index so make sure it contains at a minimum a </BODY> and </HTML> statement.

HM_MHTMLHEADERFILE "path"
Define path as the path to a file containing valid HTML formatting statements that you wish to included at the top of every message page. Hypermail will print this file as the header of the message so make sure it contains <HTML>, <HEAD>, and <BODY> and other statements that suit your local customized needs.

HM_MHTMLFOOTERFILE "path"
Define path as the path to a file containing valid HTML formatting statements that you wish to included at the bottom of every message page. Hypermail will print this file as the trailer of the message so make sure it contains at a minimum a </BODY> and </HTML> statement.

HM_HMAIL "list submission address"
This is the email address used to send a new message to a hypermail archive. "NONE" means don't use it. Since this is different for each hypermail archive, you should probably leave it set to "NONE" here, and let it be specified at runtime by command-line parameters in the list specific configfile.

HM_BODY "HTML <BODY> statement"
This is the <BODY> line to use when generating the HTML pages. Define as "NONE" to use the builtin <BODY> line by default.

HM_SHOW_HEADERS "list of RFC 822 Headers to display"
Define the list of headers to be displayed if the variable HM_SHOWHEADERS is set to 1 (ON). If it contains the special character ``*'' hypermail will display all header lines. This is a comma or space separated all on a single line such as
show_headers = From,Subject,Date,Message-ID
or they can be listed individually or any combination of.
show_headers = From
show_headers = Subject
show_headers = Date
show_headers = Message-ID
HM_INLINE_TYPES "indicate which data types should be inlined"
This is the list of MIME types that you want inlined as opposed to simply linked into the message. They can be listed individually on multiple lines or comma or space separated on a single line.
inline_types = image/gif image/jpeg
or
inline_types = image/gif
inline_types = image/jpeg
HM_IGNORE_TYPES "indicate which types of attachments to ignore"
This is the list of MIME attachment types that you do not want to do anything with. They are quietly ignored. They can be listed individually on multiple lines or comma or space separated on a single line.
ignore_types = text/x-vcard application/x-msdownload
or
ignore_types = text/x-vcard
ignore_types = application/x-msdownload

HM_TEXT_TYPES "list of types to be the same as text/plain"
This is a list of MIME types that you want hypermail to treat exactly as if they were text/plain. They can be listed individually on multiple lines or comma or space separated on a single line.
text_types = text, text/plain, message/rfc822

HM_PREFERED_TYPES "multipart/mixed types to present"
When mails using multipart/mixed types are scanned, this list of MIME types defines which part you want presented in the result.
prefered_types = text/plain, text/html


Order of Options Processing

Settings are processed in this order:

  1. From the program's hard-wired internal defaults (specified in options.h),
  2. From runtime environment variables,
  3. From command-line options,
  4. From the configuration file.

It is important to note that this is different from many programs. In Hypermail's case what you put on the command line DOES NOT override what is specified in the configuration file. This may change in the future.


Other Things

Filenames: In the specified directory, articles will be read out in the order that they were read in from a mailbox or standard input. Filenames start at zero and increase in this fashion: 0000.html, 0001.html, 0002.html, etc. In the same directory:

Sorting: In the date and thread index files, note that these lists are sorted by the date the articles were received by the system's mail daemon, not by the date they were written on. The order of articles in the date index may not necessarily match the order in which the article files are written and linked together. Because of this, it is a good idea to make sure the mailbox is sorted by date with the most recent messages towards the bottom.

Running Hypermail automatically: All that's needed to start archiving email messages is to set up Hypermail to do incremental updates in your /etc/aliases file. Here's what an entry might look like (the last line is one unbroken line):

#
# WU-FTPD Mailing List Archives
#
wulist: "|/usr/local/bin/hypermail -i -u -d /wu-ftpd/mail-archive -l \"WU-FTPD Mailing List Archive\""
After adding the entry, make sure newaliases is run to update the mail aliases. This entry will run Hypermail and update/create the archive whenever a new message is received. Hypermail also works well as a cron job. Because sendmail may run Hypermail as different users, you will want to make sure that archive directories and files are made readable and writeable by a trusted sendmail user (or read/writable by everyone if you can't do that) when they are created. This will ensure that there will be no problems incrementally updating the archive.

Including HTML in messages: One can include formatted HTML in message bodies by enclosing the HTML with the <HTML> tag (in either uppercase or lowercase). This tag must be on a line by itself:

   This text will not be parsed...
   <html>
   this text will be parsed as HTML.
   </html>
   This text will not be parsed...

There is no limit to how often the <HTML> tag can be used in an article.


Getting Help With Hypermail

If you are are looking for more information on Hypermail and its features and developmental status, checkout The Hypermail Development Center. Additional documentation and the hypermail development list archives are available there.


Getting Hypermail Software

Hypermail is available free of charge under GNU Public License. More details about the GPL are available at http://www.fsf.org/copyleft/gpl.html.

The source is available via FTP from ftp.landfield.com/hypermail/hypermail.tar.gz. This is a link to the tar file of the most current version.

The Hypermail Development Center also has beta development versions of hypermail available from time to time.


Credits

I would like to thank Tom Gruber, who originally designed and developed Hypermail in Common Lisp, for the basis of a GREAT tool.

I'd also like to thank Kevin Hughes for developing the initial C version of Hypermail. Kevin also provided a great deal of assistance with restarting the current hypermail development.

There are a great deal of people that also contributed to Hypermail's development. The Hypermail Development Center Credits page is an attempt to let you know just who they are.

Hypermail development is currently being fostered by <Kent Landfield>.


See Also

hypermail(1),   hmrc(4),   Hypermail List Configuration File.   and   Customizing Hypermail Pages

Please send any feature requests, bug fixes, and comments on Hypermail to <hypermail-bugs@landfield.com>.

Last updated October 9, 1999