Mailing List Etiquette (HPR Show 1740)

Dave Morriss


Table of Contents

Overview

In February 2015 I created a script to add a section to the monthly Community News show notes. The added section summarises the discussions on the HPR mailing list over the previous month. My script processes the messages archived on the Gmane site and reports on the threads it finds there.

In writing this script I noticed the number of times people made errors in replying to existing message threads and initiating new threads on the list. I thought it might be helpful if I explained some of the do's and don'ts of mailing list use to help avoid these errors.

List Etiquette Summary

Since this document is long I have included a brief summary here.

  • Threads - keep related messages together in time order
    • Use "Reply" in your mail client - it knows how to do threading properly
      • Use Reply to List or Reply To All - or the list or sender might not get a copy
    • Do not change the "Subject" line - make a new thread for a new subject
    • Do not try to start a new thread by replying to an old one - make a new thread for a new subject
    • Do not start a new thread to reply to an existing one - just use "Reply"
    • Do not reply to digest messages - digests are poison for threaded email
  • Formatting replies - tidy stuff up before sending it
    • Quote the text you are replying to - make it clear who said what
    • Trim the text you are replying to - you know it makes sense
      • Don't send back the PGP/GPG signature! - doh!
    • Do not top post - backwards reading like correspondents your and you unless
    • Use an email client that can do the right thing! - MS Outlook anyone?

Threads

The term thread, meaning a collection of messages relating to a subject, is quite old. It goes back to a time before the Internet. I certainly encountered it in the context of Usenet News before email existed. In current mail systems the term conversation is often used, but it still boils down to a way of ordering messages according to which one is a reply to another.

Many mail clients offer a threaded view of mail messages. I have used Thunderbird for many years, and now as a Debian user I use Icedove, the Debian version. Threading is enabled on a per-folder basis and to my mind Icedove does an excellent job.

While researching for this episode I found an add-on for Thunderbird called ThreadVis which displays a graphic at the top of a message visualising the thread to which it belongs. I have included an image of what this looks like:

Threaded messages
Picture: Threads in Icedove with ThreadVis

This is the thread from February 12th this year where Ken Fallon forwarded a message to the list from an organisation called Cybrary.

Notice how the thread is displayed in the Thunderbird pane. I have enabled threading in the folder and have expanded this thread by clicking on the triangle to the left, and the subject of each message is displayed. There are lines connecting messages, with indentation to show their level.

The ThreadVis display also does a nice job of showing the thread in my opinion, and it indicates that there was an external message which began the thread (the message forwarded in the first email from Cybrary), which it represents as a grey box. The messages are represented as coloured circles connected by lines. The lines show the reply relationship between messages and the length of the line represents the time between messages. Each of the messages can be viewed in a pop-up by hovering over the coloured circles, or can be opened by clicking the circle. The various authors in a thread are colour coded.

The only slight down-side I have found with ThreadVis is that the Global search and indexer option in Thunderbird has to be on. I had switched this off in the past because it made my Core 2 Duo workstation run slowly. On my current Core i7 with 16Gb RAM it seems to run just fine. ThreadVis uses this index to enable it to thread across all folders, which Thunderbird itself does not do.

How Email Threads Work

To look into email threads we need to examine the way email works in more detail.

The structure of an email message is defined by an Internet specification document known as an RFC (Request For Comments). The particular one covering email is RFC 5322 "Internet Message Format".

An email message consists of two parts, the header and the body. To be precise, when the message is in transit it is enclosed in a structure called the envelope, but that is removed upon delivery. We will not go into a lot of detail on the structure of email messages here. There are many sources of this information, such as the Wikipedia article on Email linked to at the end.

The message header contains lines known as fields in the format:

Name: Value

The body part contains the actual message content and can vary in structure from simple text to an arbitrarily complex hierarchy of MIME objects such as HTML, pictures, videos and so on. We will not look at the structure of this part of the message any more here.

Some examples of the header fields in a message are:

Date: Thu, 12 Feb 2015 15:08:12 +0100
From: Ken Fallon <ken@fallon.ie>
To: HPR Hacker Public Radio Mailing List <hpr@hackerpublicradio.org>
Subject: [Hpr] Fwd: Cross-Promotional Opportunties

These are frequently used by the mail client when displaying the message, as can be seen in the picture above.

The "Message-ID:" Header Field

Mail messages also contain a header field which contains an unique identifier for the particular message. This field is named Message-ID:, and contains a value which looks a little like an email address but is not, for example:

Message-ID: <54DCB3CC.3090906@fallon.ie>

This message identifier (the value part) is intended to be machine readable and is not necessarily meaningful to humans. The message identifier is intended to be a globally unique identifier for a message.

It is not mandatory for this field to be present according to the standards, but without it a lot of the important features of modern email systems fail. Email clients which do not generate a Message-ID: can be regarded as broken I think.

The "In-Reply-To:" and "References:" Header Fields

When an email client is used to reply to a message it generates header fields which refer back to the ancestors of the message. These fields are named In-Reply-To: and References:.

The In-Reply-To: field normally contains a single value which refers to the parent message. It does this by using the Message-ID: value from the parent message.

The References: field can contain much of the information required to build a thread. Sadly it cannot be relied on to contain all of the thread information. Normally it will contain the contents of the parent's References: field (if any) followed by the contents of the parent's Message-ID: field.

If the parent message does not contain a References: field but does have an In-Reply-To: field containing a single message identifier, then the References: field will contain the contents of the parent's In-Reply-To: field followed by the contents of the parent's Message-ID: field.

So the first reply to the above message contains the following fields:

References: <1423749551922.21495@cybrary.it> <54DCB3CC.3090906@fallon.ie>
In-Reply-To: <54DCB3CC.3090906@fallon.ie>

As expected, the In-Reply-To: field contains the contents of the parent message's Message-ID: field. The References: field contains the same, but, perhaps surprisingly, it also contains the contents of the Message-ID: field of the message that was originally forwarded to the mailing list.

That is because the first message in the thread contains the following header fields:

References: <1423749551922.21495@cybrary.it>
In-Reply-To: <1423749551922.21495@cybrary.it>
X-Forwarded-Message-Id: <1423749551922.21495@cybrary.it>

This is how ThreadVis was able to show that another message was referenced in the thread. It did this with a grey box to signify that the message was not present, as we saw.

So, what is a thread then?

You have probably realised from the description so far, an email message thread is defined by these links. Each message points to its ancestors, and if the whole collection of such messages is analysed it is possible to build details of the children of each message as well.

While researching this topic I came across Jamie Zawinski's description of his algorithm for analysing a thread as used in early Mozilla products. I found this fascinating since I'd worked out my own algorithm when trying to analyse the messages from the HPR mailing list on Gmane.

However, I appreciate that you might be less enthusiastic about this and will leave it here!

List Etiquette

Leaving the ultra technical stuff behind, let's look at the etiquette subject itself (also referred to as netiquette). There are a several behaviours which constitute good list etiquette. Maintaining thread consistency is one, and the other concerns the citation of the previous message.

Threads

Use "Reply" in your mail client

As we have seen, all you need to do to ensure that your reply on a mailing list is properly threaded is to use your mail client's Reply facility. This will perform the steps necessary to insert the correct headers and all will go along fine.

If your mail client can't do this then I'd be fascinated to hear about it. I'd guess you either need to configure things properly or discard it in favour of a properly standards-compliant client.

As an aside: you should pay attention to where your reply is going. In the case of the HPR list it is not configured to direct replies to the list. In this situation most mail clients will reply by default to the sender, and this will usually not include the list itself.

My email client has a Reply to List function, but that does not send a copy to the sender. It also has a Reply to All option which replies to the list and the sender. I usually use the former since I can usually assume that the sender will get the reply through the list.

Do not change the "Subject" line

It's usually seen as bad etiquette to change the Subject: field in a thread. Sometimes people will correct a misleading subject, but for clarity this should be done as follows:

Subject: Aardvarks
Subject: The price of beef [was "Re: Aardvarks"]

Keeping a reference back to the original subject makes it clear that the change was well considered and (probably) appropriate.

Do not try to start a new thread by replying to an old one

Sometimes you will see users of a mailing list trying to start a brand-new thread by replying to a message in an old one, and using a new subject. This is bad etiquette and can also be counter-productive in some circumstances.

For example, the script that summarises message threads in the HPR Community News show notes will not see such a message as a new thread and will not list it in the summary. See the example in the image above, where an existing thread is used to try and start a new topic with the subject "Intro and Outro". Notice how the summary in the notes for the HPR Community News for February 2015 does not include the topic "Intro and Outro" for this reason.

Do not start a new thread to reply to an existing one

Another common mistake when intending to join a conversation is to create a brand new message and copy the subject of the relevant thread. As we have seen, this will result in the message not being joined to the thread because the mail client will not be able to generate the necessary headers.

However, some thread analysing systems try very hard to get around this problem. The strategy is to look for "orphaned" messages like this with a subject matching an existing thread, then join them into this thread at a position based on the time stamp. The Gmane system does this, as does Jamie Zawinski's system (according to his description). My HPR summary script also does this. However, Thunderbird does not do this when displaying threads.

Of course, no algorithm is able to perform a repair like this if the subject line has been altered, so please do not rely on it.

Do not reply to digest messages

Many mailing list systems provide a digest facility, where all messages in a period such as a day, or a certain number of messages, are bundled up and sent out together, rather than messages being sent individually. This can be a great convenience if the list is very busy or contains newsletters or other "read only" material.

Many mailing list systems, including "Mailman" used for the HPR list, are able to generate plain text or MIME digests. The plain text format conforms to RFC1153 which is a good format for human readability, but removes all headers from each message, including those required for threading. The MIME format sends each message as a MIME attachment to a digest message. This format sends the full headers, but since they are embedded in another message, not all mail clients can deal with them.

If the list subscribers use it for discussions, receiving digests can be a problem if you ever want to reply to anything. Replying directly to a digest will not result in your reply being part of a thread. The message you are replying to will probably be one of several, and will be encapsulated in the digest message. The digest will not usually convey the identifiers of its constituent messages, and even if it does, most email clients are unable to reply to a message within a message.

For a low traffic list like the HPR list it would be better not to subscribe to the digest list. If you do, it would be best not to reply to these messages.

Formatting replies

When replying to a message it is highly desirable to format the original message and your reply, for reasons of clarity, legibility and economy. See the Wikipedia article on posting style for an in-depth treatment.

Many mail clients offer the ability to perform formatting on the original message when replying, and it is recommended that this feature be used wherever available.

Quote the text you are replying to

It is regarded as bad etiquette not to mark the original text in a reply. The method used most often is to start with a line in a format similar to:

On date time, author wrote:

Where date and time are the time stamp for the original message and author is the sender's name and/or email address.

The text of the original message then follows marked with the characters ">" (a greater than sign and a space). For example the initial reply might look like this:

On 01/01/1505 20:30, Fr. Benedictus wrote:
> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod
> tempor incididunt ut labore et dolore magna aliqua.

If a third person then replies to the reply, they should also do the same thing, keeping the original quoted reply, such as:

On 01/01/1505 20:34, Fr. Alessandro wrote:

> On 01/01/1505, Fr. Benedictus wrote:
> > Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do
> > eiusmod tempor incididunt ut labore et dolore magna aliqua.
>
> At vero eos et accusamus et iusto odio dignissimos ducimus qui
> blanditiis praesentium voluptatum deleniti atque corrupti quos dolores

Most, if not all, mail clients will do this, or something similar, for you.

Also, many mail clients will make this layout much easier to read by various methods. For example, a Thunderbird add-on can colour the different levels of quotes to make them easier to follow. Another will collapse all quotes, replacing them with buttons which can be clicked to expand the collapsed quoted text.

Trim the text you are replying to

It is considered bad etiquette to leave the entirety of the original text in the reply. Some degree of trimming is most desirable, but this must be done so as to leave the meaning intact. Someone reading the thread in the future should be able to understand the conversation.

Salutations and signatures can and should be removed from the original text.

It is important to remove the sender's PGP/GPG signature if there is one. Without doing this mail clients which understand these items will become confused about what is being signed and by whom.

It is my experience that clients which are capable of signing and encrypting/decrypting messages will do this removal for you.

Do not top post

The term "top posting" refers to the practice of placing the reply before the text of the previous message. This is generally regarded as bad etiquette since it reverses the normal flow of conversation and requires the message to be read from the bottom up. In the case where several people have replied to a message, some top posting and others replying beneath, the end result can be almost indecipherable.

Most mail clients will offer the facility of positioning your text after the original text, and this feature should be enabled.

Some people feel that a top posted reply is more convenient in that they don't have to scroll past all the preceding material to read it. However, using an email client which can collapse and expand quotes is a good compromise here. If all but the last reply is collapsed this shrinks the message down considerably, yet the intermediate text can be consulted if necessary.

The screenshot below shows a reply to a message where the previous quoted text has been collapsed. Ironically the hidden message started with a top post!

Message with collapsed quote
Picture: Message with collapsed quote

To be fair, the subject of top posting seems to be controversial and possibly in a state of flux. While preparing this show I found a lengthy discussion of the right way to reply to a mailing list on the Mailman-Users mailing list. You can read it here. There are some interesting points made in this thread, including the fact that the authors of many modern mail clients are now forcing users away from the more normal posting style. I certainly experienced this in my working life when the installation of Microsoft mail products in the organisation changed posting behaviour for the worse.

Use an email client that can do the right thing!

As you might have noticed if you have read the Wikipedia article on posting style below, some mail clients are not capable of following these guidelines. Microsoft Outlook seems particularly challenged in this area, so if you can, avoid it and other clients like it!

  1. Gmane archive of the Hacker Public Radio mailing list: http://dir.gmane.org/gmane.network.syndication.podcast.hacker-public-radio
  2. Wikipedia article on message groupings referred to as conversations, topic threads, or threads: http://en.wikipedia.org/wiki/Conversation_threading
  3. A brief note on how to punctuate the phrase "do's and don'ts": http://www.visualthesaurus.com/cm/wc/dos-and-donts-or-dos-and-donts/
  4. Wikipedia article on Usenet: http://en.wikipedia.org/wiki/Usenet
  5. Thunderbird add-on ThreadVis: https://addons.mozilla.org/en-US/thunderbird/addon/threadvis/
  6. Wikipedia article on the RFC document: http://en.wikipedia.org/wiki/Request_for_Comments
  7. Text of RFC5322: http://www.rfc-editor.org/rfc/rfc5322.txt
  8. Wikipedia article on Email: http://en.wikipedia.org/wiki/Email
  9. Wikipedia article on MIME used in email: http://en.wikipedia.org/wiki/MIME
  10. Description of a threading algorithm from Jamie Zawinski: http://www.jwz.org/doc/threading.html
  11. Text of RFC1153: http://www.faqs.org/rfcs/rfc1153.html
  12. Wikipedia article on posting style: http://en.wikipedia.org/wiki/Posting_style
  13. A recent large thread on the Mailman-Users mailing list discussing the subject of replying to lists: http://www.mail-archive.com/mailman-users%40python.org/msg66089.html