Dovecot, SES email on Amazon LInux

MIME emails are not designed for structured analysis and manipulation. There is little functionality available. To have otherwise would require a new email standard. As companies spend millions of dollars on their services, we can benefit from that work at around AUD $75 subscription per year. However, if we love our technical work, (or have cost restrictions) exploring Dovecot is good.

Some examples of limitations:

If you append content to the raw email file, it is not displayed and is removed when the email is forwarded. One can specify content as base64 but still use plain text. One cannot treat sections as modules in order to work with them – e.g. determine if a section is base64 and then examine for tracking code. Tracking tools as advertised by various vendors pertains to non-base64 text, so what is the point when base64 has tracking? One cannot change the style of the subject header – e.g. colour it red. A big list continues.

Paid services have their own characteristics:

For example, Rackspace has an interesting service, but the minimum price is USD $10 / month ex. gst. Members of the public cannot afford this.

In Amazon WorkMail we can automate inbox backups to an S3 bucket rather than deal with complex backups in MS Exchange. Activating DMARC is a click of a button. Workmail is easily de-configured and cancelled.

In MS Exchange we get an integrated Calendar function. MS Exchange requires configurations to setup DMARC. MS Exchange is complicated, involving the Azure and Defender services.

In years to come, will BIMI logos be available to the general public, and if so, will Amazon update their service? All services have non-identical filtering rules, whereas a basic Dovecot service would need shell scripting.

There are real solution problems for companies. For instance, migration from one provider to another is no simple question. Even a personal user can go through the hoops to change providers – e.g. backup files to a friendly client like eM Client, change provider, wait for security settings to propagate with some failed emails along the way, use eM Client to put the files back to their folders with original timestamps and so on.

Various encrypted services sound amazing or promising. The reality for me was highly disappointing, having to cancel my service. What is the point of waiting several seconds each time you try to view your list of emails, or have no ability to automatically forward emails.

Then there are the issues of security, blacklisting and grey listing. We can’t get mixed up in all this. The idea of Dovecot is to have control over your own service, at the same time without complications as we use the SES Relay Service.

If you place Dovecot/IMAP onto an existing web site’s Linux server and the same domain name, there is basically no cost, or for additional users. For instance, 1GB disk space is 16 cents per month. It is possible to install Dovecot onto a Linux instance without a website, so long as we have the SSL certificates.

I do see higher risk in placing a serious email service onto the same hardware as a website. We mitigate this with a separately mounted disk volume on /dev/xvdb and data backups. Amazon uses Security Groups to block unwanted ports, which is strong security.

We use the AWS SES Relay Service. This takes care of email headers, security and use of IP addresses – we won’t be blacklisted and we have an email health dashboard.

We can use SES blocking on IP addresses, or via Lambda, domain names and specific users.

Emails (via SES Email Rules) are stored into an S3 bucket. This means we can run a shell script to archive all inbox emails for say, 365 days, at virtually no cost. It does not matter if we delete an email we want to see again. We can also base64 encrypt archived emails.

As we are running Linux and using AWS services, we can do some nice tasks via shell scripts that use the “aws” command.

You can manage emails under a /data directory (or any nominated folder) with crontab automated backups at virtually no cost to S3 storage. This separates data in a best practice way.

If you ever decide to remove the service permanently and its associated email addresses, you can use the SES Email Rules to forward emails to your new email address even though the Linux instance and Dovecot service are completely removed. Forwarding cost is negligible.

After we develop a stable system, we use a convenient Amazon snapshot as best practice along with our own scripts for frequency of backups.

Is Dovecot a fully developed commercial product?

To put into context, there are many email services that “do their own thing”. It is fair to say Dovecot is a secure and stable IMAP service.

We use manual editing to add/remove users and maintain the system. For instance, we renew the SSL certificate each year. There can be unexpected problem solving. It is IT after all. We manually add or remove users with several configurations needed.

A basic installation does not have filtering rules apart from shell scripting. If we get into these complexities, we may consider developing knowledge about a virtual mail system. We are not looking at use of vmail or a database. We ensure a good user password of at least ten characters, but please avoid the “!” character which is a headache in shell scripts. Long passwords may one day be broken into by quantum computing, but not for now. We cannot stop hackers attempting to log into our imap service. We don’t see blocking IP addresses as viable, but even so, doing this on ports will stop the service working through SES.

I also do not configure the /etc/dovecot/conf.d/90-quota.conf file for multiple users, as I have not spent sufficient time testing it. I provide configs for a single user.

As we use the SES/S3 Bucket structure, Dovecot does not directly receive emails into the inbox, meaning some Dovecot plugins may not work – e.g. encryption. Emails are transferred from S3 as .eml formatted files.

We use the Unix/Linux crontab facility to check the S3 bucket once a minute as we must copy files into the user’s Dovecot inbox in the Maildir/new folder. If we were to do this once every 30 seconds, we run two crontab entries on two identical shell scripts, excepting the second script has a “sleep 30” command. Crontab has been built deeply into the Unix kernel without impact on performance. (I don’t know how it is built with Amazon Linux.)

Dovecot does not allow email user names that already exist in the /etc/passwd as system names. You are not able to use mail@domain.au. I have tried various “tricks” such as renaming “mail” to “mailer” in /etc/passwd, /ect/shadow, /etc/group and /var/spool entries, but something stops the emails syncing on New Outlook even though they will sync in Old Outlook. I think at this stage the best I can do is “email” instead of mail, however emails can be sent to mail@ in order to arrive in the person’s S3 bucket without issue.

You would need to test after adding the postfix package, look at /var/spool with “ls -l”, do a dnf upgrade to current release, and see if the changes held on /var/spool, configure a basic postfix setting and test an email under mail@, then continue to the Dovecot/postfix full configurations and see how it goes.)

Whilst an IMAP service connects to an email client quickly, being port managed, syncing files may not be as fast as a commercial service. If for some reason syncing is having issues, (it shouldn’t) one can manually restart the service from a terminal command and re-sync on the client end. I personally have found MS Exchange the fastest (and best) system I have used for speed and sync. I would suggest no system is without peculiarities or hick-ups over time. And, who knows what price rises are on the way?

A Dovecot service cannot be installed without many Amazon AWS services being first established and tested, safely under your belt. There is no way around this longer term challenge. There are a few key ideas to be aware of as you install a system. For example, certain errors indicate your instance static IP address is either blacklisted or grey listed, meaning you have to delete that IP address before you go live. (Live being other people using your address.)

Certainly a backup plan is needed, fair enough, but what if there is an unknown in the future? We have dependency on certain Linux packages being installed. If this changes, how do we proceed? What if the Dovecot software changes, introducing an incompatibility or a software error? We will have risk around the SSL certificates and the version of Openssl we are running. These interrelated packages can suddenly bring a system to a halt. For instance, my use of the free Let’s Encrypt certificates became unusable on a newer release of Linux2. When adding Dovecot to an existing domain web server on Linux2022, a critical file would not install. Linux 2022 already has saslauthd, but the package is manually installed as cyrus-sasl on Linux 2023. I have been able to manage these changes. (There is no point providing Linux2 configurations at this point in time, but I do have the documentation.)

I think it is rewarding to engage in the configuration of an email server.

I would like to give you an idea of the AWS services you need running, as a checklist. This will make sense as you learn to configure them, or if you have done so already. It is not intended for an understanding at this point, but to show there is considerable scope involved.

SMTP Credentials, added by SES to IAM. We keep a copy of the keys on our PC and never lose them.

IAM – We set up a role to be able to add Lambda functions.

Lambda – We create functions in Oregon for Australia, using the above IAM role to process e-mails in SES.

S3 Buckets – We add an e-mail bucket in our local regions (Australia, not Orgeon), with permissions via the JSON code.

SES – We verify our domain name, ensuring Route53 has the entries for DKIM etc.

SES – We add an SES Email Rule to receive emails into our S3 bucket, and forward them to another address such as gmail. These rules need Lambda functions to do these file transfers. When Dovecot is working, we remove the forwarding rule.

SES – We verify at least two email addresses for our testing. For example, and external gmail address, and an admin@ domain address. As email is not fully working at this point, we manually view an email verification file in the S3 bucket. Once verified, we can continue testing correctly.

SES – At some stage we ensure we are out of Sandbox mode for our entire Amazon AWS account, if not already from prior work we did.

Cloudwatch – We view logs and errors in Oregon’s Cloudwatch logs.

IAM – We create a role that gives the Linux EC2 instance access to AWS resources, and then add that role to the Instance.

EC2 Instance – we have e-mail ports open in addition to our normal http/https/ssh ports.

S3 – Our email bucket has public access. We create a subfolder called .archive to work with our shell scripts – no other folders unless we modify our shell scripts. Our scripts use commands that ignore the hidden .archive subfolder.

aws – We ensure the Amazon Linux aws package is installed, and have the ~.aws directory and config file configured. This uses the SMTP keys mentioned above.

For preliminary testing of the above configurations, we should be able to send an email, e.g. to “fred@snotbat.com” and see it land in the S3 Bucket. Our Lambda function for forwarding emails can forward it to any other address via our SES Email Rule.

Once you know this all works and is running, we are ready to add the Dovecot service.