Setting Up OAUTH2 Support for Fetchmail and Postfix

This web page describes how to piece together various patches, plugins, configurations, and scripts to support relaying the local machine's mailbox through gmail with OAUTH2. Then you can use any mail client that uses your local machine's UNIX/Linux mailbox to read and send email. (I personally prefer mutt, but any email client, gui-based or not, should work.)

It might also work to serve out the local mailbox(es) to other machines using basic POP/IMAP/SMTP services to bypass gmail OAUTH2, although I haven't tested or documented that.

It is kind of complex to get all the pieces set up, but on the plus side, you can leave out (incoming only?), replace (prefer something other than postfix?), or tweak the details of any of these pieces as desired.

Table of Contents


What is OAUTH2?

OAUTH2 is a fancy mechanism for apps/websites/etc to delegate arbitrarily complex multi-factor login capabilities to a central authentication management web site. It is primarily driven by Google for use in mobile-phone-based apps, but can be used in other ways as well.

It is usually optional on the part of the individual user, but in some cases can be required. One such case I've encountered is when my $DAYJOB offloaded company email to Google's business "G Suite" service, and configured it at the organization level to require the use of OAUTH2. It is really just gmail, even though the hostname belongs to $DAYJOB instead of using "gmail.com".

Basically OAUTH2 works as follows:

"OAUTH2" is the general name for the overall system. "XOAUTH2" is the name of the original scheme for embedding the access token into POP, IMAP, SMTP, or similar protocols, and is still supported by Google and others. The newer standardized way of embedding the access token is called "OAUTHBEARER", and is documented in various internet standard RFCs. OAUTH version 1 is a little different and not directly compatible (it requires various higher level digital signatures in token operations, instead of simply relying on TLS/SSL).

For more details about how it actually works under the hood, see Google documentation such as Using OAuth 2.0 to Access Google APIs, A Run-Through of semi-manually going through the various steps with the aid of small-ish "oauth2.py" script, etc. (My "fetchmail-oauth2.py" script (below) was derived from this script, and should still support the arguments used by the run-through.)

It is doubtful that oauth2 is really more secure than just picking a strong password and protecting it carefully. Especially for "application specific passwords" generated by the server. The extra complexity significantly expands the vulnerability footprint, even if in some vague theoretical ways it might have some slight advantages.


Common Setup

These instructions are mostly geared towards "gmail" and/or "G Suite", but can probably be adapted to other providers that use OAUTH2. For others, you may need to find an equivalent to the "Google API Console" and research various optional settings for fetchmail-oauth2.py's config file (some settings not mentioned below default to Google).

  1. Configure your gmail (or equivalent) account to enable IMAP and/or POP access. If it will let you set up an "application password" instead of OAUTH2, that would generally be a whole lot simpler and arguably just as secure. If not, continue:
  2. Download oauthbearerScripts-2023-01-01.tar.bz2. It contains various patches and scripts that are needed by the instructions below.
  3. Pick where to "install" scripts, a config file, token files, etc.

    "Makefile" and "lockedMake" probably belong under /etc/postfix, some files are patches only needed when rebuilding packages, and others should go somewhere under your home directory. Perhaps re-use existing directories where you've put cron jobs or other mail-related scripts and configurations. I'm intentionally avoiding specific recommendations to make it slightly less likely worms and rootkits will try to look for sensitive information in variable locations, although that is a pretty weak form of so-called "security".

  4. Copy "fetchmail-oauth2.py", "cron.fetchmail", and "cron.oauth2" to a desired scripts directory. They don't need to be in your PATH. You can rename these scripts if desired. If you rename them, then fix up the cross-references while editing them below.
  5. Edit the copy of "cron.oauth2" and customize as desired. You at least need to specify a config file name, which you'll actually generate below. It should probably be somewhere under your home directory unless you want to experiment with running oauth2 and fetchmail as a separate no-login "service" user so that the token files aren't readable by you, or any malware running as you.

    You might want to comment out the "sudo /etc/postfix/lockedMake" line until you get to the postfix setup section below.

  6. Edit the copy of "cron.fetchmail" and customize as desired. You at least need to either specify a log file name, let cron email the output (very noisy in your mailbox), or feed it to "/dev/null".
  7. Refer to Google's Instructions for how to use the Google API Console to generate a "Client ID" and a corresponding "Client secret" for fetchmail and postfix to use for oauth2. This console tool doesn't seem to be very intuitive, so here are some notes about how I've navigated through it:
  8. Create a configuration file for the "fetchmail-oauth2.py" script. For full details, see the output of "fetchmail-oauth2.py --help", but the following example may be adequate:
    client_id=FROM_GOOGLE_CONSOLE_ABOVE
    client_secret=FROM_GOOGLE_CONSOLE_ABOVE
    refresh_token_file=/FULL/PATH/TO/REFRESH_FILE
    access_token_file=/FULL/PATH/TO/ACCESS_TOKEN_FILE
    max_age_sec=1900
    

    Some other settings (not in example) default to Google, so if you are using some other provider, you might need to research them.

  9. You probably want to "chmod 600 THE_CONFIG_FILE" so that no one else can read it.
  10. Obtain initial tokens by running "fetchmail-oauth2.py -c /PATH/TO/CONFIG_FILE --obtain_refresh_token_file". It will print a URL that you need to log into using a browser. After you do, the browser will print a code you need to cut and paste into the still-running script.

    Depending on how your account is setup, you might be able to make use of the extra "googleAuthenticator" script while logging in. When you first set up the server to use Google Authenticator, click on the "I can't scan it" link to get the secret authenticator string. Strip out the spaces and save it somewhere (probably your password manager). Then when you need to generate an authentication code, run the "googleAuthenticator" script and paste the secret into it. It will then print the current 6 digit code. Your system clock needs to be reasonably accurate. (It doesn't use command line arguments, to try to keep the secret out of .bash_history.)

  11. You may occasionally need to repeat the previous step, but not regularly. For example, it may be necessary after you change your password, or after some other major change on the server side. It might be worth wrapping the command in a short shell script so you don't need to remember the correct arguments.

    UPDATE: In May 2022, Cajus Hahn (cajus_hahn AT arcor dot de) reported that the renewal token only seems to work for one week, but I haven't seen that myself.

  12. DIFFICULTY 2022: Google has indicated that it is in the process of disabling "out of band" token management, to be finalized by October 2022. Presumably this means using copy/paste to the browser will no longer work. I haven't had a chance to dig into the details, but presumably this will involve reworking things to embed the tokens in a custom URL or something, rather than just cut-and-paste from the browser to the still-running script.

    UPDATE (2022-10-16): The simplest way to get a non-OOB option working seems to be to edit the REDIRECT_URI line in the script to read something like "REDIRECT_URI = 'http://127.0.0.1:8018/fetchmail'" Pick any unused port number. After logging in, your browser will fail to load the redirect web page, but you can carefully select and copy/paste the token part from the middle of the URL out of your browser's address bar and into the waiting script. FUTURE: The URI can't currently be overridden with a config file. It also would be more convenient if the script actually ran a small localhost-only web server to capture the token, and you didn't need to copy/paste anything. Maybe someday I'll get around to such enhancements, or maybe someone else will, but in the meantime, this technique works.

  13. Microsoft Office 365: My $DAYJOB switched providers to Microsoft Office 365 a couple of years ago, but I still haven't taken the time to track down how to adapt these instructions to it. Probably need to register a client some other way. There are indications that a tenant's aministrator can lock down registering additional clients, so instead it may be necessary to "borrow" some other client's ID and secret.

    UPDATE (2023-01-01): Alexander Zangerl (az at breathe-safe dot com) has reported success using Thunderbird's client_id and client_secret for O365 (which I hesitate to post here, to avoid getting into a three-way war over pretending to be a different client, despite the fact that the secret probably isn't well protectected...), along with the following values (some of which probably require modifying scripts, not just the config file):

    imap_server=outlook.office365.com,
    smtp_server=outlook.office365.com,
    scope="https://outlook.office365.com/IMAP.AccessAsUser.All
           https://outlook.office365.com/POP.AccessAsUser.All
           https://outlook.office365.com/SMTP.Send offline_access"
    auth_url=https://login.microsoftonline.com/common/oauth2/v2.0/authorize
    token_url=https://login.microsoftonline.com/common/oauth2/v2.0/token
    redirect_uri=http://localhost
    max_age_sec=1800

ALTERNATIVE: Some people have reported success with a different sasl plugin: https://github.com/tarickb/sasl-xoauth2 Jamen Lang (jamenl at gillettewy dot gov) mentioned it it 2020, and Freek de Kruijf (f.de.kruijf at gmail dot com) mentioned it in mid 2022. He has also indicated an intent to document his procedure at https://en.opensuse.org/Configure_Postfix_to_send_email_with_xoauth2_HOWTO, although it is currently still a stub as I write this. From its github readme, this plugin looks like it is much more actively developed than my cobbled together instructions, and it comes with a oauth script that is presumably similar to my own. For postfix it can apparently use the renewal token to get new access tokens on demand instead of requiring a cron job. It might help with the DIFFICULTY and/or Microsoft cases mentioned above, although I haven't dug deep enough to be sure. The readme instructions are focused on outgoing email (postfix), but given that the same tokens can be used by both, maybe it could also be used with incoming email and/or fetchmail. Currently I intend to look at this more carefully when I have some time.


Setup Fetchmail

Officially released versions of fetchmail do not currently support any form of oauth, but there are at least two ways to get a version that does:

Option 1: Patch fetchmail 6.X

The "fetchmail-oauth2-passwordfile-*.patch" files are patches that can be applied to the 6.3 or 6.4 branches of fetchmail. Fetchmail 6.X development has continued for far longer than I would have expected (with only an unofficial alpha releases of 7), and that development keeps (slightly) breaking patches. "...v2.patch" applies to 6.3.* and early 6.4 development, "...br64-v3.patch" applies to 6.4.1, "...br64-v4.patch" applies to some intermediate 6.4.x versions, "...br64-v6.patch" applies to 6.4.8 through at least 6.4.27 on gentoo, and "...br64-v8.patch" applies to 6.4.34 and later (and includes an increased PASSWORDLEN to hopefully allow its use with Microsoft's long O365 tokens).

Andrew C. Aitchison (andrew at aitchison dot me dot uk) has supplied two more versions that I haven't tried: "b64-v7.patch" appears to be the same as v6 except it corrects line numbers and nearby context lines to apply more cleanly (but identically), and 650b7.patch is intended for the 6.5beta7 release. Francois Manchon (Francois.Manchon at sita dot aero) supplied fetchmail6.4.34-xoauth2.patch, which I haven't tried but apparently adds support for POP3 (like the unreleased version 7.0).

Once you build and install the patched fetchmail, configure ".fetchmailrc" using something similar to the following:

poll imap.gmail.com protocol imap
   auth oauthbearer username "USER@gmail.com"
   passwordfile "/PATH/TO/ACCESS_TOKEN_FILE"
   is LOCALUSER here ssl sslcertck sslproto tls1.2+
The way these older, rough patches are constructed, it actually implements xoauth2 only, but "oauthbearer" is a synonym for future compatibility with version 7 below. This version of the patch only reads the token file once, and so fetchmail's daemon mode is unlikely to work. Run fetchmail using cron instead. It also only supports IMAP (not POP).

I have a report about some trouble trying to apply the patches with "git apply" rather than "patch". Most of the time my testing is mostly automated and consists of the gentoo mechanism described below, which is basically based on "patch"ing the "stable" version of fetchmail, so I might be slow to notice problems that this approach doesn't test. "git apply" is pickier than "patch" about failing if nearby lines have changed, so if you have trouble applying the patches you might try "patch" and/or basing on a slightly older "gentoo stable" version of fetchmail. (See https://packages.gentoo.org/packages/net-mail/fetchmail for information about which versions are currently testing vs stable.)

If you are using Gentoo Linux, then it is possible to use the /etc/portage/patches (documentation) feature to automatically apply the patch when re-emerging fetchmail:

mkdir -p /etc/portage/patches/net-mail/fetchmail
cp fetchmail-oauth2-passwordfile-br64-v6.patch /etc/portage/patches/net-mail/fetchmail/.
You could optionally try to tweak names to restrict the patch to specific versions; see documentation link above.

Option 2: fetchmail 7

The development branch for fetchmail 7 has included an enhanced version of this patch for years. Additional improvements include: daemon mode will re-read the passwordfile regularly, it actually implements (and prefers) standardized "oauthbearer" in addition to "xoauth2", and adds POP support. See merge requests.

Fetchmail 7 is apparently going improve how to configure ssl, so the "ssl" keyword in the configuration file will probably need to change to "sslmode wrapped" to avoid warnings from fetchmail.

Unfortunately, it isn't clear when 7 will go "live". There was an alpha release over a year ago, but 6 has had various releases since then, including a new "6.5" branch...


Setup Postfix

  1. Install Postfix. It is usually available pre-packaged as part of most Linux distributions. It needs to be configured and built to support SASL plugins, which is probably the default. On Gentoo Linux, enable the "sasl" USE flag.
  2. Install the xoauth2 SASL plugin from the "release" tab of https://github.com/moriyoshi/cyrus-sasl-xoauth2 into a place where SASL infrastructure can find it. You can install it manually, or on Gentoo you can use my "cyrus-sasl-xoauth2-0.2.ebuild" ebuild file. Copy the ebuild into your own custom portage overlay (perhaps under a "dev-libs/cyrus-sasl-xoauth2" directory), run "ebuild cyrus-sasl-xoauth2-0.2.ebuild digest" on the copy, and then "emerge cyrus-sasl-xoauth2". See also the plugin's own instructions, although they are a bit terse. There may be other tools that can use this plugin (both as a client and as a server), although I haven't tried any.
  3. Configure and copy the custom postfix configuration "Makefile" and "lockedMake" files into the "/etc/postfix" directory. In the "Makefile", you need to set up the XOAUTH2_MAP variable appropriately for one or more users, and you may want to adjust exactly which ".db" files the "all" rule depends on / generates.
  4. Use "visudo" or similar to set up a sudoers rule similar to "LOCALUSER ALL=(root) NOPASSWD: /etc/postfix/lockedMake". Replace "LOCALUSER" with your username.
  5. Create an "/etc/postfix/tls_policy" file with the line "[smtp.gmail.com]:587 encrypt", since xoauth2 generally requires SSL in order to work.
  6. Create one or more lines in "/etc/postfix/relayhost_map": "USER@HOST [smtp.gmail.com]:587".
  7. Stick any "extra" (non-xoauth2) passwords in "/etc/postfix/saslpass". Or just create an empty file. Unfortunately, such passwords might not actually work because of limitations of the "smtp_sasl_mechanism_filter" setting below. As a workaround, at least you can set it up to work by just commenting/uncommenting the "smtp_sasl_mechanism_filter" setting as necessary.
  8. Postfix has lots of configuration settings for "/etc/postfix/main.cf". The following snippet shows most of the settings that are relevant to oauth2-based mail relaying, but not any general settings:
    # single or default/fallback relay:
    relayhost = [smtp.gmail.com]:587
    # optional multiple relays:
    sender_dependent_relayhost_maps = hash:/etc/postfix/relayhost_map
    
    smtp_sasl_password_maps = hash:/etc/postfix/saslpass-autogen
    
    smtp_sasl_auth_enable = yes
    smtp_sasl_security_options =
    
    # Filter out other mechanisms.  It might work without this,
    # but not reliably.  If you want multiple relays and only
    # use xoauth2 for some of them, then it is probably best to
    # setup multiple transports in master.cf with different filters,
    # and use sender_dependent_default_transport_maps to select which
    # transport.
    smtp_sasl_mechanism_filter = xoauth2
    #smtp_sasl_mechanism_filter = plain, login
    
    smtp_tls_security_level = may
    # The "may" above should work most of the time, but xoauth2 won't
    # work without ssl:
    smtp_tls_policy_maps = hash:/etc/postfix/tls_policy
    
    #### Optional related:
    # Potentially useful if the machine wants to relay multiple different
    # local users to gmail:
    smtp_sender_dependent_authentication = yes
    
  9. Also configure the rest of postfix appropriately. Set it up so that "local" mail has a separate appropriate hostname, and stays local. See Postfix Documentation.
  10. If you commented out the "sudo /etc/postfix/lockedMake" line in the cron.oauth2 script above, you should uncomment it now.
  11. FYI: It should be safe for multiple users to set this up, as long as they all use oauth2 (see FUTURE below). They'll all need to be in the various map files, in the XOAUTH2_MAP variable (Makefile), and be allowed to run lockedMake via the sudoers file. The "lockedMake" utility is designed to avoid actual corruption by using a lock, although there is still a remote possibility that if run simultaneously by multiple users, timestamps might end up such that some updated token values might be delayed until the next time one of the access token files change (if so, it will continue using the old value until then).
  12. Postfix doesn't directly/robustly support using different authentication schemes for different email account relays, which would need different smtp_sasl_mechanism_filter values. However, Christian Hansis (christian at eeproto dot com) reports success by defining an additional transport option in master.cf, selecting among transports using sender_dependent_default_transport_maps, and configuring them with different smtp_sasl_mechanism_filter values. I haven't tried it, but it sounds promising.

Older versions of this web page outlined future ideas for making the configuration of multiple relays easier, but the above multiple-transport option and/or the ALTERNATIVE plugin (further above) apparently cover those ideas fairly well.


Final Setup


Gmail Troubleshooting Notes

A few years ago one of my gmail accounts spontaneously stopped working in fetchmail with just my normal password. Other gmail accounts were still working. I was never able to figure out why it just stopped working, but here are various details:


History of This Page


By Matthew Ogilvie. [online copy of this page], [up/unrelated software]
I last updated this line on 2023-01-01.