The technology behind E-Mails (or ‘emails‘) always interested me. This is because in some sense it was way ahead of its time by being a decentralized communication system. Unlike WhatsApp, Telegram or other messenger services, where the service provider has a centralized server (or nowadays a network of servers) processing all the messages; with emails, everybody can host their own mailserver to send and receive messages to everybody else. If Meta or Telegram decides to shut down their server, all communication is lost and the service cannot be used anymore. But with email, you can just move to another server (as long as you own your own domain at least).
One of the major components that power our modern email systems is the IMAP4 protocol. It is used to connect your email client to your mailserver. It’s the language used by the client to ask the server for the newest unread emails and the server responds with its contents. POP3 is (or was) also very popular and still widely supported but not used so much because it comes with a few drawbacks (a topic for another day). You probably have heard about SMTP as well, and this is the protocol that is used to send emails.
But those protocols alone don’t allow us to send emails. Emails heavily rely on other technologies like Domains, DNS, IPv4, SPF, DKIM, DMARC, SSL/TLS, S/MIME, PGP, NTP. In general, Emails are not too different from websites as they share a lot of the same technology (and nowadays you can even send HTML emails).
However, in this blog post, I want to focus on IMAP, which stands for Internet Message Access Protocol. IMAP4 was released in 1994 as RFC 1730. The most recent version is IMAP4rev2 (released in 2021), as defined in RFC 9051 but most mail servers (as far as I can tell) still use IMAP4rev1 as defined in RFC 3501. It can be used to retrieve, manage, and manipulate email messages stored on a remote mail server, allowing users to perform actions such as searching, flagging, deleting, and organizing messages into folders while maintaining synchronization across multiple clients. Similarly, HTTP is a text-based protocol that relies on human-readable text commands and responses for communication between clients and servers.
So now, enough introduction - let’s get right into business. How do we connect to a mailserver?
Like with HTML, we can use the telnet utility to build up a simple TCP connection to a mailserver:
After connecting we are greeted with OK followed by a list of ‘capabilities‘, which is a list of features supported by the mailserver in the current state. It also shows us which IMAP version is supported. Sometimes (but not in the screenshot above) the server will also tell you what kind of software it is running. Most of the time you will see “Docevot“ being mentioned. Dovecot is a secure open-source IMAP & POP3 server running on Linux.
If we are fast enough and are not getting kicked out of the session because we idled too long, we can try to log in to authenticate ourselves.
In order to do that, we have to understand how the IMAP commands are structured.
<tag> <command> [arguments]
The tag prefix is supposed to be a unique tag within the session used to match the request and response. It can be anything but has to start with a letter and cannot contain spaces. While it’s supposed to be unique and typically look like A001, I’m lazy so I’ll always use just a. The server will then respond with:
<tag> <status> <response>
There are multiple ways to authenticate. The LOGIN command is the most straightforward one but is also considered insecure because it transmits the credentials in plaintext (especially when using unencrypted connections). I will use it anyway 🙃. There is also the AUTHENTICATE command, which uses SASL (Simple Authentication and Security Layer). The most basic way to authenticate using SASL involves encoding the credentials in base64 but can go as far as sending hashes or using token-based authentication.
Now if I try to log in, the following happens:
a login michael@myemail.com MySecretPassword
a BAD [CLIENTBUG] Login is disabled
On most other servers, this will actually just work. But on this specific server, unencrypted connections like this are disabled.
So instead, let’s use OpenSSL to build up a secure connection and log in. Notice that we now have to use the 993 port instead of 143:
❯ openssl s_client -connect imap.myemail.com:993 -crlf -quiet
Connecting to 123.123.123.123
depth=2 C=US, O=DigiCert Inc, OU=www.digicert.com, CN=DigiCert Global Root G2
verify return:1
depth=1 C=US, O=DigiCert Inc, OU=www.digicert.com, CN=RapidSSL TLS RSA CA G1
verify return:1
depth=0 CN=imap.myemail.com
verify return:1
* OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE LITERAL+ AUTH=PLAIN AUTH=LOGIN AUTH=DIGEST-MD5 AUTH=CRAM-MD5] Dovecot ready.
a login michael@myemail.com MySecretPassword
a OK [CAPABILITY IMAP4rev1 SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY PREVIEW=FUZZY PREVIEW STATUS=SIZE SAVEDATE LITERAL+ NOTIFY SPECIAL-USE QUOTA] Logged in
Finally, we get an ‘OK‘ and a new list of the now unlocked capabilities (not every server sends us the new capabilities automatically; sometimes you have to issue the CAPABILITY command manually). This should also increase the time we have until we get thrown out of the session for inactivity.
We can now list all our folders:
a LIST "" "*"
* LIST (\HasNoChildren \UnMarked) "/" CustomFolder
* LIST (\HasNoChildren \UnMarked \Archive) "/" Archives
* LIST (\HasChildren \Marked \Trash) "/" Trash
* LIST (\HasNoChildren \UnMarked) "/" Trash/OldStuff
* LIST (\HasNoChildren \UnMarked \Drafts) "/" Drafts
* LIST (\HasNoChildren \UnMarked \Sent) "/" Sent
* LIST (\HasNoChildren \UnMarked \Junk) "/" Spam
* LIST (\HasNoChildren) "/" INBOX
a OK List completed (0.018 + 0.000 + 0.017 secs).
Even though the main inbox folder is named INBOX by convention, this helps us to identify the structure of our mailbox and where to look for new mail! We can also see folders marked with flags like \Sent which IMAP uses for special functionality. Next, we need to select one of those folders:
a SELECT INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $Forwarded Junk)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft NonJunk $Forwarded Junk \*)] Flags permitted.
* 4183 EXISTS
* 0 RECENT
* OK [UIDVALIDITY <secret>] UIDs valid
* OK [UIDNEXT <secret>] Predicted next UID
* OK [HIGHESTMODSEQ <secret>] Highest
a OK [READ-WRITE] Select completed (0.007 + 0.000 + 0.006 secs).
And this tells us that I have 4183 emails in the inbox!
Now that we have selected a folder, we can search in it! If our capabilities contain SEARCH=FUZZY we can add the FUZZY keyword to enable a fuzzy search (otherwise just leave it out):
a SEARCH FUZZY SUBJECT "buy"
* SEARCH 1082 1236 1830 3141 3263 4149 7401 10113 10118 10240 18106 18107 18113 18139 18140 18142 18270 18966 19226 20675 20684 20695 20709 26406
a OK SEARCH completed (12.310 s)
Amongst others, you can use BODY to search in the body, SUBJECT to search in the subject or TEXT to search in both. I can then use the FETCH command to display one of those IDs (or any others):
a FETCH 26046 (BODY[])
* 26046 FETCH (BODY[] {130059}
X-Envelope-From: <****@email.epicgames.com>
X-Envelope-To: <michael@myemail.com>
X-Delivery-Time: 1725393397
X-UID: <secret>
Return-Path: ....
It starts with the headers and will eventually show the contents further down. This format is called MIME (Multipurpose Internet Mail Extensions) as defined in RFC 2045-2049.
Now we can search and read emails via the command line without the help of any email client. Of course, we can do a lot more with IMAP but this will get you started. Remember for sending emails we need to use SMTP instead - I might cover this in future as well!
