Bubblegum, duct tape, and digital handshakes: how email has evolved over the last 40 years to support modern communication copy

Andrew: 00:00:04 We took really simple technologies, turned those into protocols and managed to build a system that everybody relies on. At the core, the decisions that were made back then, we still live with a lot of them. The elegance of that, it's just really interesting to me. That it's been able to survive all of these different things that have happened in the 80s, 90s, 2000s and what we're going through right now.

Marek: 00:00:29 Hello and welcome to talking email with Postmark. I'm your host Marek Loder. In today's episode, Andrew Theken, the software architect for Postmark and Chris Nagele, the CTO of Wildbit, Postmark's parent company will discuss how SMTP, the protocol that powers email came into being, and explore how the protocol has evolved to keep up with the demands of modern users. In sharing this history, we hope that you'll come away with a new found appreciation the next time that you open your inbox. Hope you enjoy the episode.

Marek: 00:01:03 Thanks for joining me today guys. Let's start with some quick intros. Chris as founder and CTO of Wildbit and Postmark. You've been in the business of sending email for I guess almost 10 years. How did that come to be?

Chris: 00:01:17 Actually, I was just looking and it's been closer to 16 years, which is really crazy and kind of scary as well. Our, or my, email experience comes from Newsberry, which is one of our first products that we launched at Wildbit. We were still doing consulting at the time and we had a bunch of clients who wanted to send out email newsletters but didn't really have a tool to do that. They didn't have a way to do it. We came up with the idea of let's throw together a product so our customers could do this. Newsberry came out of that. It was more of an email marketing tool like Mail Chimp or Campaign Monitor.

Chris: 00:01:58 In the process we learned a ton. Everything from being listed on Spamhaus and dealing with getting off of blacklists too. We are actually one of the first providers to implement SPF and domain keys, which are really important standards these days to fight against spam, and email spoofing. Since then, Newsberry is no longer a product. We shut it down in 2012. It's really what gave us the knowledge and the confidence to build Postmark back in 2009 in 2010. What's interesting is even today, Postmark is still in .net which Newsberry was originally implemented. We even took some of the codebase and some of the tools that we built in Newsberry to build Postmark.

Marek: 00:02:51 Just to be clear, you folks literally on a whim said, “Hey, let's design an email marketing solution.” Did you have any real experience with that or is Newsberry just, that's where it began where you just said, “Hey, let's figure this out.”

Chris: 00:03:03 I think when you're doing client services, the thing everybody wants to do is build a product because it's repeatable income. You're not always chasing after the next client end. We had ended up building an internal tool for some of our customers. We had something that was already there and already working and we are able to use that as a base to build a subscriber or a SaaS product, before SaaS was a thing.

Marek: 00:03:30 Gotcha. Andrew, tell us about your path to software development and how you landed at Postmark.

Andrew: 00:03:39 I spent a lot of time. Maybe a wasted youth playing with computers and growing up on BBs’ and AOL and everything. When I got into college, I worked. I was working as sort of like a sysadmin at the university I went to. That paid quite a bit better than other jobs at that time. That was the thing I was doing while I was going to school. When I graduated, I ended up working at a company and I split responsibilities between a few different things, but a major part of it was IT administration. We managed exchange servers, windows networks and NAS storage units and all sorts of stuff like that. I ended up almost on a whim building out a couple of small software projects to support the small business.

Andrew: 00:04:34 That was ultimately way more enjoyable. I think it's interesting what Chris said about what you chase when you're a small business or what part of the equation you're on in terms of running after client work or building something that creates revenue. I think it's similar in building is that software you sort of like on the more creative side and a little bit more control over your destiny. That was more enjoyable to me. Especially compared to managing exchange servers, which most pubic managed exchange. I pray for you.

Chris: 00:05:14 Some horrible stories of how we used to send email at Newsberry, but I'm not going to get into that.

Andrew: 00:05:23 Maybe we should get into it. At any rate, that progressed and ultimately I shifted away from doing more of the IT administrative work and more into pure software development. I worked at a couple of different places. I did some open source work and ultimately that was how I got introduced to Wildbit and Postmark. When someone on the Postmark team had used open-source work that I had done and created an introduction. That was about five years ago and it's been really nice to be part of this team working on this product.

Marek: 00:06:01 Chris, this guy came knocking and you opened the door for them, huh?

Chris: 00:06:07 Over the years, we've used various databases to store all of the email that we send so you can search it in your activity. One of the systems we used was MongoDB and we had a lot of trouble with it. One of the things that we had used consistently and that worked really well was this open-source library to interact with MongoDB and that's what Andrew had written.

Marek: 00:06:30 Well, I'd like to thank both of you guys for being on here with me today. Before we jump in, I would like to give our listeners a bit of context in preparation for this episode. I had actually asked Andrew to provide a list of topics that he'd be most interested to talk about on this podcast. He quickly landed on this topic of telling the history of SMTP, how it came into being and how it's evolved over time and what the future might look like. Andrew, some folks might see this as being a somewhat dry topic, but not you. What about the history of SMTP and email in general tends to get you excited?

Andrew: 00:07:12 I'm glad that you at least raised the elephant in the room. I guess it could potentially be a little bit dry. I've been working, like I said, on Postmark for about five years and did a lot of IT management and other stuff, prior to my software development career I g uess. Something that's always been super interesting to me is how we took really simple technologies, ASCII text, turn those into protocols and managed to build a system that everybody relies on.

Andrew: 00:07:44 Ultimately it's evolved over 40 years plus. At the core, the decisions that were made back then, we still live with a lot of them. Some of them were really good. Some of them were continuing to expand email to try and fix or to try and address the new types of problems that have come up. To me there's something about a system that is Simple Mail Transfer protocol. For example, the elegance of that is just really interesting. It's been able to survive all of these different things that have happened in the 80s, 90s, 2000s and what we're going through right now.

Marek: 00:08:24 Andrew, do you want to just give us a quick overview of how SMTP came to be in the first place?

Andrew: 00:08:31 I did not graduate from college in 1982, so I wasn't there. I'll try to at least give a little bit of the story as I understand it. Which is that essentially, we're all familiar with ARPANET, and a lot of the academic research that gets talked about in the lore of the internet. There was this idea of email floating around that people had started to build out. They were passing messages around, between the servers. Probably late 70s, early 80s and two people wrote, what they called RFCs.

Andrew: 00:09:08 These are requests for comment documents, which essentially outline the specs that the internet uses to work. This is how different server is written by different people can communicate. HTP servers, mail servers, that kind of thing. The joke is sort of that David Crocker and Jonathan Postel sort of wrote down what they did to make mail work and that became the standard by which everything else had been built up after it.

Andrew: 00:09:39 There wasn't really any idea about security built-in. There wasn't any idea of including images or that sort of thing. It was really basic. Like, “Let's get this out here so that people can start building mail servers themselves and we can make this available to a broader audience.”

Marek: 00:09:57 I think for those who may be listening who aren't necessarily familiar with this whole kind of process with respect to building a document request for comment, can you just give me a quick overview of what is that process? Is this basically where any developer can throw something out there and say, “Hey, this is something that I've built, this is something I'm working on that I'd like to get peer reviewed.” Is that really what's going on there?

Andrew: 00:10:22 It is in a general sense. Basically, RFCs are managed by an organization called the IETF, Internet Engineering Task Force. That's sort of a bunch of people working together to try and come up with solutions to a lot of the major problems to build the internet. It's coming up with how we're going to roll out IPv6. Which is sort of the next version of the internet that may or may not ever come if anybody's familiar with it. Also, how we're going to deal with new versions of HTTP, which had been ... We've had HTTP, HTTP2, and those sorts of things. That organization will essentially assign you an RFC number that later can be converted into a document that describes a proposal and a specification that can be used to build out services.

Andrew: 00:11:19 To your point, Marek it is a peer reviewed process. There's actually a lot of work that happens in committees at IETF and with other internet organizations to make all that work. Ultimately the goal is to have a document that's pretty close to the standard. Of course, mistakes are made and you end up with many revisions later on. New RFCs that sort of add things or fix problems that weren't obvious. There's erratic documents and those sorts of things. That's the basic premise of how a lot of this stuff comes to build the internet.

Marek: 00:11:56 Like any paper or speech, as a first draft there are obviously things that you leave out or things that need to be revised. I imagine with RFC 821 or 822, there were plenty of limitations. Do you want to speak to some of those Andrew?

Andrew: 00:12:13 Yeah, sure. It's interesting because my wife's in academia. A lot of that work is based on how many times your publications get referenced. RFC 821, 822 are referenced by everything all the time. They are just kind of like if you want to go to the source of truth, that's where you start. But, they were written at a time, like I said, they were sort of written to describe something that had already been built rather than from whole cloth, designing something and then seeing if you could build it. Which has pluses and minuses. I think one of the minuses probably historically is that it doesn't address things like security. We're on an internet where everybody trusts everybody else. You know everybody else on this network at this point.

Andrew: 00:13:08 Especially once you get into the internet being available to a lot of people, that becomes a massive problem around spam that in phishing and all of the things that we're dealing with. Just right off the start, that was sort of what I would say, is probably the most glaring omission or something that we've been dealing with ever since. The other thing, at that time it was built around ASCII texts. This is like a basic Latin alphabet.

Andrew: 00:13:36 It's very hard to describe texts and other languages in SMTP as it says. We had to build things onto it to support internationalization and allow non-English or non-Latin based languages to kind of communicate with this protocol. Ultimately computers became more powerful and the wider audience wanting to add more stuff like cat GIF's.

Andrew: 00:14:03 The protocol had to be able to support including those things in your emails so that somebody could get a laugh, if you sent it to them. Or including calendar parts so that you could schedule a really important business meeting with your boss every week or whatever it is. Those are a couple of things that the original RFCs left room for, but there were a lot of revisions later that built on top of that initial technology and made it work for those cases.

Marek: 00:14:33 There’s certainly a lot there. I want to explore that Andrew. We're definitely going to dive into the security piece in a moment. One thing I'd like to understand is this notion of building onto the original spec. Do you want to just give me a quick walkthrough of what that process would look like?

Andrew: 00:14:49 Well, I think one of the nice things about RFCs and the work that a lot of these organizations do is that if you're willing to commit the time, you can be part of it. There's nothing that says, because you don't have this specific type of degree or this specific background that you're allowed to write about a problem that needs to be solved on the internet. The RFCs can come from a lot of different places.

Andrew: 00:15:17 One of the main places that they come from and something that we're involved in is that they're different working groups out there for different problems that the internet is facing essentially. People that want to deal with sort of the web standards and how those things get rendered or deal with the abuse problems with email and how to mitigate those problems.

Andrew: 00:15:40 A lot of these things will come up in those working groups and turn into an effort maybe between one or two people that kind of look at that problem and see if there's a solution to it or propose some solutions. Those will get discussed sort of in smaller groups and then they'll start to grow until they become something that maybe the ETF or one of these other organizations is going to say, "All right, we have something to say about this problem." There are a lot of different paths that those things can become standards or RFCs rather. It's really a lot of it just has to do with, do you see a problem and do you think you have a way to contribute and sort of decide to be involved.

Marek: 00:16:29 Just to add even a bit more context in or something like multimedia computing. That's something that would allow people to start to include things like those cat GIF's that you'd mentioned. Is that a working group that comes together and puts that together? How does that come to be?

Andrew: 00:16:48 Well, first Marek, I don't know what a GIF is. I know what a GIF is, but ...

Marek: 00:16:55 We're not going to go down that rabbit hole on this one.

Andrew: 00:16:57 There is an RFC about whether it should be GIF or JIF but I know which one is correct. I think I know your question there. It's a lot of different places that these things are coming from. Companies also have an interest in making sure that they have a compelling product for their customers. There’s people at large ISPs that you know the names of that are actively involved in writing these specs. Sometimes that's good and sometimes that may not be good. I think most of the time the intentions are good in writing the spec and it really is about helping people get more access and more out of the technology.

Chris: 00:17:48 When it comes email evolving and RFCs changing, I always thought one really interesting part was the support of emojis in email and how the support for different characters has evolved in email over time. I don't know exactly how it came to be or where the pressure came from, but I imagine something like emojis support came more from the people using email than it did from the people writing the RFCs and creating email.

Chris: 00:18:22 Again, I don't know exactly the pressures or the forces that did that, but it's always interesting to see how a protocol, like something like SMTP evolves over time and where the pressure happens and where the support comes in. I remember in Postmark emojis were something that we had in our backlog. It was like something that we wanted to get to and we had to write support for it in the product, but we kind of fought it because we had other more important things to deal with. It's just emojis in subject lines or whatever so who cares?

Chris: 00:18:57 It turned out that customers really, really wanted emojis in Postmark. It was interesting to see how our customers and our customer's recipients and our customer's customers influenced that change and made that happen. I imagine some of the RFCs changed in the same way.

Andrew: 00:19:16 Yeah, as someone who worked very closely with the emoji support in Postmark I'm happy to talk about that a little bit and it did actually go back again. These are documents that we really care about to get to this. There's one called RFC 2047, two, zero, four, seven and it outlines a process that you can include nine ASCII text in the headers of emails. If you remember it from a little bit earlier, I mentioned SMP being up here, ASCII protocol didn't have this support for like extended characters or for internationalization, those sorts of things. In fact, the character sets we use commonly today didn't even exist. RFC 2047 was put out there to give a path forward where we could include these extra characters or different characters from that base set of characters that we have in ASCII. That enabled arguably more important things like getting people's names right, putting [inaudible 00:20:27] where it's supposed to be or using an ñ instead of an n. Those sorts of things.

Andrew: 00:20:35 It worked out that that was a path that we could put a Unicode text, which is fairly ubiquitous at this point. Really where emojis came in, emojis got added to Unicode, arguably people think we're adding too many or we're not adding the right ones or whatever it is. Ultimately that basic spec that allowed us to encode that information in headers create a path where we could start to include these additional characters and really create sort of a new way for people to interact with email.

Marek: 00:21:10 It's funny sitting here listening to you talk about these new standards that start to emerge. In my head I'm imagining somebody looking at a paper and just clicking the delete button and deleting and then writing something new or adding a new paragraph. I guess one of the things that I would imagine is quite difficult when you're making changes to a standard that has actually been adopted and being used in software around the world, that these changes are quite complex. I just have to imagine that's the case. Is it extremely difficult to make those changes? As an ESP ourselves and just imagine thinking through like how we then incorporate those new changes. Is that a difficult process, Chris?

Chris: 00:21:56 I think that's a really good point. Even if the change happens with ISP, like Gmail or Yahoo or somewhere else or even in the RFC, it still has to be implemented in the software. You have open source mail servers like Postfix or in Postmark in our code base, we have to actually catch up to support these things. Whether it's in open source libraries, we use open source software or the custom software that we've written. The emoji stuff was a good example of that, but so is UTF-8 support in SMTP.

Chris: 00:22:32 It's something that has been decided on and has large support from ISPs and other software vendors. Now it's going to take a long time to be implemented because there are so many pieces of software that it touches, whether it's custom software that the different products have written like us or open-source software that a lot of people use.

Chris: 00:22:56 It's just is one of those examples as the protocol and the RFCs evolves. It actually takes a long time for the adoption of it to kind of trickle down to all of the different people and services that use it.

Marek: 00:23:12 What you guys are describing really sounds like an evolutionary process. A new trait emerges and then that trait kind of proliferates through the ISPs, the ESPs. It has to effectively gain the adoption of the industry to really make an impact. I guess on the flip side of that, going back to high school biology, I imagine there may be some things that also die off. Are there pieces of the standard, are there things that ultimately lose support and eventually get written out of a spec?

Andrew: 00:23:47 Yeah, I think that there are. I wouldn't say that they necessarily get written out of the spec, but they sort of get overcome by better or more comprehensive solutions. Chris mentioned one which is SMTP, UTF-8, which is essentially like saying rather than this being an ASCII protocol, it's going to become a UTF-8 protocol. All these things that we did, how we're going to encode stuff in subject headers. Any of that work basically can start to go away because we don't have to re-encode it to fix it, to get it into that shoehorn into a narrower set of characters. Ultimately once that particular protocol is out there and adopted widely, hopefully, you can get rid of some of those things that you did to kind of evolve you into that.

Andrew: 00:24:43 In that same example for SMTP UTF-8, one of the things that it supports as far as to my knowledge, none of the previous RFCs did is actually Unicode in the email address itself. Okay. You can have Unicode on the display name or you can have Unicode in the subject line sort of thing. But the mailbox itself still has to be ASCII. Okay. There's reasons for that. There's reasons like way beyond mail. Like if I'm writing these into a folder that is on a file system that doesn't support UTF-8 file names, I can't handle that mail.

Andrew: 00:25:22 You're dealing with at least hundreds of thousands of servers out on the internet and probably millions that are going to eventually have to be upgraded. There's a lot of other work to be done that isn't supporting that particular thing. A lot of the adoption here just takes a long time. Just due to the fact that you're waiting for everybody to do upgrade the thing that they don't even want to log into.

Marek: 00:25:45 I mean it's fascinating because you think about it, it's just software, right? You can always change it. You can always make modifications and just release it or publish the latest update. At the end of the day, what you're describing, it's like the United States Postal Service saying, “Hey, we need to add new numbers to addresses. Right?” I mean all of a sudden now this is a system that's in use. It's widely adopted. To make a change like that is actually quite significant.

Andrew: 00:26:10 I think we kind of stumbled into a great analogy there, which is the same thing that happened sort of with phone numbers where you had a system in place. Then there were enough people using it that you needed to be bigger. You added area codes. In a lot of ways, if you look at how SMTP has evolved, that's exactly the same idea. That how can we paint on an additional layer that isn't going to break the existing system but allows more people access to it or better access to it. I think that's postal service or phone service are both great examples of that.

Chris: 00:26:46 Another interesting piece to that is that some standards are introduced and whoever wrote the RFCs or who is behind them really wants them to get adoption, but in some cases they don't. Or it just takes a very, very long time to get adoption and they may end up taking so long that it actually ends up getting replaced by something else. So, a good example of that, and we can get into more detail later, but as Sender ID and an SPF and there two kind of competing standards that were introduced in one ended up kind of replacing the other. How was adopted really changed what became the permanent standard.

Marek: 00:27:31 SPF and Sender ID. I mean these are basically security related feature functionalities, correct, Chris?

Chris: 00:27:37 Yes.

Marek: 00:27:38 Great. Circling back to one of the key limitations that Andrew had brought up is the gaps in security. SMTP emerged in a time where the internet was a safe space to play and there weren't really very many bad actors. Of course humanity came into the picture and a lot of kind of negative uses have started to emerge. Chris, do you want to first start with what some of those key gaps were with respect to security?

Chris: 00:28:06 Yeah. If you look at email, the amazing thing about it is that it's so open. All you need to do is set up a server on the internet with a mail server and you can receive and send and relay email. It's very open and easy to use. That's also the worst part about it. A lot of the things we hate about email spam and spoofing and all this stuff is because it's so open and easy to use. I think email in general when it comes to security and compliance and the problems around it, it's really evolved in two categories. The one I look at as the security aspects of it. The other is trust and maybe the abuse handling of email. In the security side as Andrew mentioned earlier in the beginning, email was just sent over plain text.

Chris: 00:28:59 What that means is that if someone was looking over the network as messages were being passed back and forth, you could actually inspect the messages themselves and read it in plain text. So, to combat that, first TLS or first SSL and then TLS was introduced to actually make it so that mail servers talking to each other could encrypt the communication between them. Very similar to what we expect these days on a browser instead of communicating with a website.

Chris: 00:29:32 Again, getting back to standards and adoption and all that stuff. The interesting part is even today when TLS has been out there for so long, there are still many, many servers that don't support it or don't have it enabled. SO we have to have actually like a backwards compatibility or a fallback where if an email is sent, it's called opportunistic TLS. If an email is sent to a server, we try to connect to it using TLS to encrypt the message, but if TLS is not accepted on the remote mail server we will actually default to sending it over plain text.

Chris: 00:30:12 It's that decision sometimes you have to make of, "Should we just drop the message or should we send it through?" That's one of the big introductions when it comes to security in an email that's changed over the years. The other side of it, which I think has evolved and changed even more dramatically is handling, spoofing, spam and abuse. To me, that really comes down to trust. How do you build trust in emails? When people are sending messages back and forth, they can both protect their own inbox, but also have trust that the emails that they're receiving are legitimate and who they say they're from. That's evolved in a long history and I've been able for over 16 years to watch quite a bit of it.

Andrew: 00:31:03 I think Chris, one of the interesting things that you point out there are ways that TLS was layered on to email and going back to other parts of this conversation. What we have right now is not actually the thing that we started out doing to get that encryption. We started by wrapping SMTP and putting on a different port other than Port 25 and we put it on, if I recall correctly, Port 465. In doing that, that requires an additional step of coordination between all of the servers on the internet to get that working. Ultimately, maybe it wasn't getting adopted for that reason.

Andrew: 00:31:42 Instead another RFC allowed SMTP to be extended and that gave us a way to do it on Port 25, which is what all the servers know how to talk to and to opt-in it in a sort of a more proactive way, in a more compatible way. It's just another example where you're dealing with this huge infrastructure and you've got to find a way to get some of the stuff that you wish you had but not break everything else.

Chris: 00:32:08 That's a great point because you mentioned, I think there's Port 465, was it? And then there's Port 587 and there's actually two sides of it. Right now we're talking a lot about mail going between two servers, but there's also protocols like POP and IMAP, which it originally POP to actually retrieve your email from a server to download your inbox that was not over secure connection as well. So security has evolved a lot in SMTP over the years.

Marek: 00:32:42 It sounds like there are really two sides of the fence here with respect to security. There's the trust amongst people who are doing the sending and receiving and obviously then on the flip side of that, there's the actual security of the email in-transit and the encryption of the email as it goes from one destination to another. Chris, I don't know which of those two paths you want to go down, but I'd certainly love to explore both of those further.

Chris: 00:33:10 Yeah, I think to me the interesting one has been the trust abuse side of things because when we launched Newsberry back then, the main method people had to handle abuse, like spam coming into inboxes was through blacklists. Since then it's evolved quite dramatically. Back then you had blacklists providers like SORBS and Spamhaus and a whole bunch of others that people could subscribe to. IP addresses would get listed on these blacklists for people who were sending spam or bad email. Then ISP, AOL and Yahoo would also subscribe to these blacklists or even create their own blacklists for whoever was sending bad email. It was all IP address based. The issue with that was IPs are something that you can switch easily and it doesn't really have an identity.

Chris: 00:34:10 IP addresses belong to an ISP or a larger organization, you might have a thousands of IP addresses. Like a spammer or even someone's sending email, a company might have an IP address but the next day they could change providers and have a different one. It became this problem where it was like whack-a-mole where they're always trying to find a new IP address to identify spam or even good email.

Chris: 00:34:40 The other problem with it is that the Sender, like us as a company, we didn't have any control over who could send email on our behalf. We couldn't control how we wanted an email to be sent from our own domains. As all this stuff was evolving, a new standard came out and this is what I was talking about earlier, they were conflicting standards. One was called Sender ID and the other was called SPF or Sender Policy Framework. The idea behind this was that instead of having a list of bad IP addresses that exist on the internet or maybe in addition to, I could say as Wildbit or as Postmark or an organization like Walmart could say, "Here are the IP addresses that we send email from, only receive email from these IP addresses."

Chris: 00:35:34 Then what it did is, it actually switched the control from trying to stop it at the receiving end to allowing the sender to have control who can send email on their behalf. That was the beginning of moving the control and the trust into the Sender's hands.

Andrew: 00:35:55 I do think one kind of interesting thing about the differences between ... when we talk about security areas is that one is to address privacy and that's more the TLS part of things. The other is to address what I would call authenticity. Anybody can connect to any server using TLS and have a private connection, but that doesn't say anything interesting about whether or not the person trying to give you the mail is allowed to do it.

Andrew: 00:36:23 I think a lot of the standards that we've seen and really where a lot of the interesting and difficult work is in trying to prove that authenticity and that authentication, rather authorization outside of any predetermined agreement about it where different servers who don't know each other can say, "all right, with some level of confidence, I think that you're allowed to give me this mail and it's from who you say it's from." That is one of my very interesting part of this problem for me.

Chris: 00:36:55 Andrew, I think that's a really good distinction. Getting back to something like Sender ID and SPF is where we get into the authenticity of a message being sent. The way Marek you asked about the difference between Sender ID and SPF and how that evolved. Sender ID and SPF are based on the domain. So if I'm sending from wildbit.com, I can say, "Here are the IP addresses where I send email from," and Sender ID thought that the domain in question should be the from address that the email comes from. Then SPF had a slight distinction on what we think it should be the mail from address. If you know anything about email headers, there's a mail from an a from and the mail from is more like the bounce address then the from address is the one that shows up an email client.

Chris: 00:37:56 I'm not sure exactly how it evolved and how one ended up becoming implemented over another, but what ended up happening is SPF is actually based on the mail from header, which is what people primarily use for bounce addresses. It kind of works out because the mail from is most likely in most cases the same as the from address unless you're using something like Postmark or an email service service provider which is where it can get kind of tricky. That's one of the evolutions of a standard and how it changed. Again, I'm not sure exactly how one became more prevalent than the other.

Andrew: 00:38:38 Well, just the one interesting thing about that is I believe Sender ID was mostly ... I think it was mostly pushed by Outlook and Microsoft. I don't know if that's right, Chris.

Chris: 00:38:49 Yeah.

Andrew: 00:38:50 It's interesting that that's not the one that won even though Microsoft's an 800 pound gorilla, that ultimately there were reasons to pick SPF and that's pretty much what's adopted on the internet now. It is kind of back to what we've been talking about for a while. It does kind of show that there is... trust the process, there is some level of pushback even if it is Microsoft or Google or whomever that is trying to push a standard the internet is going to try and do the right thing or what they think is the right thing.

Chris: 00:39:26 I have a follow up on that because I think the interesting part, and let me go into a few of the other standards that were introduced and I'll come back to this. After SPF, which gave domain owner's control, Yahoo came out with something called domain keys and domain keys were trying to solve a problem where if a message is in transit between two mail servers or several mail servers, it's possible that that message could actually be altered in transit and then sent along to the recipient looking like nothing happened but it's a completely different message.

Chris: 00:40:06 Domain keys came out to actually solve that problem. You could believe that the message that someone sent is actually the same content that you received and it uses public and private keys, which are registered in your DNS for your domain. Again, like Sender ID in SPF, it turned into a new spec called DKIM. Which is kind of similar, has some subsidies, but now these days SPF and DKIM are really the main standards we use for building trust and authenticity around messages and email.

Chris: 00:40:43 What ended up happening is SPF is ... because we need backward compatibility, there were some limitations in some people accepting it and some people not. Then DKIM uses a domain, but you don't necessarily have to say which domain, you can sign a message with any domain when it goes into your messages. So a new standard came out called DMARC, which actually ties everything, this is why it's comes back to Sender ID, everything back to the from address.

Chris: 00:41:13 DMARC basically says if I'm walmart.com and I'm sending an email from marketing@walmart.com and you're using DMARC, I can say only accept messages from me that have SPF and DKIM enabled in my messages for marketing or for walmart.com in the from address. I think all of this stuff is interesting and difficult as well because as these things have evolved, we've almost put bandaids on them. SPF has these limitations, DKIM has these limitations. Let's put DMARC on this to bring them together and solve them.

Andrew: 00:41:57 Just add one more layer. That's how you do it.

Chris: 00:42:00 Now looking back, we have something like Sender ID which was based on the from address and we're trying to almost bring Sender the validity of a Sender ID back with the introduction of DMARC. Sometimes I laugh at all the stuff coming out cause there's always a new protocol trying to ... with DMARC, there's a new standard called ARC, which is trying to fix some of the limitations in DMARC. It's an evolving thing and we're always trying to learn from the small thing where it's falling apart and fix it with something else.

Marek: 00:42:36 I have a smile on my face because it really truly is like evolution. It's like short beaks are in favor now and then all of a sudden the long beak comes back in and there's a reason for it to exist in the species. Right? And it sounds like exactly that. With Sender ID that's now reemerging as something that is a favorable advantage to making sure that there's authenticity in the sending that's happening. It's fascinating.

Chris: 00:43:03 Yeah, you're definitely onto something Marek, the good people, the good senders around the world are ... they do a thing, they try and improve the landscape a little bit and then the spammers evolve a little bit and they figure out a little way that they can skim a little money or this or that by cheating DKIM or by doing this or by doing that and then you respond to it and escalate and it's ... I am not going to say it's fun, but it's interesting.

Marek: 00:43:35 Certainly we could go down any of these paths and explore any of these standards in great detail. I will certainly in the resources section of this episode include links to some literature we've put together. While I have you guys on here, one of the things I want to do is just get a sense of whether you guys see any gaps that still need to be addressed with respect to security in email. Do either of you guys want to speak to that?

Chris: 00:43:57 I can talk about that a little bit. As I said before, it kind of feels like a lot of these things are bandaids, but it's not all bad. I think DMARC is a really great direction for where we're going to build trust in the email and to combat spam and some of the other problems that revolve around email. One of the big benefits I see from it is it really comes down to domain reputation, which we didn't talk about much and it's based on DKIM and SPF and DMARC. What it does now is, instead of before where sure you have spam, you send them the really nasty emails to millions of people, but there's that gray area that also causes problems in email. The marketing companies that are sending out to purchase lists and people not being responsible for the emails they're sending from their domains.

Chris: 00:44:54 Previously, you could actually use ... you just switched to different IP addresses or different ESPs to send your email and it didn't matter. You could send a bunch of junk and then move to the next ESP or change your IP addresses. What's happening now with the adoption of DMARC on top of DKIM and SPF is domain reputation. So, at Wildbit we are actually building a reputation on our domain that follows us everywhere.

Chris: 00:45:25 And I think that's really strong because it means now we have an upkeep for that reputation no matter which ESP we use, no matter where we go. I think that's been one of the best evolutions for compliance and abuse and spam over the years because now there's a responsibility of the sender to keep that reputation going.

Marek: 00:45:49 Sounds like the reputation that sticks with you is something that can really improve overall sending across the internet.

Chris: 00:45:57 Yeah, and those are the good things. To get back to your question, what are the limitations? DMARC still has to evolve a little bit to get the adoption that it needs. There are parts of it where DMARC actually gives you control in your DNS to tell places or ISPs is like Gmail or Yahoo or anyone else to only accept messages from us that are signed with DKIM and have the IPS listed in SPF. There are some gaps in it with email forwarding where DMARC actually breaks down and it's not completely accurate. So there's some new standards coming out. One is called ARC and that's to fix some of the forwarding issues. I think stuff like that is really important before we get more adoption on DMARC to then get more adoption on domain reputation.

Marek: 00:46:57 Andrew, do you have anything to add there?

Andrew: 00:46:59 I'm happy to see the trend go more towards reputation also for a bunch of different reasons. I think ultimately what it does is it allows somebody that's going to accept mail to make informed decisions around what they trust or don't trust. They can do it more actively. Prior to that, if you're talking about blacklists or whitelists or whatever else, those get updated whenever they get updated and they're sort of like whatever the person that wrote that particular list is feeling, depending on what list it is that IP is going on there or not.

Andrew: 00:47:35 I think the move towards domain reputation and looking at what the history is just going to help over time. Google obviously builds their own reputation models around everybody and if you use Gmail, I think you've benefited from that. So I think that that's just going to spread to all the different ESPs and mailbox providers. I hope.

Marek: 00:48:00 I'm just interested to ... in terms of making sure that these things take hold. Do you think that ESPs have a responsibility? We're an ESP obviously, but do we have a responsibility to enforce some of these standards as they emerge?

Chris: 00:48:17 I think maybe not enforced, but one of the biggest issues of a lot of these standards and fighting abuse and building trust in email is really getting the adoption. When it comes to ESPs or services or even the inbox providers like Gmail, it's really important that we make it extremely easy for people to adopt these standards.

Chris: 00:48:41 The easier we can make it for people to understand and adopt these standards, the faster the adoption is going to be. And really the standard will survive and persist. We've always tried to push a lot of these standards very early. We were very early in supporting DKIM. One of the things we really wanted to do for new customers coming in is as soon as you set up your account, we take you directly to setting up your domain, your domain authentication with SPF and DKIM.

Chris: 00:49:16 And..while we don't enforce it, we make the process extremely simple so we can get more adoption in our customer base for people who use DKIM and SPF. We've actually done the same for DMARC, so when you're setting up your account in Postmark we give you the DNS records and the ways to set up DKIM, SPF and then a custom return path so that you can be DMARC compliant.

Chris: 00:49:41 As kind of a side project, I think it was a hack week project awhile ago, we realized that DMARC reporting is one of the most valuable pieces of the DMARC standard. We created a side project where not only do we support it from a sender side, but we allow you to, through our service to set up DMARC reporting and to actually give you a weekly email to see how your emails being received and what your DMARC alignment is to different ISPs.

Chris: 00:50:13 I think it's really our responsibility as email service providers to educate our customers, to maybe in a way enforce it because the more people who can sign their messages and everything else, the more receivers see that as being adopted. But yeah, that's definitely our responsibility to make it happen.

Marek: 00:50:36 It's interesting. Maybe enforced isn't the right word, but it sounds like we have gone to great lengths to try to make it super simple and straightforward for our customers to adopt these latest standards.

Chris: 00:50:48 Yeah, and that's the key. I think the important part is making sure we make it as easy as possible because the more people, even in our customer base who have authentication set up who use DMARC, it actually makes it easier on us to have good compliance and abuse standards for our own systems so we can fight off spammers and people who are abusing the system.

Marek: 00:51:11 Certainly a symbiotic relationship there.

Chris: 00:51:14 Yep.

Marek: 00:51:16 Before we wrap things up, I'd like to just spend a few minutes exploring some of the things that you guys foresee coming next with respect to email or things that are just beginning to emerge now. Perhaps, Andrew, we could start with you.

Andrew: 00:51:34 Well, I think if it isn't already a theme, it's going to be by the end of this, is we've seen a bunch of things start to evolve really recently around the idea of adding lists on subscribe headers and those sorts of things which have been nice to improve the overall experience that people have in mail clients. If you have an iPhone or whatever and you get a mail that you don't like, those headers being added now allows you a quick and easy way to unsubscribe for example.

Andrew: 00:52:03 I think it's also interesting to see things like Google AMP coming out and seeing where that might go. I know it's controversial and there's a lot of opinions about the motivation around doing it. I have seen at least a few cases where there is actually a really nice user experience that you can get out of adopting that format for email.

Marek: 00:52:30 If you don't mind me asking, Google AMP is?

Andrew: 00:52:35 I don't want to go too far into describing it because I know enough about it to be dangerous, but not so much that I'm an expert. Google AMP is basically a format that Google has published that allows you to deliver a mail with a slightly different format than HTML. One example that I think is really interesting is an account confirmation email you might get where normally you'd get the confirmation email, click the button that says activate account and then you end up on a website and you read a page there and close the tab and go through all that.

Andrew: 00:53:10 AMP provides enough machinery to allow you to do that all within your email clients so that you don't leave that experience. It’s definitely going be a while to see that really roll out and what all the implications of it are. I'm sure that there's going to be more than enough security implications to keep everybody busy. The idea that that's maybe something that email could do and the next couple of years is interesting to me.

Marek: 00:53:37 Didn't mean to derail you there. If there were some other things that you see that are likely to emerge in the future here.

Andrew: 00:53:42 One thing we talked about a little bit is that there's sort of ... DMARC is something that has been around for at least five years probably. I don't know exactly how long it's been around. It started out as something where you could monitor how things were. Whether you had mail coming from weird places that you didn't know about or those sorts of things and you didn't actually cause ISPs to stop delivery or those sorts of things. You can monitor it. See what was going on and then make adjustments. Finally, the last step is going to be to tell ISP is to reject mail that isn't authenticated.

Andrew: 00:54:20 We’re right at the cusp of that being something that's common practice. I think that that's going to be a really major improvement for the experience that people have with any reducing spam and phishing and all those problems that hopefully that standard solves.

Marek: 00:54:40 Chris, I don't want to leave you out here. Are there some things that you see?

Chris: 00:54:43 Yeah, and actually building on what Andrew just said, there is a new standard that I think Gmail just started accepting maybe in a pilot program or something, but it's called BIMI. What it does is if you follow certain standards like if you have a reject record or a quarantine record and DMARC and you add a certain DNS record and I think you may have to apply for it, but it basically ties your corporate branding or whatever branding you have to your email.

Chris: 00:55:20 What that does is actually build trust with the people who are receiving your email. I think that's just another step, something like that could actually, because it is branding and it has some value on the marketing side, a standard light that could actually push people to be DMARC, compliant more. I'm looking forward to seeing how that evolves and gets adopted.

Chris: 00:55:48 The other interesting things like I said earlier, blacklists were a really prominent thing you had to be aware of years ago and they still are. They come up now and then and you have to be aware of them, Spamhaus is still a very prevalent blacklist...and you definitely don't want to get on any of those, but as things shift it's going to go more towards domain reputation and I'm really excited to see that happen.

Chris: 00:56:20 I'm also curious if just like IP addresses, where you could actually ... there were services like Sender Score, where you can go out and look up the reputation of IP address. I'm kind of hopeful that in the future you'll be able to see the reputation of a domain for email sending. It would be really needed for that site. Almost like a transparent thing. There's a lot of issues in that too.

Chris: 00:56:46 Another side of it is we're all used to clicking the, this is spam button in our email clients and what we've seen ... over the years, we use that as a metric to fight against spam. Like what are your spam rates and how many spam complains are they receiving, but as more and more email goes to Google and Gmail who doesn't actually support sending spam reports back to receiver or senders. It'll be interesting to see how we use that metric and how that metric changes. Whether spam rates in five years are really going to be something we look at, or did it get completely replaced by something domain reputation?

Marek: 00:57:33 One last thing that I'd like to explore a little bit. I'm just thinking through that, what you just brought up, where spam complaints may not be the thing, but it sounds like maybe things like domain reputation but also perhaps things like engagement, right? Whether people are really engaging with those emails is going to become more and more a metric that ISP and receivers are going to use to decide and make decisions about whether mail should even make it to the inbox at all. I had to anticipate that more and more machine learning and AI and things like that are going to start to process all of that data that's going to start to inform those things as well.

Chris: 00:58:10 Absolutely. I think something else to think about with engagement though as privacy becomes more of a concern. Will that change as well? We'll open tracking and link tracking, five years, 10 years from now, be something that's even acceptable, right? That we use a lot of the engagement data from open tracking and link tracking. I'm curious to see how that evolves and how kind of the privacy rules change in that.

Marek: 00:58:38 Anything else you want to add, Andrew, before we wrap this up?

Andrew: 00:58:43 I don't really want to do a lot of predictions, but I think there is going to be one interesting one that I'm not sure we've solved yet. That's going to be that, we talked about privacy between servers, but we still don't really have privacy after you're on one server or the other with mail. As customers become aware of that, they're going to want to have that available to them where they know that their communication over email is completely private from their email client all the way to the other side.

Andrew: 00:59:17 There's some standards around that, but just like every year is going to be the year of desktop Linux, those standards are adopted by the people that are known to pay attention to it, but eventually, hopefully it'll become something that's more commonplace.

Marek: 00:59:38 What about holograms, 2030 holograms?

Andrew: 00:59:41 There is something in the RFC process right now for holograms and we'll talk about it, but yeah, we'll see.

Marek: 00:59:49 Well, I think it's time for us to wrap this up. Andrew and Chris, thank you so much for being with me today.

Chris: 00:59:55 Yeah, thank you.

Andrew: 00:59:56 Glad we could do it.

Marek: 00:59:58 To our listeners, thank you for joining us today for this episode of talking email with Postmark. If you enjoyed the episode, please leave us a review on iTunes and subscribe to receive updates in all future episodes and be sure to check out the resources section of this episode or you'll find interesting articles on many of the topics that we touched on today.

Marek: 01:00:16 Lastly, if you're ever looking to improve email deliverability or reporting for your business or side project, be sure to reach out to us at support@postmarkapp.com and we'd be more than happy to help. See you soon.