The perils of internal spam

Let me tell you a little story.  The names have been changed to protect the not-so-innocent and it's vague because I'm not really so great with specifics.

Let's call the subject of this story Steve.  Steve sent out a message asking how he could set up his corporate email on his iPad.  Unfortunately, instead of sending that message out to the proper person or mailing list in his company, he figured "bigger is better" and sent it out to as many people as he could - 4000 users seemed to be the limit, so Steve was sure to use up all the resources he could.

In 1989, it would have been a lot of paper.

This little action seems to be pretty innocuous at first, but it has huge ripple effects that still affected message delivery hours later.

The problem starts out simple.  Let's do a thought experiment so we can see the repercusions of this one message.

Let's assume that the mail server can do only one thing at a time, and that copying this first message to each user's mailbox and transmitting the message to the mail client takes about 500ms on average (it's got a lot of addresses, so there's a lot of text and transmitting it takes some time).  So for this first message, we're looking at roughly 30 minutes of the server's time to process and fully deliver the message.  Even if the server is provisioned to just handle normal traffic, this small spike will normalize reasonably quickly.

It's more like an "Oh, darn!" sort of problem at this point.

Unfortunately, this is not where the story ends.  Somebody had to reply-all to the message and repeat the same question.  Again, this isn't so bad - except that when you reply to a message, the header (the from / to / subject fields) is copied into the message.  This relatively small message is now about 250KB, due to all of the addresses that were in the original.  So assuming about double the time to deliver a message twice the first message's size, this second message should take about an hour to transmit fully.

Of course nobody is content to be spammed like this and let it go, so somebody has to reply-all and ask to be removed from the conversation.  Now the message is about 750KB.  We're looking at 90 minutes at this point - to deliver one single message.  Keep in mind that each one of these is going to have a cumulative impact, so we're REALLY slowing down the server at this point.

Now we have to have the series of people complaining about people replying to all (quite ironically) by replying to all.

This picture is irony.

What makes it even worse is that a decent number of those addresses are group mailing lists - so there's a very good chance that we are looking at closer to 10,000 recipients instead of 4000.

All in all, at least a dozen separate messages were sent, the biggest of which was about 7 replies deep.  It's about 1.7MB in total size.  Sent to 4000 addresses.  Probably 10,000 recipients.  Even if every one of those recipients is on a 1Mbps connection, it's going to take about 15 total seconds per recipient just to transfer it - not to mention copying it to each mailbox.  This will ultimately end up costing about 17GB of space on the server - and that's just the ONE message.  It's not crazy to think that the rapidly filling disk on this server is only going to exacerbate the problem.

It's like crashing a train by screaming out questions about your seat belt.

At 4:30 PM people were still getting their other messages (you know, the important ones that don't have anything to do with how to set up an iPad) from 10:30 AM.

I really hope Steve didn't figure out how to get his email set up on his iPad - with two ways to send messages he might black out the entire continent.