Skip to main content

The perils of internal spam


Let me tell you a little story.  The names have been changed to protect the not-so-innocent and it's vague because I'm not really so great with specifics.

Let's call the subject of this story Steve.  Steve sent out a message asking how he could set up his corporate email on his iPad.  Unfortunately, instead of sending that message out to the proper person or mailing list in his company, he figured "bigger is better" and sent it out to as many people as he could - 4000 users seemed to be the limit, so Steve was sure to use up all the resources he could.


In 1989, it would have been a lot of paper.

This little action seems to be pretty innocuous at first, but it has huge ripple effects that still affected message delivery hours later.

The problem starts out simple.  Let's do a thought experiment so we can see the repercusions of this one message.

Let's assume that the mail server can do only one thing at a time, and that copying this first message to each user's mailbox and transmitting the message to the mail client takes about 500ms on average (it's got a lot of addresses, so there's a lot of text and transmitting it takes some time).  So for this first message, we're looking at roughly 30 minutes of the server's time to process and fully deliver the message.  Even if the server is provisioned to just handle normal traffic, this small spike will normalize reasonably quickly.


It's more like an "Oh, darn!" sort of problem at this point.

Unfortunately, this is not where the story ends.  Somebody had to reply-all to the message and repeat the same question.  Again, this isn't so bad - except that when you reply to a message, the header (the from / to / subject fields) is copied into the message.  This relatively small message is now about 250KB, due to all of the addresses that were in the original.  So assuming about double the time to deliver a message twice the first message's size, this second message should take about an hour to transmit fully.

Of course nobody is content to be spammed like this and let it go, so somebody has to reply-all and ask to be removed from the conversation.  Now the message is about 750KB.  We're looking at 90 minutes at this point - to deliver one single message.  Keep in mind that each one of these is going to have a cumulative impact, so we're REALLY slowing down the server at this point.

Now we have to have the series of people complaining about people replying to all (quite ironically) by replying to all.


This picture is irony.

What makes it even worse is that a decent number of those addresses are group mailing lists - so there's a very good chance that we are looking at closer to 10,000 recipients instead of 4000.

All in all, at least a dozen separate messages were sent, the biggest of which was about 7 replies deep.  It's about 1.7MB in total size.  Sent to 4000 addresses.  Probably 10,000 recipients.  Even if every one of those recipients is on a 1Mbps connection, it's going to take about 15 total seconds per recipient just to transfer it - not to mention copying it to each mailbox.  This will ultimately end up costing about 17GB of space on the server - and that's just the ONE message.  It's not crazy to think that the rapidly filling disk on this server is only going to exacerbate the problem.


It's like crashing a train by screaming out questions about your seat belt.

At 4:30 PM people were still getting their other messages (you know, the important ones that don't have anything to do with how to set up an iPad) from 10:30 AM.

I really hope Steve didn't figure out how to get his email set up on his iPad - with two ways to send messages he might black out the entire continent.

Comments

Popular posts from this blog

Sheetz Sandwich Standoff: El Gringo vs Twisted Swiss

My wife left me alone for dinner tonight so I decided to check out the latest GetGo offerings... but to my great chagrin, they have no promotional subs. My travels led me to the local Sheetz, where I'd be able to keep eating the best gas station sandwiches around. To keep tradition alive, I picked the two most outrageous "Burgerz" on the menu: El Gringo and Twisted Swiss. The ingredient list is promising: Twisted Swiss is the burger with topped with swiss cheese, cole slaw, pickles, bacon, and whatever "Boom Boom Sauce" is on a pretzel bun.  El Gringo is the burger topped with pepper jack cheese, chili, Doritos, and BBQ sauce on a regular old bun. I unwrapped them both and stood back to admire the majesty before me. They're not pretty, but they do look a lot better out of the wrapper than many fast food burgers I've eaten. Twisted Swiss I expected this sandwich to be an awful mess.  It just seemed like a bunch

The Gobbler from Arby's

Stop.  Stop what you're doing and go to Arby's. Right. Now.  Have them make you a Gobbler .  This is not something you'll regret. Go. Eat this thing. Look at that bacon. Go. Arby's has a new sandwich.  It's called "The Gobbler" and as far as I can tell it's two things: a vehicle for their new deep fried turkey, and an attempt at a Thanksgiving themed sandwich.  It's also a third thing: magically delicious. move over Lucky, there's a new holiday mascot on the block Unwrapping: this actually looks like a sandwich.  It looks appetizing.  It looks like something I want to eat.  It doesn't look like the promo photo above, but it doesn't look like someone was flailing around and accidentally smashed up a sandwich, either. sexy Instagram caption goes here First bite: Wow.  I mean, "WOW."  Holy h*ck this is good.  The turkey has a really bold, meaty flavor.  It tastes a lot like turkey sliced fresh from your

A Story of Student Loan Debt

I was paying bills today and it struck me how much easier things are since my wife and I paid off all of our student loans 2 years ago.  I’m simultaneously happy with a system that allows a broke kid like me to go to a decent college, and shocked by the crushing burden of debt that it can create.  The story of my own college debt is convoluted, uninformed, and fraught with my own mistakes and missteps. I grew up in Ohio always thinking I’d be going to OSU. I ended up doing in better on my SAT than anybody expected (myself included), and the University of Dayton offered me a big scholarship. Nevermind that OSU would still have been significantly cheaper. I decided to chase the dollars. It felt great to be offered so much money, though I didn't quite grasp the idea that it was more of a discount than a cash offer. It went fine; I sometimes felt unprepared, but I did OK overall and finished the year with a decent GPA. I had a girlfriend when I started college. The long distance was t