by Jake Kouns, 2010-08-12 06:19:12 UTCCloutage Outage - The Cloud Poked Back...
As you have read, we have been working hard to backfill past events while delivering the most current news on cloud security. Each time that we approve a news post or new incident we update the site, post it to our Twitter feed (@cloutage) and of course send it out to our mailing lists. Well, it appears that EC2 has a limit on the amount of mail sent out and if you exceed the limit, you receive this canned email:
--------------------------------------------------------------------------------------
Dear EC2 Customer,
You recently reached a limit on the volume of email you were able to send out of SMTP port 25 on your instance:
In order to maintain the quality of EC2 addresses for sending email, we enforce default limits on the amount of email that can be sent from EC2 accounts. If you wish to send larger amounts of email from EC2, you can apply to have these limits removed from your account by filling out our online request form.
If you are unaware of your instance having sent emails, we advise checking your instance application(s) to confirm that this activity was intended. It is your responsibility to ensure that your instances and all applications are secured against unauthorized use. For suggestions on securing your instances, reference this whitepaper: Tips for Securing Your EC2 Instance
Regards,
Your Amazon Web Services EC2 team
--------------------------------------------------------------------------------------
Its nice to see that it is our responsibility to secure our instances. It was also nice to see that we apparently sent out too many emails (our mailing lists are certainly growing) and they cut us off without a warning message or indication of what the limit is. Either we overlooked the information that explained this policy, which is entirely possible, or it wasn't posted. Since this is a volunteer run project we were not able to handle it as soon as this email came to us early in the morning. However, after getting home from the day job and spending some time with the family I saw the email and was able to log into our account to address the issue. I provided what I thought was a very simple but valid reason basically explaining that we have several mailing lists on the website and we need to have this limit removed if we are to continue hosting with EC2.
Once I submitted the reason in the online form we were redirected to a page that stated:
--------------------------------------------------------------------------------------
Request to Remove Email Sending Limitations Received
Thank you for submitting your request. It is our intention to meet your needs. We will review your use case and normally respond to requests within 2-3 business days.
--------------------------------------------------------------------------------------
So...... we started to get worried that anyone on our mailing lists wouldn't get our updates for several days and there was nothing we could do about it. Then we thought, hopefully people will continue to check out the website and follow us on Twitter so it wouldn't be that big of an impact. But wait, if we were hosting at EC2 and this was critical to our business could we get this resolved immediately? We started to look around the support pages and couldn't find any number to call and in fact it wasn't very clear how to escalate the issue. The only thing that looked reasonable was to fill out another form for additional inquiries... so that is what we did. Would this help speed up the process?
Just when we started to think it would be another couple days before we heard back we received an email that stated:
--------------------------------------------------------------------------------------
Hello,
We've reviewed and approved your request for the removal of the EC2 e-mail sending limitations on your Amazon Web Services account. There are no longer limitations on your account for any IPs and instances under your account. If you requested removal of e-mail sending limits on Amazon Elastic IPs, they've also been removed.
Thank you for your recent inquiry. Did I solve your problem?
--------------------------------------------------------------------------------------
So here is the bottom line: The bad news was that they shut off our ability to send outbound email without any notice or warning. Our own total "cloutage" was approximately 21 hours but the good news was that once we contacted support via the support form they were responsive. The initial scare of having to wait 2 to 3 business days really amounted to approximately 3 hours before they replied and removed the limitations on the account.
If you are thinking of using EC2 for sending emails please keep this in mind and we would suggest you proactively contact them to get an exception. You should be able to use this link: http://aws.amazon.com/contact-us/ec2-email-limit-request/
As always if you are interested in getting involved with or sponsoring the project please contact us as stewards@cloutage.org.