Introduction
The story originated from my volunteer work on Saturdays. I help three local Chinese language schools to run their websites. The schools operate over the weekend only and one of them has more than 1000 students. Typically, we send e-mail notifications to all enrolled families to announce various events of the coming week.
Due to reasons that do not need to be mentioned here, I have to find new web hosting for the biggest school of the three. To simplify my future work, I proposed to move websites of all three schools onto a dedicated virtual server. The server is running Windows 2008 Web Edition. Although I have been developing web applications professionally for a long time, this is the first time I need to manage a server all by myself. It is quite a learning experience.
The web applications that need to run on the server are online registration systems, each school has its own variation branched out from the same set of ASP.NET source files. People can use the system to sign up for classes on the web which saves a lot of time and resources for the schools. The system also handles teacher's timesheet, payment records, and academic data (test and homework scores, etc.). Guess who wrote the system? 
The first problem I encountered was, I could not install SQL Server 2000 on Windows 2008 Web Edition, not even the developer's edition. This was a big problem because I didn't think the school would be willing to spend extra money on a different version of the operating system (yes, we have got SSBS, Shoe String Budget Syndrome) and it would erase all the advantages of my proposal if we had to host the databases on a separate server shared with other businesses or individuals. Fortunately, there was SQL Server 2005 Express Edition. SQL Server 2005 Express and related software were installed and all three databases were uploaded successfully and ran without a problem after some hard work.
However, it was far from over. Another problem was waiting for me ...
The E-mail Problem
E-mail function is vital to us. We depend on it for routine communication among school management, teachers, parents, and students. We also need to send emergency school closing or class rescheduling announcements as well as "nasty" payment requests to people who drop their kids into classroom without paying tuition. Our parents rely on it for things like account creation, password reset, and communication with teachers, etc.
The new web hosting company has an e-mail limitation published on their website and it is enforced by a filter program on the SMTP server. The limitation is, at least that's what I thought in the beginning, each IP address can send at most 10 e-mails per minute and each e-mail can have at most 10 recipients. When I first learned the limitation, my reaction was "that's great", because it means the SMTP server we are going to use is less likely to be blocked by major companies for spamming. A big problem with the previous hosting company is that there is no limitation on e-mail so some user (not us) probably sent a lot of spam, damaging the reputation of the SMTP server shared with many other users. As a result, some of our e-mails never reached intended recipients.
As web application developers, how many times in your life time will you be required to make your application run slower? In this case, we have to prevent our application from sending too many e-mails too fast. Piece of cake, I thought.
Originally, all three applications used EMailService.dll (source code included in the download) to send SMTP e-mail. SendEMail is the method that has been used so far. There is also a SendMassEMail method, which takes a string array of recipient e-mail addresses (among other e-mail parameters) and sends e-mail to at most n recipients at a time, where n is a configurable value which can be modified in web.config file. I quickly did the following:
- Modified
SendMassEMail in EMailService.dll to save e-mail requests in an internal queue
- Added a new thread to process the queue
- When processing the queue, make sure each e-mail sent had at most 10 recipients and make sure the processing thread sleeps for at least 60 seconds after sending 10 e-mails
- Changed application code to call this new
SendMassEMail method
That should do it. But it failed miserably. When testing with sending more than 10 e-mails (each with up to 10 recipients), the first one usually worked fine, and the rest would fail. I was getting "too many recipients" errors with all the failed requests from the SMTP server. I tried to change configurable values, such as lowering the limit of recipients per e-mail to 7, no luck. I also tried to increase the time gap between two groups of (10) e-mails, no luck either. The only test that worked every time was sending a single e-mail to a single recipient. While debugging this issue, I accidentally clicked a button sending test e-mail to all users in our database (there is no such thing as test environment for organizations with SSBS you know). This time I was glad the e-mail function did not work, only the first 10 lucky users got the test e-mail.
I was convinced there was a bug on the SMTP server side in calculating the number of e-mails and recipients. I got hold of the technical person in the hosting company, he said: "There is no problem on our side, our system has been working for many months without a glitch. Please go check your application code for possible bugs." He sounded just like me.
After more pain and suffering, I finally figured out the cause of the problem. It was due to misunderstanding or misinterpretation of the e-mail limitation. When the hosting company said "at most 10 e-mails per minute", they were counting each recipient as a separate e-mail. So if you are sending an e-mail with 10 recipients, it will be counted as 10 e-mails, therefore a single e-mail will reach the upper limit, all the following e-mails will fail! In the beginning, the sales agent from the company told me that I can send as many as 14400 e-mails per day ( 10 emails x 60 minutes x 24 hours ) which sounds a lot, she failed to mention that to in order to reach that maximum number, there can be at most one recipient per e-mail and the e-mail has to go out at an absolute constant speed throughout the 24 hour period. Haha.
On the other hand, the misinterpreted e-mail limitation was not something that would stop me, no way, especially after I had spent so much time and energy figuring it out.
The Solution is, Web Service of Course
It will not work if I simply modify EMailSerice.dll to make sure at most 10 recipients per minute are being sent e-mails. This is because all three applications are using EMailService.dll. If more than one application happens to send e-mail within the same minute, the 10 recipients per minute limit will likely to be broken, causing errors from SMTP server.
My solution is isolating e-mail function to a web service, which reuses EMailService.dll to send e-mail. All three and future applications will invoke this web service.
The web service is called MassEMailQueue. It has a SendMassEMail method. Here is the signature of the method:
public void SendMassEMail ( string sFrom, string[] pToList, string sSubject,
string sBody, string sAttachments, bool bHTML );
sFrom is sender e-mail address, pToList is a string array of e-mail recipients, sSubject is e-mail subject, sBody is e-mail text, sAttachments is semi-colon delimited string of attachment file paths. The bHTML flag controls e-mail format, if it is true then sBody will be treated as HTML instead of plain text.
Applications call SendMassEMail to submit request to the web service. An e-mail request received by MassEMailQueue will be broken into simple e-mail requests (with a single recipient each) and placed on an internal queue. A background thread will process items on the queue at an acceptably slow pace: at most 10 e-mails/recipients will be handled per minute. This is not the most efficient way to handle requests, but remember what we want to avoid is sending too many e-mails too fast.
Queue (in-memory) vs. Database. Imagine if we have to send a huge e-mail (over 1 MB in size) to over 1000 recipients, then the web service will not work well. In this case, it is better to change the design and to use a database in place of the in-memory queue. However, this is not going to happen for our schools, our e-mails are small in size and we send to at most 3000 recipients per day.
If we have more applications in the future, we can share the same instance of MassEMailQueue. Sometimes, we need to run our ASP.NET applications on laptops with mobile internet connection. Our solution will still work with these extra laptops invoking the same instance of web service running on the dedicated server.
What happens if the server is shutdown or the application pool is recycled? Do we lose all e-mail requests in the queue? MassEMailQueue will automatically save all unprocessed requests on the queue to an XML file when it is being stopped. And the saved requests will be reloaded into the in-memory queue when it is started again. So no e-mail request will be lost. I also set up a scheduled task on the server to ping MassEMailQueue to prevent the application pool from being recycled due to inactivity.
Configurations
Here are some configurations you may need to do in the appSettings section of the web.config file:
SMTPServer: The default is localhost. SMTPPort: The default is 25. EnableNetworkEMail: The default is false. E-mails will be placed in IIS pickup folder by default. If this setting is changed to true, then e-mails will be sent to the SMTP server specified in the previous setting. MassEMailBatchSize: The default is 10. This is the maximal number of recipients per batch of requests in the background processing thread. MassEMailPause: The default is 61. This is the number of seconds the background processing thread will sleep between two batches of requests. MassEMailCopySender: The default is true. If this flag is true, then the sFrom address of the SendMassEMail method will receive a copy of the e-mail being sent, unless the e-mail is sent to a single recipient. Retry: The default is true. If this flag is true, then any failed request (network error, etc.) will be put in the back of the queue to be resent later. RetryMax: The default is 3. This is the maximal number of times to retry for a failed request.
History
- 06/15/2010: Initial version posted
- 06/25/2010: Updated source code (use
SmtpClient instead of SmtpMail) and updated article text
- 06/28/2010: Minor update to article text
- 07/05/2010: Fixed a bug in loading requests from XML file
- 07/07/2010: Added capability to automatically retry failed e-mail requests
