Load testing to find the maximum capacity and the bottle neck
When a large pharmaceutical moved to Lotus Notes for email I was tasked with building the email infrastructure and this included the Simple Mail Transfer Protocol (SMTP) interface.
To ensure the infrastructure was adequately scaled to meet the company requirements I needed to know the limits of the Domino SMTP Message Transfer Agent (MTA).
The full details of the investigations can be found at this Capacity Details link but this is a quick overview. The bottle neck on the MTA was the Domino Mail to MIME (Multipurpose Internet Mail Extensions) conversion.
No matter how many processors were on the box only one conversion would happen at a time. To improve throughput, you needed a faster processor running the conversion task.
With R5 of Domino and a quad PIII X500 box the MTA could transfer 18,000 emails in an hour. I could scale the number of SMTP MTA's for redundancy and scale for future growth.
After a few years of this service going live Microsoft put out an OS patch that reduced the capacity of the transfer to 1,000 emails an hour. It was not known to start with that this was a Microsoft issue and all sorts of suggestions about extra MTA's were muted. I referred those then looking after the email infrastructure to the attached link proving that the MTA's could process way more than 1,000 an hour and that they were looking at the wrong area for a solution. They investigated the UNIX Sendmail server and the Microsoft OS and found the OS issue.
So this work also provided a point of reference at a much later date aiding problem investigation and resolution. It is a clear example how thorough load testing and documentation lets you know the limits of your infrastructure and benefits in ways not envisaged at the time.