Why You Need to Outsource Your Test Lab

“One of the most difficult tests we had to do was to simulate 200,000 DNS requests a second, all with different source IP and MAC addresses,” said Tom Henderson the managing director of ExtremeLabs in Indianapolis. They also work in virtualization consolidation scenarios, where a company wants to migrate their physical servers to virtual ones and understand what the resulting configuration, load, and performance will be.

Platform Lab in Columbus doesn’t actually conduct the testing. It is a state-funded facility and will connect clients with the appropriate consultants to perform the tests. “We just setup the hardware, software, and bandwidth to their specs. While we don’t do the tests ourselves, we have relationships with 125 consulting firms,” said Gruetter. And Applabs does a lot of work in storage area networks testing and stress testing eCommerce Web sites.

A few more things to consider when hiring a test lab:

Know what equipment, bandwidth, and test gear you need. If you want a large collection of actual PCs (versus virtual machine instances) to do your tests, the best place to start is with Applabs facility in Utah. They have 1,000 individual PCs that can run automation software for their tests. The other labs use virtual machines or synthetic test tools to simulate client and server loads.

“We are the only game in town,” said Doc Parghi, the senior VP for North America for Applabs. “The Utah test lab is a small part of our business and we have many other services, too. We have a significant presence in the UK and the majority of your staff is in India where we do automation and regression testing.”

Wherever you go, you should find out what test gear the lab has and whether it will be appropriate or not for your needs.

What kind of vendor support is expected during the actual test? “I often see that vendors don’t always send their A-list engineers though we are doing tests that may result in a purchase of tens of millions of dollars of their gear,” said Newman. “You just can’t send a junior field sales engineer who may not understand the business need or how to best deal with the flaws that we uncover during the tests, or the complexity of the application that we are testing.”

What kind of audience will be reading the resulting reports? You also need to define whether a C-level executive will be reading the results, or will you need your own IT engineers to review the reports. “Sometimes our clients want recommendations, other times they are satisfied with just the raw results,” said Newman.

A good strategy is to examine some writing samples before hiring, and most of the lab Web sites have links to download these or can send them on request. Applabs also can call on its own analysts that they can bring into a test evaluation too.

Finally, how much is this going to cost? Each lab prices their services differently, and getting an estimate might take some work. For example, the Columbus Platform Lab charges à la carte for bandwidth, machine usage and other items, such as renting any test equipment. They charge clients based on a formula of $12/MB per day of testing. But since they have access to some very large Internet pipes—up to gigabit Ethernet speeds—that can get expensive if a client wants a lot of bandwidth for their tests.

Network Test wouldn’t quote any prices, but they include all test gear, bandwidth, and equipment as part of their fee. ExtremeLabs charges $3,500 a day with a four-day minimum. Opus One charges $2,000 a day.