Corporate IT shops that need to roll out new network, infrastructure and Internet applications or who have to make expensive equipment upgrades are increasingly turning towards independent test labs as part of their decision process.
The labs can help identify weak points, application bottlenecks, design flaws, and show-stopping bugs in a non-production environment, and are also useful “bake-offs” to pit potential candidates against each other in controlled scenarios that can mimic their eventual use. There are several outsourced labs that are available from a number of small to large-scale providers.
Why go with an outsourced lab? The two biggest reasons are expertise and resources. “Some IT shops do have the testing expertise, but they don’t always have the time to set up a test given the press of daily operations,” said Joel Snyder, senior partner of Opus One in Tucson, Ariz. “It is often better to bring someone in who does testing as part of their daily job.”
The specialized test gear is more than a bunch of PCs on a gigabit switch. “We have all sorts of test equipment here,” said David Newman, director of Network Test in
Greg Shipley, the CTO of Neohapsis in Chicago said, “There are still specialized skills, test plans, environment preparation, environment validation, and a slew of other components necessary to do world-class testing that are not usually found in abundance in corporate IT departments.”
Testing can also avoid potential show-stopping issues with a roll out. “We did a test for global logistics company and found incredible poor performance of a IBM ISS security appliance,” said Snyder. “It turns out that they were sending traffic over a well known port that ISS expected to see something else and didn’t know how to decode the traffic. We fixed the configuration of the ISS box and saved our client a lot of grief had they put this into production without doing any testing.”
“The Ohio Secretary of State came to us because they expected a great deal of load on their Web site on election night and wanted to validate that they wouldn’t experience any problems,” said Steve Gruetter, the director of Platform Labs in
“Organizations will often come at product evaluations differently,” said Shipley. “Some want help assessing features and functionality, others are looking for help with performance testing, others are looking at the security of the product itself. Some want little more than helping define their evaluation criteria, where others want help with all of the above.”
Here are a few tips to consider when shopping for a test lab:
- If technical criteria matter for your purchase, make sure you understand what they are before you put together your test plan. “When the time comes to deploy a product, you have to prove that you didn’t make a big mistake. Even so, most people don’t usually buy products solely on technical criteria,” said Snyder.
- Understand what testing for performance really means, and what are the boundaries of acceptable and unacceptable performance before you set up your test plan. “Have clear goals going into tests. Don’t change objectives during the project, said Newman.
- Know what the facility specializes in, and what the actual lab testers’ expertise is before you hire them. This is just common sense, but each test lab has a different specialization as well as size and facilities. They range in size from the one-person Network Test to the multi-office global companies of Applabs and West Coast Labs.
Some, like Network Test, do more performance-related tests, while Opus One specializes in firewall and virtual private network devices. ExtremeLabs has deep expertise in network and Internet applications.