Wall Street Letter VOL. XLVI, NO. 7 - July 2014 | Page 18

JULY 2014 COMMENT Trading system failures cannot be our norm Jason Nothey, managing partner at Lasalletech, discusses the importance of continuous testing trading systems T 18 esting. Those responsible for trading systems know they need to do more of it. Few have the time to do it right or the resources to commit to it. The electronic trading industry has suffered a spate of system failures that have pointed to the need for more testing. The risk of not investing in testing is too large to ignore, yet this process goes overlooked and largely unchanged. Automating the testing process is the most costeffective approach, although it usually requires a large upfront investment and few organizations have the internal resources to implement it properly. The oncepromising ideal of automated testing is becoming more of a chore than a competitive advantage. THE ISSUES Automated testing is not always a silver bullet. The hardest part is finding a team with knowledge of the domain, a grasp of programming concepts, and experience with scripting. Testers with these skills do not stay in testing roles long. Outsourcing is an option, but the specialized domain knowledge and familiarity with your environment is gone once the project ends. An overlooked area is the cost associated with maintenance. Trading systems are always changing, driven by new regulations, functionality, or customer requirements. To avoid test suites from becoming stale, a good process must be in place to identify new use cases that should have automated tests. Test data is another source of maintenance costs. Those working in futures, options, or fixed income have to deal with expiring contracts and other date-related events. Market data is a point of contention for firms routing orders. There are cases where prices are driven by live market prices, or data from external test environments. In the best case, this market data can be controlled within your own test environment. More often pricing data is subject to counterparty systems such as exchange test environments. Simple bugs in calculations and field mapping can be caught with a simple suite of functional tests. The defects that are difficult to reproduce, such as race conditions and concurrency issues, need a completely different approach. “Flash crashes do not result from glitches, evil spirits, or sunspots,” says Bob Binder, author of Testing Object-Oriented Systems. “They happen when an unusual pattern of trading volume and message flow provokes a latent behavior that provokes even worse latent behavior. Traditional testing approaches, even stress testing, cannot trigger this failure. But, high-volume and high-variation test suites, running in a realistic test environme