Stac's Vault Gaining Momentum; IBM an Early User, Benchmark Supporter

Introduced late last year as a private repository for members of its Benchmark Council, the Securities Technology Analysis Center (Stac) has found its Vault facility is proving useful to members, especially for the filing of unaudited reports that are not quite ready for fully public consumption. IBM is one vendor that is embracing the facility, which is just one of the initiatives that Stac is focusing on for 2010.

According to Stac founder and director Peter Lankford, the Vault - a "work in progress" he says - allows vendors to make benchmark results, derived using Stac's test harness software and procedures, available to certain Benchmark Council members without incurring the costs associated with having Stac audit the results, as is necessary for public dissemination of benchmarks.

The Benchmark Council comprises trading firms and vendors (currently around 100 strong), who are responsible for specifying standard methods for measuring the performance of trading systems, thus creating the formula for Stac Benchmarks. As such, Council members are deemed knowledgeable and experienced enough to treat unaudited benchmarks with appropriate caution, while vendors are subject to the "reputational pressure" of ensuring the benchmarks are valid, says Lankford.

Only trading firm Council members with appropriate subscriptions can request benchmarks from the Vault, and in so doing they agree not to disclose the information to others. IBM recently filed its third benchmark into the Vault, the latest in a series related to Stac's M2 market data distribution benchmark.

Lankford says that IBM has been an "aggressive supporter" of the benchmark, which is based on input from seven trading firms and six messaging vendors. M2 covers performance metrics, such as throughput, latency, power efficiency and CPU/memory consumption under several scenarios, including equities and options use cases, such as black-box trading, market making, and smart order routing.

Indeed, last November, IBM was the first vendor to publish an audited M2 benchmark, which was finalised just a couple of months earlier. That benchmark focused on a messaging stack consisting of IBM's WebSphere MQ Low Latency Messaging (LLM) middleware, Red Hat Enterprise Linux, quad-data rate (QDR) InfiniBand from Voltaire and Mellanox, and IBM blade servers, using Intel X5570 processors.

Highlights of the benchmark made public show that in use cases where latency is a key end-user priority, this LLM stack demonstrated 99th percentile latency of 18 microseconds or less at message rates up to one million messages per second. Standard deviation of latency in these cases did not exceed three microseconds.

More recently, IBM's unaudited benchmarks posted to the Vault relate to stacks that incorporate 1 Gigabit and 10 Gigabit Ethernet networking. As well as work on market data benchmarks, including the M3 variant for time series tick databases, Stac is also developing transactional messaging benchmarks, such as E2, for guaranteed delivery of trade-related messages.