Kx Taps Multi-Core for "Intelligent" Data Compression

Version 2.7 of Kx Systems' kdb+ database leverages multi-core servers to implement file compression, while also featuring enhanced memory management and a performance increase of up to 20%. The file compression addresses the need to store growing data volumes for both trading and risk analytics and regulatory requirements.

Kdb+ implements what it calls "intelligent" compression - instead of a simple 'on/off' choice - allowing a choice of compression algorithms and levels, as well as the ability to specify which data to compress and how heavily to compress it. For example, current day data might not be compressed at all, data that is up to a week old could have only selected, customer-specified, fields compressed, while data that is older than three months might be fully compressed.

Kx's chief strategist Simon Garland says it's "difficult to give across the board figures" for compression ratios, though he says that based on day's worth of recent quote and trade data, trades are compressed by 85% and quotes by 75%. Currently, the actual compression is a manual process, though compression algorithms are platform-independent and available on all operating systems. Some algorithms rely on operating system-specific implementations, though third party compression hardware can also be used, as can third party algorithms.

Garland says that multi-core servers make compression a "straight latency to dollars equation," adding that allocating "a few" cores on a server than might typically contain 32 cores is a good use for them. One key reason for the flexibility over algorithms and what data is compressed and how is to optimise speed of access to it.

"It's not the case that we're having to decompress the whole column just to get a few values out," says Garland, who adds that compressing data means there is less to read from disk storage, which is often a performance bottleneck. Version 2.7 also features enhanced memory management, which directly improves overall performance - up to 20% faster than version 2.6 on similar hardware configurations.

Also new is protection against invalid incoming data, with a data structure verification mechanism, which detects invalid user data structures for a more robust environment as demanded by increased data volumes. The performance of databases for tick data storage has been a focus for Stac - the Securities Technology Analysis Centre - which recently released an audited Stac-M3 Benchmark of kdb+ 2.6 running on Oracle Sun Fire X4540 Storage Server. Stac-M3 is a database benchmark that uses community-developed testing standards, with test specifications developed by a working group of the Stac Benchmark Council, comprising trading firms and vendors.

The working group has now submitted these specifications for an approval vote of the full Council. The initial version of the benchmark focuses on I/O performance with respect to heavy historical data loads in single-user and multi-user situations.