Q&A: Cisco's Dave Malik on New Nexus Switches and Trading Fabric Architectures
Earlier this year, Cisco Systems announced new switches focused firmly on the low-latency trading space, and along with them, it announced its Trading Fabric architecture, setting out how these switches, along with other components, can be deployed to solve real-life application challenges.
Low-Latency.com checked in with Cisco's director of solutions architecture Dave Malik to get some more detail on the products, the architecture and the vision.
Q: What is special about your Nexus 3064 and 5500 switches that makes them especially suitable for electronic trading?
A: These switches provide ultra-low-latency at scale and protect against microbursts during market volatility. When talking about latency, it is critical to understand scaling requirements and the ability to manage short-lived congestion in the infrastructure. For example, by leveraging a 24-port switch, the latency profile drastically changes as the need to scale the fabric beyond this capacity. If there are packet drops, this event can have adverse affects which can cause packet retransmissions or simply loss of valuable data needed for trading applications.
For example, the Nexus 5596 with its 96 10gE ports provides 64MB of packet buffer memory and allows extending the fabric to 768 hosts with the exact same latency profile of less than three microseconds from any host to any host by leveraging the Nexus 2000 Fabric Extenders. The Nexus 5500 platform has a crossbar and port ASIC implementation with dedicated buffering per port.
The Nexus 3000 switch provides less than one microsecond of latency along with a rich feature set. The switch architecture is designed leveraging a single chip implementation providing deterministic performance on a 64-port configuration with 9MB of shared buffer.
Q: Apart from low latency, what are some of the other performance characteristics of these switches that you have optimised for the trading fabric?
A: The Nexus 3000 and Nexus 5000 have a robust Layer 3 BGP (Border Gateway Protocol) feature set which is typically required for exchange and client peering points. Both platforms provide extremely low jitter in single digit nanoseconds which make them the ideal choice for trading infrastructures. In addition, the security features, such as PVLAN, route control and filtering, Virtual Routing and Forwarding (VRF) for segmentation of traffic, and upcoming IEEE 1588 Precision Time Protocol support in hardware can be leveraged for sub-microsecond time synchronization for switches and hosts within the trading fabric.
[Low-Latency.com: There's more insight into the new 3000 switch in this video.]
Q: You put a lot of effort into testing the switches for performance. But we know, for instance, that market data rates are still increasing. So did you test to future proof them?
A: Cisco invests a significant amount of time in testing environments at scale. The testing is conducted for the particular Nexus switches themselves and at a topology level under increasing load with various traffic conditions. It is very important to understand each platform's limits before designing a trading fabric. The information is shared with clients so they can understand the various scaling parameters of the entire system. As an example, the scalability testing performed on the Nexus 3000 can be viewed here.
Cisco leverages industry benchmarks as well as application specific testing with ecosystem partners to ensure high performance with current and projected message rate growth. Several benchmarking examples can be found here.
Q: So, how do you take a switch - a product - and use it to build a trading fabric, an architecture?
A: Components behave differently when used as standalone devices as compared to when used in an interconnected fashion. The value of the Cisco architecture is that we formulate designs based on client requirements ensuring the infrastructure will perform to a client’s needs. Without architecture, the burden is on the client to integrate and validate a complete solution. The trading fabric consists of more than just a single switching component. There are compute, messaging, applications, monitoring tools, etc. which have to be considered since they are leveraging the fabric for applications which require having various flow characteristics at precise intervals.
The first step in defining an architecture is to understand clear requirements. The next step is to develop a framework that will eventually become the architecture. Once the architecture is defined, it is appropriate to select the best components to support it and how they are interconnected to meet the goals of the framework. The crucial step is to validate and ensure the design can sustain deterministic performance under current and projected peak traffic patterns as the architecture scales to accommodate growth. Monitoring and instrumentation capabilities are weaved into the fabric for real-time analysis.
Q: As well as switches, another element of your trading fabric are servers - your Unified Compute Systems. How are you engineering those for applications such as data feed handling, messaging, etc.?
A: The Unified Computing System (UCS) unites computing, network, storage access, and virtualisation in a single system which can be managed centrally with a single pane of glass. It is based on standard x86 Intel processors and also contains a high performance 10gE Virtual Interface Card for I/O intensive applications. By leveraging the adapter’s possible 128 PCIe standards compliant virtual interfaces, there are use cases where a firm can consolidate several applications on a single host and bind specific processes to a particular CPU cores which then can be associated with a specific virtual NIC defined on the physical Ethernet adapter. This allows streamlined application performance by minimising context switching and interrupt steering which can cause additional latency in the application.
The recently announced VIC 1280 adapter will be able to support 256 virtual interfaces for further scalability. For certain memory intensive workloads such in-memory data grids, the extended memory capability of UCS can be leveraged to increase performance.
Q: It's not a green field world out there - there's a lot of existing networking and servers out there. For firms that want to take advantage of your fabric approach, what practical measures are you making to help them transition?
A: Cisco’s Advanced Services consulting organisation works closely with our clients to understand their business and technical goals and help them evolve from where they are today. By leveraging vast experience working with financial services firms across the world, Cisco has developed robust best practice architecture designs to ensure clients successfully transition to new trading environments. A trading fabric can be located in a single location or span multiple locations, which is the case for several of our clients. Cisco also has large testing facilities where clients can validate future designs to ensure a seamless transition of their applications. Scale-out testing with different traffic patterns under increasing load provides greater visibility on what to expect during a production deployment. The network, compute and storage infrastructures need to work together as a system and there are methodologies formulated for firms to leverage. Direct involvement with ecosystem partners in the entire process also enables that the system is being tuned on an ongoing basis.
Q: Dare I mention InfiniBand? There's quite a lot of it out there, especially for so-called ultra low-latency applications. And it certainly has a passionate constituency. What's your position - steer clear of the fight or take it to them? And if the latter, what's your proposition?
A: In order to take advantage of InfiniBand performance, clients have to rewrite applications, which is unacceptable in most cases. With 40gE switching products becoming available in the market this year, Ethernet is delivering the highest levels of performance that firms require for demanding workloads.
There are other ultra-low-latency technologies in the RDMA and sockets-based arena which leverage Ethernet as a transport. RDMA over Converged Ethernet (RoCE) has provided a path to clients who are using InfiniBand and want to migrate to Ethernet. The industry trend reflects Ethernet as the transport of choice today. Further innovation in silicon and software which will reduce the small latency gap between Infiniband and 10gE wich exists today from an application perspective.
Q: Finally, can you put any flesh on the bones of plans to push adoption of the trading fabric, and of the products that underlie it? What can we expect in the next few months? Don't worry - everyone in the Low-Latency.com community has signed your NDA :)
A: The trading fabric is currently being implemented by several firms who are leveraging strategies across multi-asset and multiple venues. They are on a journey to capture new market opportunities. The Nexus 3000 and Nexus 5500 have been adopted in various environments depending on the firm’s strategy. Cisco is also working with ecosystem partners in the entire stack to ensure success. Cisco cannot comment on specific product roadmaps but you can expect continued innovation in this fast growing space.
Add comment
More like this
- Q&A with Dave Malik of Cisco
- Q&A: NYSE Technologies' Brian Doherty on Data Fabric V6 and MultiVerb
- With New Switches, Cisco Weaves a Low-Latency Trading Fabric
- Q&A: Alice Botis of Fidessa on Latam Trading - Opportunities and Challenges
- Q&A: Telx's Michael Cattell on Co-Lo in Chicago, and Trading Ecospheres
- Q&A: Sébastien Jaouen of Orange Business Services on MiFID II, Electronic Trading and Latency
- Q&A: Sidera Networks' Maura Mahoney on Partnerships and New Data Centres
- Q&A: Craig Betts of Solace Systems on Latency Beyond Algo Trading
- Q&A: Brennan Carley of Spread Networks on New Services for New Customers
- Q&A: Joshua Walsky of Broadway Technology on Functional Scale and "Build and Buy"
Comments
Decided to give Doug's comment a little cooling off time. Doug worked at Cisco before Arista, so he does have an inside track. That said, he of course now works for a competitor - and we've asked him to participate in a Q&A for his world views.
Anyone independent out there have anything to say on what Dave says, or what Doug thinks?
In full disclosure I work at a competitor to Cisco Systems (yes, I still use the Systems when I spell it, hard to think of the consumer-focused 'Cisco' when I learned on ciscoSystems in the 90's)
While I know and like Dave, this piece exemplifies much I find distasteful in Cisco's marketing and product promotion strategies, let's assume someone in marketing got their hands on it before Dave sent it in :)
1) I have found that the financial services industry contains some of the most sound engineers I have ever worked with in networking. I think this is primarily because information is profit margin in many of these firms and their IT infrastructures are one of the most tangible assets of many banks and funds. Thus, some of the best and brightest in technology flock to financial firms for quite gainful employment.
Therefore, I find it odd that a vendor feels they need to 'tell' or 'dictate' how to build an infrastrucutre to these incredibly competent technologists. This steam-roller technique seems all to prevalent, but feels wrong in an industry that so far, to my knowledge has rewarded vendors who listen to their requests, have informed dialogues with empowered engineers on both sides, and then build the best solution possible based on that customers unique requirements - and trust many many of them are pretty unique.
2) The 'Bait and Switch' (pun intended) is almost comical. Clearly, from every reputable 3rd party non-vendor sponsored test (or anyone with some clueful understanding of an Ixia or Smartbits or even better who can replay an actual feed) you will find that the Nexus 3000 does not have large buffers, nor does the Nexus 5000 have low latency. But with all the smoke and mirrors deployed around these 'trading fabrics' it might actually be understandable if someone were to think these products were designed 'ground up' for trading environments.
They were not. The Nexus 5000 was designed to transport FibreChannel in a layer-2 network in a lossless fashion. It was a time-to-market product created because no one at Cisco felt R&D should be funded enough to create an in-house ToR switching platform for the data center so Cisco had to acquire switching technology - heinous considering the billions poured into such lucrative markets as Telepresence, Umi, Cius, and Flip.
The Nexus 3000 is a first generation Broadcom Trident chipset. A decent enough chip, but not one I would ever position for low-latency trading. It has variable jitter and non-deterministic forwarding characteristics based on frame size and feature set turned on. Fine for clouds and enterprise data centers where power efficiency rules, but not suitable for a trading platform focusing on ultra-low latency.
Even worse I suppose is that according to a recent financial research report Cisco is obsoleting the current Nexus 3000 in the next few months to replace it with one that uses Broadcom's 2nd Generation Trident+ chipset that gives larger table sizes. (I feel sorry for anyone who rushed to buy the Gen1 product who is about to find out that it has a six-month lifespan)
But if you think that is bad Cisco is working on yet another version of the Nexus 3000 due out in 2012 that is basically a clone of the Arista 7050 switch design featuring PHY-less design (saves 100ns or so and lowers the power draw). But it still won't be the best for financial trading - but may as well wait for it if you're looking at them...
Last point, if this 'Cisco Trading Fabric' (am kind of getting sick of Fabric as the latest marketing buzzword in networking... thanks JNPR guys for that!) is designed from the outset for low-latency then why doesn't Cisco publish real-world benchmarks? Not the 47:1 incast testing you tried to say was real-world last year on a blog post subsequently removed because it was egregiously inaccurate, but a real test like say an ARCA feed or OPRA feed, using accurate counting of latency numbers.
If you are reading this site, and thus this post its either becuase you are a) a marketing guy cringing at this because someone just called your baby ugly which will usually lead to lots of spam comments from people with creative names like 'Anon'. Or b) you are a real engineer with an interest in low latency trading.
For b) challenge all of us as your vendors. Hold us to high standards of conduct and don't buy the marketing hype or BS. Test our products, make us earn your business, and make sure we know your business so we can support you and your goals - regardless of which vendor it is you choose. Think, make your own decision, be an engineer and evaluate all your technology options and then choose the best technology to meet your business goal as economically as possible while ensuring you get the support you need to be successful.
Thanks for your time.
dg
---------------------
Douglas Gourlay - VP, Arista Networks