Why Milliseconds Matter:
NASDAQ Replays the Stock Market on the Desktop via Adobe AIR

– Claude Courbois, Associate Vice President for Data Product Research and Development, The NASDAQ OMX Group

Claude Courbois, head of data product R&D at NASDAQ OMX, is responsible for creating data products that make stock trading more efficient. Claude visited with the UIRC recently to share the back-story of NASDAQ OMX’s new application Market Replay and to discuss his views about how Adobe AIR and rich Internet applications are impacting the financial services sector. 

Q. Market Replay has gotten a lot of great press. What were the business decisions behind bringing an Adobe AIR application to market?

A. I would guess that most people reading this article have likely made a stock trade or two and share a common concern with other investors — the nagging, universal feeling of not knowing what happened — why they got the price they got. For example, you get a trade confirmation from your broker saying you paid at $21.20 a share but you remember the stock was trading at $21 when you placed the order. What happened? The answer often is that you traded during a price change that lasted as little as a fraction of a millisecond — something that takes an awful lot of data to find and confirm.

For a long time, NASDAQ wanted to enable people to go back in time and look at that detailed data but we had no way of doing that cost effectively. Generally, we’re in the business of supplying market information and have left the storage and analytics challenges up to our customers — be they brokerages houses, traders or service providers like Reuters and Bloomberg. So, in the past, each of our customers had to rely on their own resources to store data and use it. That is a very expensive and server-intensive undertaking. Since we’re upstream of all of them we see an opportunity to save them costs and improve service.

Q. Can you give us some real-world examples of how Market Replay is being used and what its benefits are?

A. Our primary target is the market professional — dealers and brokers who want to give their clients a higher level of service. Let’s say an investor calls his broker to find out why he paid 20 cents more than he expected per share. This happens all day, every day.

Brokers have a variety of data sources they can use for answering these questions, but Market Replay provides benefits beyond saving people money. First of all, it’s NASDAQ validated, and the broker can send the customer a screenshot of what the price was at the time of trade, validating that they got the best price that was available at that time. That shot says NASDAQ OMX on it, so they know it’s trusted third party data. Obviously, brokers are highly regulated, so it’s not really about catching wrongdoings. It’s about building confidence in the system. From everybody’s point of view, it’s nice to have reputable, third-party, validated data. 

Second, our system is just incredibly fast compared to querying a database. It’s not that databases are slow by nature. You can build a database to hold this amount of data and it can be very responsive, but typically the cost of doing that is very high and most of the data is never accessed. You’d be storing thousands of stocks, thousands of days of data, and keeping it online at all times while most people are only investing in a small subset of all stocks. The costs are huge. At NASDAQ, when we hit our biggest databases, it can take hours to get the data back. We don’t have a customer waiting on the phone call like the broker does, so we don’t necessarily have the same need for speed, but it gives you an idea of how slow it can be to get this kind of data out.

Third, many financial firms offload data once it gets to be 30 or 60 days old. They take it offline to control costs and improve performance for the more recent data queries. If a customer calls up about a question about a trade that’s 61 days old, the broker has to say, “Sorry, I can’t get that data.  I’ll have to get back to you in two weeks, or whatever.”  Then someone has to research it and has to pull it off a disc or tape or whatever they backed it up on. With our system, we’ll never have to offload data because it’s less expensive to store the way we do it, and access to it is consistently fast no matter how many files we load.

Q. Is Market Replay affecting the trading world in other ways beside customer services?

A. Yes. One feature of Market Replay is that you can replay the market in simulated real time, like slow-motion review of a sporting event. If you see something in the market and you want to experience it again— maybe to learn from it, or to see what you did wrong, or to better understand the trading— you can go back and do an instant replay. It’s about giving investors a better understanding of the market so they feel more confident in it. Also, for day traders and professional traders, it’s about training and reviewing interesting market events and opportunities.

Another example is when someone from NASDAQ’s investor relations office called me up and asked if he could do a replay of trading during an earnings call and synchronize it with the audio from the call. He wanted to see what the traders liked to hear and what they didn’t like to hear. We did it and found out how amazingly fast the market responds to what is being said on those calls. You could see the stock immediately start to trade up when they talked about revenues being higher than expected. When they started talking about expenses being higher than expected, it started trading down. It was amazing. One thing we’re going to add to Market Replay at some point is a way to record replays as video you can easily share with anyone online. I patched this example together using third party software that records video from the desktop. But we’d like to build that functionality into Market Replay so people can do their own recordings and save them.

Q. Why did your team use Adobe technology for this project? Did you consider using any other development framework?

A. The driver behind choosing Adobe AIR for this project was its ability to provide a lightning fast data visualization experience on the client side. It removed the barriers we had been facing, which were high development costs to build servers that could store, query, and replay the market data in the detail we needed without making the user wait for endless requests back to the server. I had been pitching the concept to our team for several years, so when Adobe started working on AIR, we saw the opportunity to push the complex data processing to the client side. We began work on our application when AIR was in alpha. By the time Adobe released AIR, we had already been working with it for about eight months.

Adobe AIR lit a light bulb over our heads. It seemed to be our ticket to creating a playback tool that would do the heavy lifting on the client side so it would be a viable solution that all investors — individual as well as professional — could afford to use to dig into trading and see what was happening in the market at any given millisecond. 

Q. It seems as though this application could have been hosted in a browser. Why did you opt for a desktop solution?

A. We never tried to build it in a browser because every time I would pitch the idea of building Market Replay, the technologists would push back and tell me how expensive it would be. We’d have to keep enormous amounts of data online and ready to stream to the browser at all times. Each time the user interacts with the data by moving the playhead or zooming in and out, it would have to go back to the server to ask for calculations, sorts, all kinds of manipulation of data.

Financial market data is huge. Our requirements in terms of performance and user experience interacting with the data were just more than our technologists thought we could build in a browser without having an enormous server infrastructure behind it.

Q. Where is the data being stored and how does it get to the AIR application?

A. We prepare the files ahead of time and then load them on Amazon Web Services S3 as simple, flat, text files. One file covers one stock on one day for 10 minutes of market time. For example, the first file for each stock is from 9:25 a.m. to 9:35 a.m. each day, and all the activity for that timeframe is in that one file. So if you want to see the events around 9:31 on any given day, the Market Replay application knows to reach out to Amazon and grab the file that includes the data covering 9:25 to 9:35. We store the data in a very simple way to minimize the file size and bandwidth requirements. The Market Replay application pulls down the file and then loads it into a format that is best for replaying, zooming, making calculations — all the functionality that we built into Market Replay.

The great thing about the Amazon S3 file system is that it’s extremely fast, despite the fact that it contains billions of files. We load millions of files — more than 300,000 files a day — and we’re probably a trivial customer for them. Amazon is loading many millions of files a day. Despite all these files, the service is extremely fast. Plus, they put files on multiple servers so there’s no risk of loss. It’s a level of infrastructure that would have cost us a lot of money to build. But we couldn’t use Amazon S3 if we didn’t have such a powerful AIR application on the client side because that’s what allows us to have such a simple server architecture—the plain fact that the work is done on our client application.

It’s all about enabling us to provide our customers a higher level of service at a better price point. This will enable more professionals and nonprofessionals to have access to detailed market data that in the past was only enjoyed by those with the largest data infrastructure budgets.

Q. Is the data coming from a single source or multiple sources, and is that going to change? 

A. Our first version of Market Replay, which has been in production since early in 2008, shows the order book, which are the bids (orders to buy shares) and offers (orders to sell shares) that are available in the market at any one time. So for example, if you bought shares you would want to have bought them from the seller offering shares at the lowest price. You use Market Replay to review what was available at the time of your trade to see that you got the price you expected. The current Market Replay gets that data from a single source — a single set of files on Amazon S3.

Now we’re rolling out trade data that will show people the prices and share volumes of the trades that happened. It’s going to be another 300,000 files or so a day from a different source. We have different code preparing those data files. So when you access a replay and you ask for both quotes and trades, Market Replay will reach out to two different sources for two or more files. Right now if you just want to see ten minutes of market time, it just has to grab one file, but soon the minimum will be two files because it’ll have to grab the quotes from one place and carry it to another. All the data will be on Amazon S3.

The Market Replay application then takes care of consolidating the two data sources on the client’s computer, creating a consolidated view. It’s a lot of work to do, especially as users move around through time and zoom in and out of the details.

Q. Has Market Replay had any effect on the NASDAQ OMX brand, or is it too early to tell?

A. We just rolled it out a few months ago so we’re early in the sales and distribution process. But I will tell you that we have never had a product get this much positive feedback so quickly. We’re in a pretty esoteric business — market data. When we come out with new products it’s hard to get the story out, even to professionals. It’s mostly because we have to tell people about the data rather than show it to them. We sell data feeds. We don’t traditionally provide the user interface. We’ve been coming up with new data products for several years now and it’s hard to get people to visualize and get excited about a new piece of ones and zeros that we send out on a data feed.

But this product is so visual and so exciting in terms of its user interface, and the speed with which it can help someone solve very real problems they face on a daily basis. It’s nice to have a product that people have already been talking about before we visit them. When they see how useful it is they get excited and tell others. They’re very impressed with the user interface and the response speed. It’s just been a real pleasure to sell this product.

Q. So how can people gain access to Market Replay?

A. To try it out, visit our DataStore at www.data.nasdaq.com. NASDAQ is a data wholesaler so to gain full access you need to contact your regular source of market data. For people working in the financial industry their data manager can provide access by contacting NASDAQ directly — there’s information at data.nasdaq.com. Regular investors should request the product from their existing source of financial information whether that’s a financial Web site, an online broker or another kind of financial information provider.

Q. Like many other things of this generation, the stock market has become a totally electronic, virtual place. To wrap up our conversation, let’s take a look forward. How do you see rich Internet applications having an impact in financial markets?

A. We think about that a lot. While Market Replay is an innovative tool, it’s not meant to be terribly creative. We took familiar interfaces like a time chart and order book and gave them new features for the financial industry like simulated real-time replay, panning through time and zooming into shorter time frames. Those features help people get their work done better and faster, but they aren’t revolutionary.

With the success we’ve had in creating a new user interface, we’ve been thinking more and more about other visualization opportunities. One huge opportunity is the overall look of trading as an activity. In the past, when people thought of the stock market, they envisioned the floor of an old fashion floor-based stock market like the New York Stock Exchange. It used to be a really compelling place because it was packed with people running around. When it got extra busy, that was news and you could see it happening in pictures and video of people running around frantically. You could sense whether people were bullish or bearish about the market by how they were behaving on the floor, and so that made great TV and it made a great place to find out what was happening on Wall Street. That’s why people paid a lot of money to be on that floor.

Today, the floors are pretty sleepy places. There are still people trading on them, but they’re never very crowded even on a really big trading day. If you notice, on TV or on magazine covers, they often show the floor blurred to make it look busy. Our kids are not going to visualize stock and bond trading by thinking about human beings in motion like that.

Now that all exchanges are going electronic, there’s a real opportunity for someone to create a visualization of electronic trading to replace the video that used to come from those floors. We’re part of the way there at NASDAQ OMX. We’ve created data products called NASDAQ Market Velocity and Market Forces that measure the same order activity and sentiment information that traders got from watching and listening to activity on a floor. Unlike the information on the floor, Market Velocity and Forces is available to anyone.

Now it’s up to someone to figure out how to visualize that data in a way that’s just as compelling as the video of yesteryear. There are heat maps, but they haven’t really captured it yet. Maybe it will be something totally whacky like virtual avatars running around. Whatever it is, it’s a real opportunity for someone to revolutionize what people see on TV and in their heads when they think about trading. And presenting it is going to require lots of data and robust tools like AIR and Flex to bring it to life.

Claude, thanks again for spending time with us today, and good luck maturing Market Replay. Maybe one of our readers will hook onto the creative concepts you spoke about.

Thanks very much for the opportunity. I enjoyed our conversation. Remind your readers to check out our demo at www.data.nasdaq.com.

About Claude Courbois
Claude Courbois is the associate vice president for data product research and development at the NASDAQ OMX Group. He is responsible for developing innovative market data products designed to increase profits for investors and market participants. An online collection of the data tools is available at the NASDAQ DataStore. Mr. Courbois has been with NASDAQ OMX since 2000, and previously held positions as the director of financial and market analysis for the company’s Data Products division and as an economist in its Economic Research department. Before arriving at NASDAQ OMX, he worked at the International Food Policy Research Institute as a research analyst specializing in the development of food production and markets in developing countries. Mr. Courbois has a PhD in economics from North Carolina State University and is a CFA charterholder.