Identifying Fault Lines in the Equity Market Ecosystem

[MUSIC PLAYING]

PETER HAYNES: Welcome to episode 54 of TD Cowen's podcast series Bid Out, A Market Structure Perspective From North of 49. I'm your host Peter Haynes. And in today's episode, our topic will be Identifying Fault Lines in the Equity Market Ecosystem. Joining me today are two experts in the field of market infrastructure. And both are long term veterans of the equity trading landscape.
First, I'm pleased to be joined by Stephen Plut, Managing Director and Co-founder of ITS Online, a Canadian company focused on providing software and solutions to brokers including routing technology widely used by Canadian firms. Our second guest, Etienne Phaneuf, is well known to the Canadian equity community, as he was previously CEO of ITG Canada, and now is president and CEO of ArchES Computing, a company that builds market data and trading systems that handle the explosive increase in message traffic, which is certainly going to be a theme of this discussion. Steve and Etienne, welcome to the show.

ETIENNE PHANEUF: Pleasure to be here, Peter. Thanks for having me.

STEPHEN PLUT: Thanks for inviting me.

PETER HAYNES: I've got to do the disclaimer before we get started. I want to remind our audience that this podcast is for informational purposes. The views described in today's podcast are of the individuals, and may or may not represent the views of their firm. And of course, the content of this podcast should not be relied upon as investment, tax, or other advice.
Now I'm sure I'm going to get accused of fearmongering with this episode's premise. But I do think it is important that we don't just assume the plumbing of equity markets will always work, especially in light of the rapid and seemingly unfettered pace of change in the space, and the fact that individual firms and marketplaces are always trying to innovate for commercial benefit. Meanwhile, the market ecosystem becomes more complex and more intertwined, and in my opinion, more vulnerable. I say we should be loudly socializing these risks, and in particular, we should be doing so with the regulatory ear in mind. Let me ask you, Etienne, do you think I'm fear mongering?

ETIENNE PHANEUF: I think you ask a reasonable and a fair question. The thought of you on Bay Street holding a sign saying, "The end is nigh," it gives me a personal chuckle. I don't know about fear mongering. I think the market ecosystem has shown itself to be quite resilient. But I think in general, we don't really know where the breaking point is or could be.
It's a highly complex system with many different types of participants. The exchanges have many different types of books and marketplaces. And the regulatory environment is also complex as well. And all of this has evolved organically over the last 25 years, as we went from the floor to electronic markets.
Where are the weak links in the system? And at what sort of level of activity in message volume, which is not the same thing as order traded, but in you terms, of the messages that are generated in terms of market data and orders, where's the breaking point there? If we suddenly get twice as much message traffic tomorrow as we did today, does that cause problems for the market? Is it four times or six times?
I don't think we really know. And beyond that, it's an evolving kind of question to ask, because the system is not static. We keep getting new marketplaces, new order types, new changes, new market participants. So I think the question is very reasonable. Where does this system potentially break down, where are the weak links? And I don't think we really know.

PETER HAYNES: You touch on a really important topic of sort of where the breaking point is. My first recollection-- I'm going to talk about Nortel in a second. But my first recollection on this topic was a speech by a gentleman by the name of Jim Gallagher, who used to be the executive vice president of the TSX. And he had been hired by the TSX from the New York stock exchange.
And his presentation, and unfortunately passed away a few years ago, but I really respected Jim and his discussion at that conference in like 1993. I think I was working at the Exchange, was all about understanding how much technology capacity or computer capacity they needed at the TSX. Which at the time, was a hybrid floor and electronic with CATs. How much do they need for the busiest day ever? And how do we predict that? And how many servers do we need to buy to be ready for the day when things blow up?
And it reminds me Stephen, of around Nortel. This was another area where we started to think about marketplace instability and things like load balancing. And I remember the Exchange literally put Nortel on its own server because of the amount of activity that it was seeing during its heyday in 2000, and then its implosion thereafter.
So we've come to understand since that Nortel day, 20 or so years ago, that technology problems will happen, that exchanges are really technology companies. So we've probably seen maybe 10 major TSX outages in those 20 years since Nortel, which at the same time, we've seen, as you mentioned earlier, ATN, the parabolic increase in message traffic. My question for you Stephen, is, with 10 problems in the last 20 years, one happening every couple of years, everyone understanding these things happen, have we just become too accepting in the equity space of this conclusion? Or is one problem every couple of years at our primary listings venue too much?

STEPHEN PLUT: One every two years, given the complexity of everything is actually-- and most of the outages are pretty short in nature. Talking about the old CAT system, Bre-X was the first one that I remember. And I was actually working at the Exchange at the time when Bre-X was crashing the trading engine. The problem with that one was it was extended over a couple of days, that it would run and then crash later in the day.
And then TSX did a very concerted effort to stabilize their systems, come the new trading engine with some of the stuff that Nortel was forcing on them. In general, once every two years is not that bad. I'm worried that there could be a bigger outage that has more impact, as opposed to these small outages of two hours, one hour. And with multi-markets, the broker dealers are getting more sophisticated on how they're able to handle the order flow.
And so we have a little bit of more resiliency in the system because of that. But the complexity has gone exponential, as you mentioned. And therefore, there's things in the weeds that we don't know what's going to happen. I think most people would say if all we had is one outage every two years, I think they would call that a win. But I'm afraid that we might have something worse on the way.

PETER HAYNES: I just got off the phone with a friend of mine in the business. And we got talking. And he's in the US. And I said to him, you know, one of these days-- I'm not wishing this to happen. I'm not willing it to happen. But one of these days we're going to come in, and one of the primary markets in the US isn't going to open, and it's not going to close. And it might not open the next day for whatever reason.
And I get in trouble from my exchange friends in Canada, because I feel like when there's an exchange, a problem in a Canadian marketplace or primary listings venue, the TSX, there really isn't a lot of public profile. Whereas in the United States on that day that may or may not ever happen, if it was to occur, the first thing that's going to happen is within one week, the president of that exchange is going to be in congress.
They're going to walk in. And they're going to get grilled. And there's going to be media there. And they're going to pay a branding price.
So my question for you is, I know that a lot of what we would see in congress would be political theater. But do you think in Canada, when there is a system outage at the primary market, that we need more attention to that? Should we have our government more involved? Or am I looking for unnecessary negative attention to Canada's equity market?

STEPHEN PLUT: I don't know if the government is at the congressional level or the equivalent in Canada would have the background to be able to ask the right questions. But there are regulators that are very close. It wouldn't be an unreasonable thing that they get involved when there's major outages. And they ask questions. And they verify that what caused this, is there any systematic problem that caused this issue, and how to prevent it in the future.

ETIENNE PHANEUF: Yeah, I'm going to I'm going to jump in here, Steve and Peter, and lean on my experience at my prior role. ITG ran a marketplace called MATCH Now. And we had outages. Technology is technology. And they happen from time to time.
I can tell you that the regulators did play an active role. And those outages were painful, from a business disruption perspective and a reputation perspective for that business. It defined and fixed the problem.
You just deal with the client aftermath. They wanted to know what's the post mortem what happened. Can I safely route my orders back to you tomorrow? Some of them didn't. You had to really hold their hands.
There was a business penalty that came with that. And then the regulators would come in, not in a public fashion, but ask questions about what happened. Is it a systemic risk? What are you doing about it? Can we expect this to happen again? Are you are you running too lean, et cetera?
They dug in very deeply on that. And that was its own process that ran several weeks, sometimes several months of back and forth, which also disrupted any dialogue you were trying to have with the regulators in terms of driving something new. So I would say that while it's not public, it's not in the news, it's not in front of a congressional committee or a parliamentary committee in Canada, there's a significant amount of back and forth with regulators that carries a business disruption element to it, for whatever you're trying to do that day and that week and that month. So it's a good incentive there, I think, to not have those things happen.

STEPHEN PLUT: And they're also being held to it by their customer base, which has much more direct impact on their bottom line than let's say, a regulation from Ottawa. So there's a very strong benefit for them to be very open with their customers, because they can route away as Etienne mentioned. In a dark market, people could easily route away. And so his reputational risk, he needs to do everything in his power to make sure that he runs a very clean shop.

PETER HAYNES: But you can't run away from the primary market. If it goes down, we don't have as good a failover mechanism as people want. We all understand why that is not easy to fail over. But might, I guess maybe a happy medium then, Etienne. And I appreciate that you've got the wrong people asking the questions for political theater.
But my feeling is when our primary market goes down and it has a black mark on the Canadian capital markets, and I know it happens elsewhere, maybe at the very least, we need public hearings to discuss this so that we as an industry, the users, have a better understanding of what happened and the remedies that are being taken by those primary marketplaces. Maybe we don't need all the political theater. But I think the industry, I feel like we're a bit in the dark.
And yes, I do understand regulators are behind the scenes doing their homework, and I respect that. But I've said this to the regulators too. I don't understand why we don't get more public profile, to at least have a comfort that the right steps are being taken to remediate and this risk for the future.
So Stephen, my impetus for this particular podcast came following a problem that you and I discussed at length, actually, that occurred back in February. And it was during a reopening auction for shares in TECHB. TECHB was halted in the morning for news that was material. The stock was rallying. And there was an order that was placed to sell 10,000 shares of TECH Short during this reopening auction, where every single time you put that order in, it repriced every other order that was in that auction.
So what ended up happening was that order ended up being routed to the Exchange, using a third party router that had a loop. And it would cancel the order and reenter the order. And it ended up doing it 5 or 6 million times in the span of 10 minutes. And every time it did that, it would cause a burst of outbound messages, as every single order in that reopening auction got repriced.
So while the Exchange tells us very clearly they do not throttle-- sorry, they throttle on the inbound to avoid a DDoS or a DDoS type of activity, they don't throttle the outbound messages. And sure enough, what it ended up happening was industry routers or industry ticker plants or whatever it might be that receives this data, literally got overloaded. And I'm sure you can share your personal experience at ITS. But it really caused Bay Street to shut down for literally a half an hour.

STEPHEN PLUT: Yeah, it was down for about 30 minutes. We were down for, because of our quote, "plant got overloaded," we were down for about 5 minutes. So we actually came through it quite well. But yeah, no, I remember the day very well. And because I was scratching my head, wondering about what actually happened to cause this amount of messaging, knowing that the Toronto Stock Exchange does throttle on the input. And usually, input output messages are 1 to 1.
And then it dawned on me after a couple of conversations, that it was because of what's called a Calculated Opening Price Assignment. And the TECHB book was extremely large. Every time this order came in and modified the calculated opening price, it had to send out all these messages. And that caused the message storm. And yeah, it wasn't good.

PETER HAYNES: So what do we learn from this episode? We learned-- what I learned was number one, routers are outside of the regulatory purview. So even though this may have been a marketplace router that was being used, it had nothing to do with the marketplace itself. So therefore, there's really probably no reporting mechanism to a regulator. So the regulators are left figuring out what went wrong.
Meanwhile, all of Bay Street is frozen. And the screaming starts on every trading desk, including ours. What's going on here? Do you think, on a post mortem basis, that the vendors like yourself, have looked at this and thought, we need to have a better solution. And maybe Stephen, you can dig in a little bit on the fact that when there-- I don't think people fully understand what happens when the messages all come in really quickly to a router, and then all of a sudden data starts being dropped, a so-called drop packet. Can you help educate our audience on what that means?

STEPHEN PLUT: Yeah, because there's been a leaning towards the trading engines to be extraordinarily fast, these engines are actually a marvel of modern technology in terms of their latency, to the point where a trading engine is now responding in 20 to 30 microseconds from the order going in, which is astronomical. 20 years ago, you'd be happy to get 2 milliseconds. So it's like almost a three level order of magnitude improvement.
And what's happened is, is that because these exchanges are being judged by their latency, they really lean towards having very, very fast systems. And in some cases, that means that they're being forced to not have the resiliency inherent. One exchange in particular, if you blow through their queues, you will not be able to recover any lost messages.
And that's happened. It's happened to me and one of my large customers. We actually now have software or code in there to protect against that situation, where even though we lose the messages, we can continue processing. Because most of these protocols are designed so that they have to be handled in a sequential manner. We had to build a way around that because they were breaking it.
So it's just this general trend in the industry that we're tailoring our infrastructure and our industry to these low latency, high frequency trading. And unfortunately, that puts us a little bit more at risk. That's happened over the last, let's say, 10 or 15 years.

PETER HAYNES: And so, one of the-- Etienne, as we look a little bit more, dig a little bit more into this issue around tech and potential remedies and solutions for the industry, it was clear that one of the problems that we ran into was at the TSX. They were using one of the router services that was overwhelmed to calculate the MMBO for one of their marketplaces alpha. And when other marketplaces started to see cross bids and offers and other things, they declared self-help, and essentially were able to route away, even though alpha wasn't a protected market. They were just basically be taking alpha off of the routing tables.
So this was unfortunate for the TSX, that they were still using this router service. Because what I understand, is they were just in the process of transitioning to something from a software solution, which they were using to something known as an FPGA card, which is a hardware solution. It's something your company is in the business of building, this particular piece of hardware. Can you explain to our listeners what exactly is an FPGA card, and what solutions does it solve, and how does it reduce the risk that a future event, similar to what happened, will be less likely to happen for exchanges and brokers?

ETIENNE PHANEUF: Yeah, I'll tell you a little bit about FPGA, what it stands for and what it is in a second here. But to sort of directly and head on answer the question, the reason that they're irrelevant as a solution is that they handle capacity overall, in terms of message traffic, much more efficiently than software CPU systems. They don't get overwhelmed.
A CPU is a beautiful piece of technology. It does one thing at a time. So it stands to reason that the more you throw at it, eventually, it just gets a backlog. And that's when you start to see some of these delays in processing. FPGAs don't suffer from that because they're a different kind of semiconductor. They don't suffer from that particular problem.
And so, they're really useful across sort of three dimensions. One, just overall capacity, much bigger bandwidth in terms of being able to handle messages. Number two, they provide a deterministic outcome, which means that you get a consistent performance. So you don't suffer from these lags. And so if you're processing market data in 1 microsecond, then you're going to get 1 microsecond, whether tomorrow's twice as busy as today, which is not typically what happens with CPU software. You start to get performance that degrades.
Those are the two things that our clients most often look for in terms of the solution. But the third thing is just overall speed. They have been used in the financial industry for 15-plus years, typically by market-making firms, whose primary use case was to be able to receive the quotes, process the quotes as fast as possible, because they were in a race with the other market makers to be able to react and respond to those quotes. So it's not a new tech. It's been around for a while. But the use cases are starting to shift away from just I want to be as fast as possible, to I need to be able to handle this amount of message traffic that's coming, the bursts that come with it, and get a consistent level of performance.

PETER HAYNES: Stephen, when we talk about bursts, how much different is the message traffic that you see right around the FOMC decisions or a rate hike, whatever it might be, some news that happens in the market? This comes back to my original point about Jim Gallagher years ago is, how do we know how much capacity we need? And Stephen, in your mind, does the FPGA solution have an unlimited bandwidth to be able to handle these types of bursts?

STEPHEN PLUT: Yeah, well, I'm very familiar with the HN's technology. And in industry, it's considered wired speed, which means that it is capable of processing as quick as the network is able to put out. So yeah, it is capable of handling it.
But at some point in time, you always end up with a system that's farther down the FPGA that is going to be fed information from that FPGA. And sooner or later, you're going to have a system that is not going to be FPGA-based. It's going to need to be able to deal with the messages.
FPGA, as HN said, it was one of its first things that it was designed for, was the HFTs. And the HFTs have a very specific methodology on what they're trying to do. So if they see a quote change, they're racing to get their passive order on top of the book.
Well, if an FPGA system has an issue, that HFT that's behind that FPGA has no problem basically saying OK, I'm going to shut that down. I'm going to find my position. And then I can restart.
If I have the large retail order flow that has many days of multi-day orders, I don't have that luxury. That's the difference between the exchanges going towards the HFT-type flow, which can be shut down, restarted and they can start from a clean slate, as opposed to a retail order flow that you cannot do that. You have hundreds of thousands of clients out from Questtrade, TD Waterhouse, RBC, they have orders on the market. If all of a sudden, they come in the next day and they find their order is gone, they're not going to be happy.
So when we build systems for that retail order flow, it has to have a different level of resiliency to be able to protect those customers' order flows. We don't have the same capability of working with the HFT desk and just kind of restarting and going over. So HFT is fabulous. And just as a slight correction, it wasn't the routing plant that the TMX. It was the ticker plant.

PETER HAYNES: Sorry, yes, the ticker plant, just to use the proper terminology. So Stephen, on that note, you mentioned that the the exchanges are catering to the HFTs. And maybe you can just talk a little bit about some of the impact that the catering to low latency businesses has on natural investors. You mentioned the example of retail resiliency.
And I know we get this question all the time when there's an outage at the TMX. There's no failover mechanism. And one of the problems is we have good til cancel orders. We have order types on the TSX that aren't in line with order types and other marketplaces. So when you start to dig into all the reasons why it's virtually impossible to mirror the order books and map something over in the event the primary is down, but what are some of the other things that essentially, the exchange or marketplaces have sacrificed in terms of natural investors?
How are they being impacted by the fact that there's a race to this low latency? Is it all about stability? Or are there other issues that are impacting the institutional community?

STEPHEN PLUT: They try to do their best to-- know they still realize that the retail order flow is very important. And they do their best to do it. But there's been times where some exchanges, in order to have the best performance, won't allow a vendor to recover all the messages on the feed. So they'll say OK, you can only recover the last 500,000 messages, stuff like that.
And just to give you an idea, I was kind of looking at my notes back. And in 1998, the largest number of messages per day for the combined Canadian exchanges was approximately 80,000 messages. About a year and a half ago, when we were in the massive trading burst about a year and a half ago, I pulled up the stats. And it was about 1.8 billion messages.
So that gives you an idea of where we've gone in 25 years. It's an incredible increase. And given all that, we've probably not done such a bad job. It's just that at the end of the day, as you pointed out at the start of the conversation, is the stability of these systems is very, very important.
And there's things that you can do your best to improve the stability of the systems. And some things that they do actually take away from the stability of the system. And just also one correction. TSX has done a failover. They had a system outage a little while ago. And they did do a failover. And they were able to get their system back running in, I think it was about an hour or so.

PETER HAYNES: And it used to be a lot longer than that. If we had a failure like we had a couple of times at like 1:30, they're sitting there going--

STEPHEN PLUT: Be done for the day.

PETER HAYNES: And that causes a major problem because a market on close in particular.

STEPHEN PLUT: So the TSX is--

PETER HAYNES: I'll give them credit there.

STEPHEN PLUT: Absolutely. So we don't want to throw rocks at anyone. It's just that there's little things that have happened that if there was a more concerted effort and to realize that very few people ever sit down when they come up with a new product offering and they say, how is this going to affect the vendor? Or how is this going to affect the broker systems? And that's a question that really usually doesn't get answered. And I think that's one of the big things that we need to start approaching.

PETER HAYNES: OK, so let's dig in on that. Marketplaces are doing a technology change. Is there a consistent protocol for they go to the regulator, they say we're doing something. Is there a role the regulators should play that every technology change has to happen on a certain day each month, in order to allow organizations to align their resources around that particular day? is there a better way we can be doing this as an industry?

ETIENNE PHANEUF: I'm going to take that day off if that happens. I don't think that's a good solution.

STEPHEN PLUT: I thought you were going to jump in and give us the answer to the solution. It's like, oh, that'd be awesome. The answer is that it's a-- it's an area of expertise that most of the regulators are not staffed up for. You bring up some issues in terms of some of the design. And the regulators are fabulous at knowing the UMA rules, and all the rules of trading and all that, and IROC in terms of their trading--

PETER HAYNES: CIRO.

STEPHEN PLUT: Sorry, I just dated myself. But if you point out some technology issues, usually what you get back is they say it passed all of our criteria. We're not able to hold it back. And my point is that I think sometimes that criteria needs to be expanded. There was one case in which a major exchange did a complete trade changeover on their systems. And it was done on a Monday night. So if you have to think about that, there's probably 300 to 500 gateways into that trading engine.
And that night, 300 to 500 gateways had to be switched over. Whole groups of software had to be updated on dozens and dozens of client and vendor systems on a Monday night for a Tuesday open. It just so happened that that was the last day of the month. So you can see what was driving it. It wasn't the technology. Something else was driving it. It really put our backs to the wall as a vendor.
And our customers are getting pushed by their customers that they need to be trading that come Tuesday morning. We tried to do everything in our power to pre-stage the software, have everything pre-tested and pre-configured. And then just literally do a massive switch. But the level of risk on that is unimaginable.

PETER HAYNES: So, Etienne, that leads to a question, when we think about the role of the regulator, and I'm not blaming the regulator for this, but every time a new marketplace makes a change like that, every time a new marketplace introduces a new order type, every time a marketplace makes an engine upgrade, every time a new marketplace comes along and has some new level of complexity, whether it's a speed bump, or a new order type, or something, it adds to the risk that something's going to blow up.
And it's not intentional. It's everyone doing individual things to try to solve client needs. And we end up with something that, arguably, is a bit of a ticking time bomb. What role might the regulator play on limiting the next venue or next order type that might come along that could essentially tip this?

ETIENNE PHANEUF: It's a good question. I think that inherently through the process that the regulators apply to evaluate new marketplaces, new order types, and then ultimately approve them, put them out for public comment and so on, they are already limiting how quickly things can move.
Now, we can debate whether that delay should be longer. We can debate whether the implementation phase should be longer. But I can tell you from experience, again, leaning on my past-- my past role, that we were able to accomplish-- to roll out a lot less from a business perspective than we would have liked to because we had to fit into the regulatory process of first vetting the idea with them, walking them through, educating what we were trying to do, making sure that it was within the scope of the rules, putting it out for public comment, responding to the public comments, et cetera.
That's multi-month. So in terms of what we would have liked to have done as a business is probably roll out 10 things a year. And now whether that's practically feasible or not, to Steve's point, probably not. But that's what we would have liked to have done. But in fact, we knew that we could only do probably one major release per year and maybe one or two minor releases per year.
So there's already a speed bump in place that I don't think is really visible to two market participants unless you happen to be running a marketplace. And of course, we can debate whether the implementation periods should be longer. And I think in some cases, they probably should be. There should be, I think, some concept of this is a really big change that impacts everybody. And people need to have sufficient time to do it because everybody's not going to just drop whatever they're building at the moment to just implement the new-- the new change because it's come on board.
So there's probably a discussion to be had there around the implementation timeframes and whether that's sufficiently long. But I think the regulators already inject a speed bump, pardon the pun, into that process. And given that I think our ecosystem is vibrant and that, generally, we get good results for it, and that I'm a capitalist, and that I don't really want to see the regulators dictating what comes and what doesn't, I think we need to live with that as an industry to reap the benefits that come from innovation. Otherwise, I think we're going to stifle innovation.

PETER HAYNES: And Steven, talking about important upcoming regulatory implementations, one of the issues that people are starting to talk about that they're concerned in terms of a burst, we'll call it a message traffic, relates to the SEC's proposal to lower the minimum increment between the bid and ask of a liquid stock, which will happen in the United States and coincidentally happen here in Canada to some degree.
First of all, should regulators be thinking about the risk that they're introducing to capacity in their decision making with respect to these types of proposals? Because what I understand from what I've read, the SEC didn't mention that in their risk category in terms of when they were discussing lowering tick increments, that this could have an unintended consequence on message traffic. So clearly, it wasn't written as an issue. Should they be thinking about that? And do you have any sense of exactly how much traffic downside increase there'll be as a-- with respect to this particular change?

STEPHEN PLUT: So they're going from a penny or a half penny increment to a 10th of a penny increment?

PETER HAYNES: Well, they're proposing going from a penny to a 10th of a cent, or 2/10 of a cent, which the industry has really pushed back on. And the expectation from the industry is that we'll end up with about 1,000 to 2,000 stocks in the US that move to a half cent increment from the current one cent. And so obviously, narrower increments, more pricing changes. I'm curious if you think that that's a risk.

STEPHEN PLUT: Whether it's a risk or not, will it increase the message volume? And the answer is yes. It has no choice but to because if you have more tick increments and HFTs were always going to be reacting any time there's an NBBO change or a tick change, they're going to be following the tick. So it will increase the message rates. And it will-- given the fact that a lot of our message traffic is based on HFT flow, it's not--

PETER HAYNES: In those most liquid names--

STEPHEN PLUT: In those most liquid-- absolutely. So it will definitely increase the message rates.

ETIENNE PHANEUF: Let me just jump in here. Informally in discussions I've had with a variety of participants in the US market, both marketplaces and sell side, the estimates I've heard range from like 1 and 1/2 times current sort of baseline message traffic to 10 times. And there's probably a bunch of folks sort of in the middle, around four or five.

STEPHEN PLUT: I think all of those are reasonable numbers. It will probably be somewhere at the lower end of it. But a two times increment in message rates is not a trivial thing.

ETIENNE PHANEUF: It's not a trivial thing because the ecosystem technology is sort of built to handle a current baseline. Now, that said, does the technology exist to handle this? Absolutely, and you need to only look at what's happening in the US options market, which is order of magnitude, perhaps even more message traffic than US equities. So they handle that. Can US equities handle 10x more message traffic? Absolutely, they can. Now, I wonder what happens to options message traffic which tends to get multiplied.

PETER HAYNES: I moderated an options panel and asked that exact question. And they're like, no, no issue. We can handle-- we handle a billion messages every minute already. And they feel like they have the capacity through [INAUDIBLE] to be able to handle that message.

STEPHEN PLUT: But the US market's quite a bit different than Canada when it comes to the data. It's very rare that a trader in the United States has a level two market display. It's almost unheard of. Whereas in Canada, it's a standard thing.

PETER HAYNES: Right, because the order of protection rule goes to full depth of book.

STEPHEN PLUT: For many different reasons, it's a lot of historical reasons and all that kind of stuff. So to say that the Americans can handle it. OK, except that we handle things in a different way. And therefore, we're more exposed than they are because they've built up their systems. And it's unheard of that a trader will have a level two market display. In Canada, it's almost unheard of a pro trader not having a level two market display.
So not having a level two market display cuts away your message rates by 10 to 15 times. It's just how it is. So it's not necessarily a 1 to 1. And to say immediately that because the Americans can handle it, we can handle it, our infrastructure is a little different. I wouldn't just say if we changed our tick increment that everything would just be hunky dory. You need to find out.

ETIENNE PHANEUF: It's going to require a technology investment. Whether it's the entire industry or some subset of it, there will be market participants who are not equipped to handle it today. And they will need to upgrade their infrastructure.

STEPHEN PLUT: And [INAUDIBLE] the whole FPGA stuff came because he had customers that were coming out. And in fact, you were a customer of the original FPGA stuff. And it was a very, very good way of handling. It's a fabulous piece of technology. Unfortunately, there's always something that it can't handle.

PETER HAYNES: OK, so the lowest common denominator vendors that are unable to handle message traffic bursts and become, in some ways, a systemic risk to the market, what can regulators do, especially when we're talking about routing firms that are not within their purview? US has something called REG-SEI. And they kind of try to umbrella as many people they think can be systemic in the market. Is there something Canadian regulators should be doing to consider that fact?

ETIENNE PHANEUF: I think that they're not regulating those vendor entities.

PETER HAYNES: But they're systemic. And we clearly saw that on in February with tech.

ETIENNE PHANEUF: So I think that what they can do is go to the people that they do regulate, the marketplaces and the brokers, and hold their feet to the fire. And then it becomes a commercial--

STEPHEN PLUT: That's how my feet are held to the fire. My customers call me up, the regulators call me up. And even though they don't directly regulate me, it's very easy for them to go to my customers and say this is what we need to do. At which point, 20 minutes later, I get a call. And I sit down. And I said, OK, how are we going to solve this issue that you're having?
So we're not directly regulated. But we're regulated through an intermediary which are regulated entities, i.e. the broker dealers. They have incredible sway over me as a vendor. That outage that we had the six months ago, two of the organizations, two large vendors organizations, they were down for about 30 minutes. And we were down for about five. By having multiple vendors, they kind of spread the risk.

ETIENNE PHANEUF: And I mean, isn't this what should happen? In some sense, the better vendors should win, right? I don't think this is an uneven playing field where every vendor should be able to produce the same level of results, right? So you handle that better, that should be a feather in your cap. People ought to say maybe I should be with Steve.

PETER HAYNES: And it's such a difficult task. And I was talking to one person on the buy side who had made a decision that they wanted to stop working with a particular brokerage firm that offered a vendor service for various reasons. And they went to look at how they could just, essentially, I don't want to use-- we use the word cut off that particular broker. And they couldn't because their entire infrastructure was built around this particular vendor service.
So I think the same thing with respect to that issue in February, when all the brokers-- or all the sell side block desks are all down because they're all using the same router service. Then they go and say, OK, we need to change. And then they realize that's going to involve 462 technology people and 8,000,000,000 hours. And they just move on to the next thing.

STEPHEN PLUT: As Etienne was trying to say, or he mentioned, is that we are really the capitalist market and the capital system, we're held accountable by our customers. It's incredibly demanding. And our customers are demanding of the vendors because my customers are held by very high demands by their customers, the retail trading public, the large buy side firms, and all that.
And so there's many times where I would love to be able to do something, but my customers tell me that no, Steve, we need you to do this. And I do it. And so we're held in check. When things get out of hand is when an organization has a monopoly power or something along those lines, in which you have no choice. And then they can just do whatever they want. Realistically, we actually are pretty good in terms of no vendor has so much power that they can hold people hostage. It doesn't work too badly.

PETER HAYNES: So, Etienne, I've heard you use an expression before not all messages are created equal when in terms of impact on message capacity. What do you mean by that?

ETIENNE PHANEUF: Well, that message traffic layer is a few different things. It's market data that's being published by the marketplaces that is then being received by vendors and by brokers. So that's the view of the market. The other end of the spectrum is the orders that are going out from a broker into a marketplace.
So there's an order message that gets routed from the broker to the marketplace. And then there's an acknowledgment that comes back. And then there are fill messages, partial and complete fills, and cancels, and so on. So that's the other end. Now, these things are tied together because if I route an order to the exchange, it generates a market data message on the other side. So there's a bit of a circle here.
But I think those are fundamentally different types of messages. And typically, the types of messages that cause problems for vendors and broker dealers to be dealt with is more so on the market data side.

STEPHEN PLUT: It's not within our control. It's a third party organization that's creating a message storm that we don't control. And therefore, it really can get us blindsided.

ETIENNE PHANEUF: So you have to be you have to have the capacity in your system to be able to handle it. You have to be able to handle the burst. You have this sort of theoretical upper limit of, well, how many messages can I expect? Every time we have a crisis of some nature, whether it's a financial crisis, or a geopolitical, or whatever the case may be that generates a lot of activity, we burst to this new level, this new high of message traffic.
But we never know what that high is actually going to be. And so that's when the system actually gets stress tested in real life. And so that's a hard problem to solve.

PETER HAYNES: I don't think anybody in the regulatory world wants to hear that we're stress testing things in real life like that. They want those solutions to have been built in advance.

STEPHEN PLUT: It's the truth. What Etienne is saying is absolutely the truth. There has never been a massive message rate stress test. They do testing in terms of connectivity, and application outages, and all that. But it's never been a load test.

PETER HAYNES: How quickly can a vendor increase its capacity to deal with, for instance, the issue that happened in February? The vendors that were down for a half an hour, like you said you were down for five minutes. And you built solutions to hopefully avoid that in the future. But how quickly can you react to those types of things?

ETIENNE PHANEUF: Depends, I think not in real time is the answer. Can you then afterwards do things and say here's the specific circumstances that we observed, and now we can do these four things to mitigate that scenario? I think the industry is actually very good at not getting bitten twice by the same thing. This situation caused this outage. Great, now we won't get bit by that again. The next time there's a there's a problem, it's usually something-- usually not the same thing. So the answer to your question of like how quickly, I think it depends from vendor to vendor.

STEPHEN PLUT: If it's a matter of spinning up a new server or two, you could probably do that in a week or two. If it's a matter of an architectural change that your application just cannot go any faster than this, and you need to move over to an FPGA environment, you need to change your database structure or something like that, these are 6 to 12 month cycles. So it could either be something that can get spun up in two or three days or 6 to 12 months.

PETER HAYNES: I just want to finish on the topic of messages and ask you, Steven, in your experience, how often do you see message bursts that don't make sense and may be nefarious? I know we have the throttling on the way in. Not all marketplaces throttle too, I understand. So how often do you see what you look at as potentially nefarious activity? And if you see that, do you-- does that get reported to regulators?

STEPHEN PLUT: The regulators have track of all that stuff. As a broker dealer, some of the reports that a broker dealer needs to produce is message rates and things like that. So it's very hard to hide it because everyone sees it. So that's very little nefarious situations like that.
There used to be-- there was cases down in the States where people were stuffing the feeds because they were trying to overload feed handlers because it gave them an advantage to get their order in.

PETER HAYNES: That's what I was getting at.

STEPHEN PLUT: I've never seen that in Canada. But what happens is, is that you'll have a-- you'll see just your servers just get pegged. And you have no idea why. And then you'd be searching Reuters or any of the news services. And you'll be like, oh, that's what happened. And you can see all these HFTs picked up on that. And they immediately jumped on it. That's more along the lines. It's not nefarious because it's hard to be-- it's hard to do nefarious stuff when your tracks are in the snow that everyone can see.

PETER HAYNES: The reality is the HFTs reprice stocks so quickly in the same way that we used to manually reprice stocks over a longer period of time. It's really hard for us to come to grips, being around for a long time, with how quickly repricing occurs now. And they don't always get it. But obviously, they're reacting to that news. You say, oh, there it is. That's why.
So Etienne, I'm just curious. This conversation has been primarily about Canada, as that's all of our primary focus and expertise. In your experience, are we much different than the US? Are we much different than what you might learn from marketplaces in the APAC or in Europe?

ETIENNE PHANEUF: I think that the underlying problem of being able to deal with the capacity side of message traffic, bursting nature of it, and to deliver consistent performance is the same across the board. The complexity of those geographies changes. Where you get into some markets where there's only one marketplaces. I've lost count of how many we have here, but seven or eight. I don't know what the number is--

PETER HAYNES: Too many for the size of our market.

STEPHEN PLUT: Trading venues is about 14. And there's probably about seven organizations, eight organizations behind them.

PETER HAYNES: That's a lot for a small market.

STEPHEN PLUT: Canada is considered to be one of the most complex environments in the world given our population.

ETIENNE PHANEUF: So I think we are different in that way for like adjusted for the size of our market. You could go into a market like Hong Kong where there's one marketplace. I think they have a fundamentally underlying same problem. But they don't have the complexity.
And I just add one other element to your question, this isn't really just about equities or geography. This is the same thing across any other asset class that's highly liquid and electronic. So it's the same underlying problem in FX, starting to be the same problem in rates, as that market's become more electronic and more liquid. It's a function of more machines talking to more machines in terms of transmitting orders and processing orders.

PETER HAYNES: So Etienne, I want to hit a topic that I know is of interest to you. And that has to do with cyber. Because when we think about the risks that might cause our systems to be down, cyber is obviously going to be a high profile one. The New Zealand exchange dealt with that a few years ago.
So I know that there was a roundtable that the OSC held. And it was a simulation of a cyber attack on a Canadian marketplace and how do we react. And it was interesting that one of the takeaways that came from that event, that it was really hard to understand who was in charge of making decisions during that outage.
But we really haven't heard much about this topic since then. And it leads one to wonder whether regulators are really doing enough to address cyber risk in equity markets. For instance, as you've pointed out to me, connections to marketplaces lack encryption. Is this one of the fault lines we should be concerned about? And is it simply a matter of sacrificing encryption for speed?

ETIENNE PHANEUF: Yeah, so I think what the world teaches us, if not every day, certainly every week, is that everybody should be concerned about cyber security and how systems are being affected or potentially affected. I mean, to me, the sort of big risk is like somebody suffers a cyber attack, and there's a catastrophic outage, and can't recover. And now you're down-- maybe the marketplace is down for five days.
And whether that's the primary listing exchange, or whether that's some of the market, that's a catastrophic type of event, I think maybe that's probably the lowest sort of risk event. I think that within the realm of cyber security, there's all kinds of other smaller things that are maybe less visible to the general participants that come into play.
For example, I transmit my orders over a dedicated line to a marketplace. And what if somebody is now got into my system and is able to see the orders that are being transmitted over there and see back to who the clients are? What's the value of that. Well, I can tell you from my days on the brokerage side that our buy side clients would be apoplectic if they thought their information was compromised.

PETER HAYNES: And that's exactly why they went apoplectic when there was some socialization of some of the products that DTCC was selling in the industry, just very basic information gathering, which people could reverse engineer who may be the actual participants.

ETIENNE PHANEUF: Exactly, so the idea that the trading system is a closed one, which is that there are central marketplaces. And then there are point-to-point connections. I connect to one marketplace, to marketplace A, I connect to marketplace B. And I have control of that endpoint. And the marketplace has control that endpoint.
And therefore, I don't need to maybe encrypt the message that goes between the two of them, sort of an interesting one. But I wonder, as these cyber attacks get more and more sophisticated, whether, in fact, that traffic should be encrypted.

PETER HAYNES: So is that encryption issue, technically speaking, because those messages are being transmitted via microwave towers and areas that-- where it's flying through the space and it's difficult to encrypt? Or is that--

STEPHEN PLUT: All those microwave links are all encrypted. So on a physical level, they're all being encrypted. So you don't have to worry about that. I actually disagree with Etienne on this one. I normally agree with them on most things. But the environment as it sits right now, most people are co-located with the exchanges.
You have a fiber run that's in probably one of the most secure buildings. If anyone's ever been into a modern data center, these are probably as secure as anything that they've ever seen before in their life. And it's a point-to-point fiber. And these fiber runs are 200, 300 meters tops. And it's within the building.
Attacking it at that point is a low risk. And the exchanges data going out on the feeds, if it's containing private information, it is encrypted. Where you might be able to attack is between the data centers, all the vendors have multiple data centers. And you can attack the data between those data centers.
Then I would have to be encrypting all the data between my own elements and my own software. What I'm trying to get at is that there's other places-- it's like squishing a balloon. You squish it at one side, it's going to come out on other places. And so I don't really see how the encryption will really help it.
Typically, any of these attacks, they attack the server. Just so you're aware, as a vendor, the amount of things that I have to attest to to my large clients, it's massive. Things like is there any private information? Is it encrypted? Where is it stored? Is it stored under secure? All this kind of stuff. There's a lot of people thinking about a lot of this stuff.

PETER HAYNES: Unfortunately, the bad actors are also thinking about this too.

STEPHEN PLUT: Etienne's point about you attack one place, and they'll come out with another place. But the amount of effort that goes into protecting these systems from the broker dealer, from the exchange, from the vendor, it's incredible.
ETIENNE PHANEUF: I think the thing with risk is that it's multi-dimensional. I mean, there may be a very low probability that a given risk will take place. But if it happens, it's catastrophic. Versus a higher probability that something happens, but who cares, because nothing's compromised? So I think it as an industry, we need to continue to think about these things.
And I don't think everything is as secure as people would think. And I think the industry has relied on the fact that this is a closed system. That you'd have to gain access to Steve's environment in order to gain access to his connectivity. But I think these are the kinds of things that we see happening with large corporations who spend millions, if not hundreds of millions of dollars on cybersecurity every day all over the world.

STEPHEN PLUT: It's not a zero probability, that's for sure.

PETER HAYNES: Well, they spend a lot of money on marketing, too. My brother is the CEO or president of a cybersecurity company. And he pointed out to me at the F1 race last week, I think every single F1 driver has a cyber company on his outfit somewhere. So I found that kind of interesting.

STEPHEN PLUT: A lot of times, when budgets are getting cut, the guys that are doing cyber security are the ones that get cut the last. It's a good business to be in.

PETER HAYNES: As we finish up here, folks, I want to ask you a final question. I'll start with you, Steven. If you were the OSC or the SEC, what would you suggest should be the priorities to ensure stability of equity markets over, say, the next 10 years?

STEPHEN PLUT: It's an interesting question. I thought about it. And the one thing I would point out is that if you talk about the bank of Canada or The Fed in the United States, one of the key things that they always talk about when they make their decisions is stability of the financial industry. And I don't think, from a technology point of view, it's actually on the radar.
And so even if they just started to look at changes and say, what are the technology stability issues that this bring up? And have people on staff, have consultants on staff that can tell them that, OK, as a vendor, this is where this is scaring me. This is not an issue. That's an issue, that kind of stuff. I don't think it's really even on the radar.
The regulators are very focused on regulation, UMA rules, trading rules, fairness to clients, all that-- all the things that they've done for 100 years. And the industry has changed so massively. There was no quants 25 years ago. There was no algos. There was no HFTs. There was no FPGAs.
And so our industry has changed from being very high performance trader centric to being high performance technology centric on the trading side. The regulators, I don't think have matched that yet. Just to bring that into the conversation and to say, OK, how are we going to view this? I think that would be a major step forward.

PETER HAYNES: Etienne?

ETIENNE PHANEUF: Yeah, I'm not super fond, I guess, of pushing all of these things back to the regulators. I mean, there's certainly a role for them to play on different things. And I'd let them sort of set their own priorities. But I think for market participants in general, there ought to be a lot more focus spent on the message traffic layer, whether there's proper capacity to handle surges, to handle growth, that's 2x, 3x from where we are today, maybe more.
Whether the performance is yielding a deterministic outcome, and what that looks like. And I think that that's down to brokers, marketplaces, vendors. And I think as an industry, we can spend more time in that area recognizing that the marketplace has changed immensely, as Steven correctly has put it. And that there's an opportunity there, I think, to reimagine what that layer looks like.
And it's about resiliency. And by the way, I fully agree that-- the resiliency part of the ecosystem is the non-sexy part. Everyone wants to tell you about the new order type, the new algo they launched, or the new order type at the marketplace.

PETER HAYNES: Or the big trade that they just did.

ETIENNE PHANEUF: But in fact, if I drop packets, I need to know I need to be able to get them back. So I think that industry can spend more time making sure that we're properly resilient on that-- on that side of it. I'd say that that's a market participant responsibility.
STEPHEN PLUT: As a pet peeve, if you talk about stability, where do you get stability from? And one of them is by testing. I'll point out that when it comes to test environments, we don't do a great job across the board. The exchanges do not. And they don't synchronize their data. They don't synchronize how they use the data. They don't synchronize their symbols. And it makes it very difficult for people like me and Etienne also, is that one of the things that we need to do because these systems are so complex, is that we run test scripts. We run thousands and thousands of test scripts. And when one exchange uses production quotes in the UAT environment and one uses UAT quotes--

PETER HAYNES: What's the UAT?

STEPHEN PLUT: User Acceptance Testing. Not real quotes.

PETER HAYNES: US has like the 4Z quote or-- there's some quote.

[INTERPOSING VOICES]

STEPHEN PLUT: We have we have something similar to that in Canada. We have a YYZ but--

PETER HAYNES: That's our airport.

STEPHEN PLUT: Exactly, so you'll see yyz.abc. Those are test symbols. And it was a great idea. But I'm talking about a test environment where it's the true test environments. And you make life difficult for the people that are doing the testing. And if you make their lives difficult, then mistakes happen. And code gets involved and into production that should have been caught. And why? Because there wasn't a stabilized, uniform test environment from all the different systems.

PETER HAYNES: Well, I don't know if we solved any of the world's problems here today. But we certainly socialized a few of them and some risk. And I really appreciate you folks joining me today. I will mention that I did invite several different exchanges to participate in this discussion. And they all politely declined. So I'm really glad that you folks were able to come here and tell us a little bit more about your expertise that you've gained in the 25, 30, 35 years that each of us have been in this business. So on behalf of TD Cowan's Podcast, thanks for joining us.

ETIENNE PHANEUF: Thanks for inviting us.

This podcast should not be copied, distributed, published or reproduced, in whole or in part. The information contained in this recording was obtained from publicly available sources, has not been independently verified by TD Securities, may not be current, and TD Securities has no obligation to provide any updates or changes. All price references and market forecasts are as of the date of recording. The views and opinions expressed in this podcast are not necessarily those of TD Securities and may differ from the views and opinions of other departments or divisions of TD Securities and its affiliates. TD Securities is not providing any financial, economic, legal, accounting, or tax advice or recommendations in this podcast. The information contained in this podcast does not constitute investment advice or an offer to buy or sell securities or any other product and should not be relied upon to evaluate any potential transaction. Neither TD Securities nor any of its affiliates makes any representation or warranty, express or implied, as to the accuracy or completeness of the statements or any information contained in this podcast and any liability therefore (including in respect of direct, indirect or consequential loss or damage) is expressly disclaimed.