I used to work in HFT. From about 1999 (right out of high school) until 2008.
In response to the OP, zedshaw and a couple of others (who make good points), I'd say:
* obviously if you're good at your job in HFT after a while you don't have to work 100 hour weeks. People wait for you. There are levels of support. You still get woken up once in a while, but not if the firm is well structured and the support training / delegation is good.
* people are right in that the closer you are to the money, the more financial upside (and downside) you usually have.
* there are some very mediocre programmers, like in any sector -- however, they tend to not last as long because the trading side is very demanding (in terms of quality) and that filters down pretty quickly.
* do the people who really know what they're doing in the trading and HFT space have an OCD level of awareness of all levels of their code? sometimes. and it's easy to then conclude that this must not be found in other fields/areas. i can't speak for the rest of the world, but there are exceptions everywhere i've been.
in fact, i wouldn't even say it's the norm that in HFT people are more capable of deep diving into assembly or whatever. however, basically there are a handful of people in HFT who have been fortunate enough to grow up in that industry and have made mistakes without being fired -- and those people are very bright and careful about their code, and the large purchasing price for quality/reliability and quickness delivered does affect things on a macro scale probably. there's also basically a lot of hard-core russian programmers (from the many different technical universities in russia) who are quite rigorous with their code and trust the idea of finance more than the idea of startups or silicon valley even -- but this is probably a generalization (just my experience perhaps).
most software rewriting is trivial once you know what you have to do. and most software projects don't 'know what they have to do' until half way through. i used to work with a guy who made a point of rewriting ALL trading-related code every two years. another guy didn't trust OSS because he thought it was mostly hobbyist. they're both sort of wrong -- but it's a different culture/mindset. and these guys weren't idiots. respectively, they were some of the lead developers / architects for some of the largest algorithmic shops in chicago.
but my point is that programming for HFT or real-time trading requires that you really know what you're doing down to each line of code, so that you can react when things do break (and they will break). if you can't react quickly, you will eventually not find yourself on interesting projects and you might even get fired.
so ironically you have to slow down and really get to know how to do programming very carefully, and then scale that up so that you can react really quickly later (with something like binary search). this is a useful thing to practice in some ways. other people probably learn variants of it in other fields.
* the recession hit large parts of the financial sector pretty hard. i went back and visited chicago somewhat recently (now work in SV). i can't speak to everyone, but if i were an undergraduate or someone trying to figure out career trajectories for the first time, i would feel much more secure in even the startup space than in financial services at this point. because you know most of that is going to be replaced (with automated, distributed technology) in our lifetime.
> so ironically you have to slow down and really get to know how to do programming very carefully, and then scale that up so that you can react really quickly later (with something like binary search). this is a useful thing to practice in some ways.
Any tips on how to practice this? I have been slowly discovering, on my own, ways to write code so that when bugs appear it is apparent which portion of code it appeared in. This can be as simple as logging often or as complicated as trying to predict when I might be doing something wrong and put a funky error message in a place I don't understand so when a bug does happen I can go to, or rule out, that section of code. Any tips?
Good, stringent mentors (esp., good, rigorous code reviews) are key.
Theoretically, depending on which language you choose, you should be able to rewrite a bit of code until there is literally not a single character that you could improve to make it clearer or more efficient.
It's good practice to do this -- eventually it becomes habitual. However, that doesn't save one from other people's code.
But yeah, there are certain virtuous cycles that work to your advantage if you take the extra time to try to improve even the most basic code (save one line here; make one idiom clearer, etc.) -- cascades over time...
My primary way of ascertaining how bug-free my code is, and the way I've never seen anyone mention, is by feel. Doesn't anyone else roughly know which code is likely to cause bugs and which isn't? Whenever I write some code without thinking it through or take my time, I know it's going to lead to bugs, even if it's very simple.
On the other hand, I have code I wrote that I trust completely. It turns out later that these metrics are, indeed, accurate. Does anyone else get this?
Had an uncle in the industry (with wall street experience who was (re)starting an algorithmic shop for the first time in the late 90s). Sometimes I think I would've saved more time if I had stayed in the Bay Area and not gone into HFT (in Chicago), but I definitely appreciate the rigorousness of some of the mentors I had there.
I didn't go right into writing backend matching engines for HFT though. Took time. Maybe not ten years, but maybe something like five years to come across good mentors that insisted I relearn everything (and get into the thick of HFT).
One mentor I had used to say he was going to crucify me at my desk, etc. (somewhat jokingly) in response to code reviews. It's a little bit annoying, but good practice.
I definitely still make mistakes (I am somewhat absent-minded). But I've learned to build protection around code, and to make checks and elegantly scale in things like test cases as unobtrusively and quickly (but as carefully) as possible. So maybe I make a bit fewer mistakes than some of my colleagues without that background.
Pretty much all custom. On the trading client side, a lot of firms used to tie into TT (Trading Technologies), etc. Then usually they rewrite with their own order gateways / price servers, and if they have enough resources, develop their own trading clients, etc.
I've been at firms that are all Java and use command-line terminals for most of the trading feedback (while having other clients to view the market depth). I've been at firms that are all C++ and C, and have their own trading GUIs.
Generally speaking though performance is key and at some point one ends up just rewriting things in-house (and tying into the exchanges or markets directly).
But there are a lot of different types of trades, etc. A lot of different ways to go, some of which (as zedshaw was mentioning) might involve Visual Basic.
But yeah, most of my experience was with C + a very limited subset of C++ on Linux. I still probably can't say too much more than that (though it's been two-three years, etc., since I was active with it).
I can probably say though that the fun for me was rewriting the backend services and trying to make them as fast (and as stable and redundant) as possible. A lot of it is translation of different market sources, etc. I experimented with a lot of languages at the time (Haskell, OCAML, etc., a la Jane Street), but at the end of the day did most of our stuff in C on Linux (i.e., with vim and makefiles).
If I'd make one bit more of advice (and I advised/mentored maybe two or three younger developers while in Chicago -- so I'm no expert) -- but it would be that if you have the interest, a lot of this stuff gets really easy with time. I suppose that's like most fields. But yeah, the productivity I had when I started, and when I left, was very, very different. And part of that had to do with writing everything 'bespoke' / custom, when there was time. Anyway, I go on...
Combination of things. I'm originally from the Bay Area (Berkeley), so I always wanted to get back (I think I could've gotten into the Chicago winters if I'd picked up following hockey more, but was sort of too late by the time I realized this). The other thing is that pure technology companies really are a bit more exciting in some ways than financial firms, even if they make technology and algorithmic approaches to problem solving their primary focus. Long-term, the Bay Area seemed to make more sense. I sort of believe in the startup mentality more than the Chicago-, zero-sum trading world mentality (though as sort of hinting at, sometimes I miss a little bit of the rigorousness). But it's hard to generalize. Good people in both places.
In response to the OP, zedshaw and a couple of others (who make good points), I'd say:
* obviously if you're good at your job in HFT after a while you don't have to work 100 hour weeks. People wait for you. There are levels of support. You still get woken up once in a while, but not if the firm is well structured and the support training / delegation is good.
* people are right in that the closer you are to the money, the more financial upside (and downside) you usually have.
* there are some very mediocre programmers, like in any sector -- however, they tend to not last as long because the trading side is very demanding (in terms of quality) and that filters down pretty quickly.
* do the people who really know what they're doing in the trading and HFT space have an OCD level of awareness of all levels of their code? sometimes. and it's easy to then conclude that this must not be found in other fields/areas. i can't speak for the rest of the world, but there are exceptions everywhere i've been.
in fact, i wouldn't even say it's the norm that in HFT people are more capable of deep diving into assembly or whatever. however, basically there are a handful of people in HFT who have been fortunate enough to grow up in that industry and have made mistakes without being fired -- and those people are very bright and careful about their code, and the large purchasing price for quality/reliability and quickness delivered does affect things on a macro scale probably. there's also basically a lot of hard-core russian programmers (from the many different technical universities in russia) who are quite rigorous with their code and trust the idea of finance more than the idea of startups or silicon valley even -- but this is probably a generalization (just my experience perhaps).
most software rewriting is trivial once you know what you have to do. and most software projects don't 'know what they have to do' until half way through. i used to work with a guy who made a point of rewriting ALL trading-related code every two years. another guy didn't trust OSS because he thought it was mostly hobbyist. they're both sort of wrong -- but it's a different culture/mindset. and these guys weren't idiots. respectively, they were some of the lead developers / architects for some of the largest algorithmic shops in chicago.
but my point is that programming for HFT or real-time trading requires that you really know what you're doing down to each line of code, so that you can react when things do break (and they will break). if you can't react quickly, you will eventually not find yourself on interesting projects and you might even get fired.
so ironically you have to slow down and really get to know how to do programming very carefully, and then scale that up so that you can react really quickly later (with something like binary search). this is a useful thing to practice in some ways. other people probably learn variants of it in other fields.
* the recession hit large parts of the financial sector pretty hard. i went back and visited chicago somewhat recently (now work in SV). i can't speak to everyone, but if i were an undergraduate or someone trying to figure out career trajectories for the first time, i would feel much more secure in even the startup space than in financial services at this point. because you know most of that is going to be replaced (with automated, distributed technology) in our lifetime.