Feb 272011
 

This post will be the first in a new category called “Scams, Traps, & Pitfalls“.  I’ve been wanting to write about this for a while because the trading business is full of unscrupulous people who want to take your money.  And I’m not talking about the traders taking the opposite side of your trades, I’m talking about the vendors.

Tradestation is a company known for its programming capabilities and automation.  If you were to visit the Tradestation forums (which you can’t do unless you’re a customer) you’ll find lots of traders claiming profitability with their automated systems.  So many that you may even believe it’s possible.

In the beginning I spent over a year going down this route, and even after giving it up I still test ideas and prove to myself time and time again that it doesn’t work.  I’m not saying it’s impossible to have a profitable automated strategy.  I’m just going to say it’s beyond the reach of the typical small retail trader.  Most funds use trading models but there are teams of quants working on these models all day every day.  They fine-tune the models and adjust them as needed.  So it’s not a “set it and forget it” thing but rather continuous.  Most trade a basket of markets.

The other way that funds automate is with high frequency trading where trades happen in less than a millisecond.

So what does that mean for the small guy who wants to write an automated strategy for a single market like ES?  It means it’s practically impossible.  We don’t have a team of quants and the resources to trade a basket of markets, and we don’t have the infrastructure to trade at the millisecond level (a typical lag for me is 100ms).

Yet those in the trading business would rather you believe otherwise.  And quite honestly it’s a lot more fun to believe that it’s possible and we can be sitting back on the beach getting rich.  But it’s a pipe-dream.

I will add some of my personal experiences in a future post but what prompted me to write this post was a recent strategy from Tradestation Labs.  They develop these strategies to build up the hopes of their clients and generate more money in commissions.

The latest example is based on RSI.  Here are the results of their strategy with daily bars:

Here are my comments:

  1. The date range is quite extensive 1993 to 2011.  The market is constantly changing and it’d be, in my opinion, impossible to find a strategy that worked over an 18 year span.
  2. The profit factor is very high at 5.30 and is “too good to be true”.
  3. There are only 108 trades in the backtest period of 18 years.  That’s about one trade every 2 months.  This startegy is very selective.
  4. The average net profit is quite high at almost $1000/trade.

Overall to an inexperienced eye this could look like the holy grail.  And that’s when they tell you the results don’t use leverage, so the actual return would be many times higher.

At this point let’s ask an interesting question:  If this strategy works and can generate millions of dollars, why would Tradestation give it away for “free” to all its customers?  Wouldn’t they make more money trading it than they would in additional commissions generated by their customers trading this strategy?

I can only think of one answer to this, and it’s the answer I believed for a while when I first started.  The answer I used to believe is “because they have other better strategies to trade”.  But now I believe another answer: “Because it doesn’t work”.

Look at the hedge funds and fund managers who lose money.  The average fund actually does no better than the indexes (which is why many honest financial advisers recommend putting money into low-fee index funds).  So all these funds have to do to beat the market is run this free Tradestation strategy right?

This seems obvious to me now but when I really wanted to believe it was possible, it was very easy to delude myself into believing in these types of strategies.

So why doesn’t it work?  If it works in the past for 18 years, why wouldn’t it work for another year?

The reason is the number of parameters that were optimized in order to produce a positive result:

I count 11 variables that were optimized.  I’m not counting a few that look to be standard settings, but it’s possible those were optimized too.  11 variables is way too many.  With that many variables, I could make just about any strategy profitable.

So basically Tradestation has taken an idea based on RSI that probably had some big losers.  They added variables that ended up filtering out these losers and making the strategy profitable.  The problem is in the future the existing parameters probably won’t filter out the losers and may not catch the winners.  And this is why over-optimized (“curve fitted”) strategies don’t work going forward.

If you want to learn more about over-optimization, I highly recommend the book The Evaluation and Optimization of Trading Strategies.

But most vendors don’t care about all that.  They just want people to believe it’s possible and start trading it so that they can generate commissions.

So next time someone gives you a strategy, just keep in mind that if it worked and they could make millions with it, they probably wouldn’t be giving it to you for free or for a small fee.  It just doesn’t make sense.

My experience has taught me that context is the most important part of trading.  These automated strategies look for an entry & exit.  However if the context is wrong (trading against the current direction) it’s not going to work.  However if one is able to determine context, then the entry becomes much less important.  If the market is going up it doesn’t matter whether you enter on a MA crossover, stoch crossing above 20, a volume ladder setup, a HH on a minute bar, or anything else.  If you have the right context then those really don’t matter.  Context is everything and it can’t be determined mechanically.   However it’s very difficult to determine real-time and even if you do see it, it can change at any time.

  One Response to “An example of strategy over-optimization”

  1. Many years ago a certain vendor (I won’t mention any names, but it was Bruce Babcock) had a marketing gimmick called System-Of-The-Month. Every month a new trading system was rolled out that had been backtested and shown to be profitable on a variety of markets.

    Like a fool, I bought the entire collection and started programming them one by one into SystemWriter Plus (pre-TradeStation). It didn’t take long to realize they were all really the same system, a channel breakout system dressed up with slightly different entries and exits and a new name slapped on.

    You would think I would have learned my lesson from that experience, but, no, I went on to waste another twenty years doggedly trying to develop a viable trading system. It didn’t work. In my defense, I will point out that in those days there was no internet, and people got their information about trading mostly from vendors. Also, commissions were a lot higher, and realtime data hard to come by.

    Looking back I wish I had never heard of trading.

    Can’t win at it. Can’t let go of it.

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)