Blog > Market Report > Is It Possible to Eliminate Cu...

Blog Post

marktheshark [40]

You are not tracking marktheshark.

<< Previous Post

24-Apr-08

Is It Possible to Eliminate Curve-Fitting?

Curve-fitting (noun): the act of adding rules to the Key Idea of a trading system for the specific purpose of eliminating bad trades. Systems that have been ‘curve-fit’ never hold up in live trading.

I began to pose this question to myself this week as I surveyed the wreckage of yet another of my “Holy Grail” stock trading strategies: a Linear Regression-based algorithm discovered through my trading software, thoroughly back-tested, all bias removed, paper-traded for two weeks. Yes, if I had a dollar for every one of these systems I’ve invented over the past 4 years, I’d be a millionaire by now . . .

Reading a web article by noted trading strategist Henry Carstens titled “Introduction to Testing Trading Ideas” got me thinking about the whole notion of back-testing ideas in the first place. Carstens goes through the entire ABC’s of software testing: (1) Choose a testing platform, (2) Identify the Key Idea, (3) Run the test and interpret the results, (4) Pass/Fail the “consistency test”, (5) Test stops and additional entry rules, (6) Avoid pitfalls (slippage, commissions, fills, curve-fitting). As an article of its kind, I suppose it’s informative and helpful.

Carstens believes that his process, if followed carefully, can yield excellent stock trading strategies. The only place where Carstens thinks you have to be extra careful is Step #6: don’t fall prey to adding on rules to improve the results (i.e., curve-fitting). But here’s the problem I have with the entire back-testing procedure. It is all the way back at Step #3 where the curve-fitting initially occurs! Think about it. A system can’t possibly get a passing grade and go forward to implementation without bias occurring at this very point. Have you ever heard of someone getting bad results from back-testing and thinking “Yeah, that’s the strategy for me, let’s go live with this loser!” Of course not.

We humans are pre-programmed to be compelled by what works, not what loses. A backtest can only expose some successful pattern THAT OCCURRED IN THE PAST. Our brains will automatically avoid any pattern that is not successful. Oh, perhaps some of us are more honest with ourselves and will deliberately expand the universe of stocks to play in the future well beyond the select group that initially fit the back-tested pattern. That’s what Carstens means by passing the consistency test: “A system that has been consistent over time is far more likely to continue being successful than one that has a history of ups and downs.” Expanding our universe of stock plays will lower the system’s historical results but ostensibly iron out the hills and valleys and make the system more believable as one that can stand the test of time.

If only this were the case! A system deliberately made more consistent is still just another pattern that worked in the past. Back-testing a system can predict nothing about forward-testing (i.e., playing with real money) the same system, even if the pitfalls are avoided perfectly and the system is played out in real life as mechanically as the software program was able to do.

Trader software is overall a boon for the industry, and I still use my Amibroker on a daily basis. But I see more and more that the successful traders are ones who combine mechanical and discretionary approaches. They do not just let the software blindly lead them along, and they are no longer naïve enough to believe that past results are any indication of future returns. Technical traders should not be seduced into thinking that brilliant software technology can do the trading for them. Instead, they need to begin thinking “out of the box” – look through the successful patterns and play with the underlying algorithms in ways hitherto unknown.

Posted at 04:51 in Market Report | Permalink | Comments () | Top

6 Comments on "Is It Possible to Eliminate Cu..."

Hello Mark, I found these comments of your extremely important. There are a great many traders here that seem to focus on fundamentals, and that's fine, but relatively few who are not only systems oriented, but also setting their ideas down on system concepts and issues. Thanks a lot for these particular points. They are spot on! Don

Posted on 24-Apr-08 15:42 by Don_Bartell

Don,

Thanks. I'm so impatient to begin trading again, it's just killing me! Thought I would at least blog on some general topic while I'm paper-trading a new idea.

mark

Posted on 25-Apr-08 06:25 by marktheshark

Thanks, Mark. I have an interesting recollection of a severe curve fitting episode by a major firm in the 70's. Donchian's weekly rule was fairly popular, so they tested many variations of it on all commodities, and found--lo and behold!--the "best" one for each commodity. As I recall, for Copper, it was 6 weeks, and for Hogs, 4. They then got millions of dollars together in a commodity fund, and within about 6 months had wiped everyone out. I'm looking forward to any and all other blogs you may have on trading. Thanks again! Don

Posted on 25-Apr-08 06:32 by Don_Bartell

Perfect! I love this story. This is exactly the point. Donchian channel break-outs were what Richard Dennis taught the Turtle Traders to use. But it wasn't the actual rule or Key Idea that was important - it was how they managed the money, stops, exits. I think Dennis could have given them ANY indicator to determine the entry, and with his money management rules the turtles would have still made money.

I'm experimenting with a new idea and while I'm paper trading it (and I'll test it for at least another week), I plan to blog about certain elements in it. Stay tuned!

Posted on 25-Apr-08 08:50 by marktheshark

Hi Mark, I'll look forward to hearing about any and all of your new ideas. About that lengthy blog of mine that I mentioned, I've decided that it's too broad so am pulling it, but want to see what you think about adding volume and range to any automatic system. Here's the snippet on that. Thanks! Don

Looking back over the various concepts in trading, I’ve come to believe that there are some concepts, if not specific parameters, that remain valid, and are therefore immune to degrading of their usefulness. I think there’s fairly good evidence that high volume and range add to the validity of signals. So I pay a great deal of attention to those. In a review of some 1388 of my trades over the last 5 years, I’ve found that a minimum volume of 2 St Devs above the 50 day average makes for a significant improvement, and an ATR of 1 St Dev above its own 50 day average. So, unless I have that in place, it’s not a valid signal. (I do sometimes violate that, usually to my regret: VIVO was a recent example of a low volume signal that I took).

Posted on 28-Apr-08 06:19 by Don_Bartell

Hi Don,

Had a birthday party for my 8 year old this weekend, and my Russian-born wife put on the ritz (as Russians are wont to do with parties). So I had many, many duties, was pulled in all directions and didn't get a chance to really delve into your weekend blog. But I did sneak a few minutes to skim through it. As for the snippet that you have salvaged, yes I've built ATR and above-avg volume into many many of my trading algorithsm. The most recent one that dived checked for volume on the day >1.5X avg 22-day moving volume (I like to look at the 22-day avg vol rather than 50-day because the the longer look can smooth out the avg too much and mask what's been happening with volume recently). And I agree that the backtested results are improved when you put in these parameters.

Still, adding them to my formulas hasn't provided the breakthrough I've been seeking. Forward-testing is the only thing that matters - the only thing that will win the respect of other traders, and I haven't demonstrated that yet. I'm looking in some new directions and am so far excited by the test results. But at this point the new directions are only that - TESTS. I'll probably go live next Monday and then we'll see.

- Mark

Posted on 28-Apr-08 08:28 by marktheshark

You must be logged in to post a comment.

Top