Posted By: Ritvars   Date: Thu May 22, 2008

Title: Curve fitting

Message:
  I am concerned by your frequent adjustment of software parameters, which looks suspiciously like curve fitting. It is very easy to come up with system parameters which show excellent results on back testing, but fall flat when applied to the real world. Do you employ any technique to avoid this trap? This also raises some questions about the validity of your historical results. Are these based on the actual system that was in place at that time, or on backtesting the latest version of your software?

Response:
  There are seven free parameters; five parameters control moving averages and two control the normalization of network input variables. These parameters have not changed in several years. It now looks like the two that control normalization will change slightly to better fit the predicted slope over the 6000 days of data used in training our model.

  We have made several model changes in the past two years. These changes are usually more involved than changing the parameters. Our research program continues to produce improvements that increase model return. We will continue to make these changes when necessary. The last 200 days of results that you see on our forecast page were not included in the training sessions so these results are totally blind predictions. Results before the last 200 days were included in the training sessions so these historical results reflect fitting within the constraints of the model. All results that you see were computed with the current forecast model.