{"id":952,"date":"2007-08-13T05:22:47","date_gmt":"2007-08-13T05:22:47","guid":{"rendered":"http:\/\/www.amibroker.org\/userkb\/2007\/08\/13\/5-io-out-of-sample-and-walk-forward-testing\/"},"modified":"2012-08-17T11:05:38","modified_gmt":"2012-08-17T11:05:38","slug":"5-io-out-of-sample-and-walk-forward-testing","status":"publish","type":"post","link":"http:\/\/www.amibroker.org\/editable_userkb\/2007\/08\/13\/5-io-out-of-sample-and-walk-forward-testing\/","title":{"rendered":"IO – Out of Sample and Walk Forward Testing"},"content":{"rendered":"

This page is obsolete. Current versions of AmiBroker feature built-in non-exhaustive, smart multithreaded optimizer and walk-forward engine.<\/font><\/b><\/p>\n

\n<\/a><\/p>\n

<\/a><\/a>
\n<\/a><\/a><\/p>\n

As a more thorough verification that a system will perform as anticipated, we should always test the system with out of sample data or in other words with data that has not been seen by the In Sample optimization process graphically\u00a0represented by:
\n\"io-oos.png\"<\/a><\/p>\n

This can\u00a0be accomplished in AmiBroker by:<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0 \u00a0Setting the from and to dates for our system to wherever we want<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0\u00a0 Performing an optimization and choosing the parameter values to use going forward<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0\u00a0 Changing the default values of the optimization statements<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0\u00a0 Moving the from and to dates\u00a0forward in time<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0 \u00a0Running a back test to see how well the system performs.<\/p>\n

Besides automating the whole process above,\u00a0IO<\/em> also offers\u00a0more advanced\u00a0alternatives\u00a0such as a fully automated\u00a0calendar or signal based, anchored or rolling Walk Forward optimization and Out of Sample testing which can be made to be very thorough.\u00a0 Graphically the anchored and rolling\u00a0walk forward processes look like this:<\/p>\n

\"io-awf.png\"<\/a>
\n\"io-rwf.png\"<\/a><\/p>\n

Without automated tools such as this\u00a0almost no one performs Walk Forward testing because\u00a0of the amount of manual intervention that is required.\u00a0 For example\u00a0think about the\u00a0manual steps\u00a0required to perform\u00a0a Walk Forward test over a 3 year period 3 months at a time\u00a0which are:<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0 \u00a0Set the from and to dates for the original optimization to begin as of some date in time and end as of 3 years ago<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0\u00a0 Perform the optimization and choose which parameter values to use going forward<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0\u00a0 Change the default values of the optimization statements<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0\u00a0 Move the from and to dates forward<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0\u00a0\u00a0Run a back test\u00a0for\u00a0the first three months of out of sample data and record the results<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0\u00a0 Then repeat the whole process\u00a0eleven times, each time moving the end date ( anchored ) or beginning and ending dates ( rolling ) 3 months closer until you run out of data.<\/p>\n

Assuming one had\u00a0the means to manually stitch together the out of sample equity curve\u00a0this\u00a0then would provide a real life picture of how the system performed over a 3 year Out of Sample period\u00a0with reoptimization occurring every 3 months.<\/p>\n

The above can be accomplished\u00a0in IO<\/em> with no manual intervention and a\u00a0single Walk Forward Directive which is written like this:<\/p>\n

\u00a0\u00a0\u00a0 –\u00a0\u00a0\u00a0 WFAuto: Anchored: 3: Months\u00a0<\/p>\n

As a result\u00a0even if it takes 15 minutes to optimize each of the 12 segments to accumulate the data necessary to build and show the\u00a0tables and the combined equity curves, it can all be done unattended.\u00a0 As a result one only need to set up a run, get it started and then go find something else of interest to do.\u00a0 Besides the tabular results that are produced by IO<\/em> it is also capable, with an included AFL, of showing an accurate composite\u00a0of the In and Out of Sample equity curve in AmiBroker\u00a0that looks like what is below:<\/p>\n

\"io-walk-forward.png\"<\/a><\/p>\n

The middle pane in the template above shows\u00a0my replacement for the standard AmiBroker equity curve for the current In Sample optimization.\u00a0 The bottom pane shows the full Walk Forward results\u00a0and is\u00a0constructed on the fly as\u00a0each new Walk Forward segment occurs.\u00a0 The section to the left of the thick vertical bar in the lower pane is the original In Sample optimization period.\u00a0 The sections to the right\u00a0separated by thinner vertical bars are each of the Out of Sample periods in the Walk Forward analysis.\u00a0<\/p>\n

IO is also capable, with a slightly different form of the directive, of performing signal based\u00a0Walk Forward processes which calls for reoptimization\u00a0every time\u00a0some user selected signal type ( Buy, Sell, Short, Cover, Entry, Exit, Any ) occurs.\u00a0 By definition this implies a variable length Out of Sample period that is dependent on when signals actually occur.\u00a0\u00a0<\/p>\n

These\u00a0are advanced\u00a0features in IO<\/em>.<\/p>\n

A\u00a0shareware version of IO<\/em> with full documentation can be found in the AmiBroker Files Section …
\nhttp:\/\/groups.yahoo.com\/group\/amibroker\/files\/IO.zip<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

This page is obsolete. Current versions of AmiBroker feature built-in non-exhaustive, smart multithreaded optimizer and walk-forward engine. As a more thorough verification that a system will perform as anticipated, we should always test the system with out of sample data or in other words with data that has not been seen by the In Sample […]<\/p>\n","protected":false},"author":14,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[115],"tags":[],"_links":{"self":[{"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/posts\/952"}],"collection":[{"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/users\/14"}],"replies":[{"embeddable":true,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/comments?post=952"}],"version-history":[{"count":2,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/posts\/952\/revisions"}],"predecessor-version":[{"id":3499,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/posts\/952\/revisions\/3499"}],"wp:attachment":[{"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/media?parent=952"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/categories?post=952"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/tags?post=952"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}