{"id":953,"date":"2007-08-13T05:43:54","date_gmt":"2007-08-13T05:43:54","guid":{"rendered":"http:\/\/www.amibroker.org\/userkb\/2007\/08\/13\/4-io-robustness-a-sensitive-subject\/"},"modified":"2012-08-17T10:37:21","modified_gmt":"2012-08-17T10:37:21","slug":"4-io-robustness-a-sensitive-subject","status":"publish","type":"post","link":"http:\/\/www.amibroker.org\/editable_userkb\/2007\/08\/13\/4-io-robustness-a-sensitive-subject\/","title":{"rendered":"IO – Robustness, A Sensitive Subject"},"content":{"rendered":"

This page is obsolete<\/font><\/p>\n

<\/a>Almost all\u00a0who have been trading for more than a short while have come to realize that without additional information, In Sample Optimization results are purely for bragging rights and as such have very little predictive capability for how some system is likely to perform where it counts … Out of Sample.\u00a0<\/p>\n

One of the important pieces of information we can utilize to have some clue as to whether or not a system is likely to perform\u00a0well out of sample\u00a0is to take a look at how sensitive the parameter values we have chosen are.\u00a0 With a two parameter system we can in AmiBroker\u00a0optimize the system using traditional methods and then look at the 3d surface area plots that put the two parameters on the x and y axis and some performance metric on the z axis like in the chart below.<\/p>\n

\"io-3d.png\"<\/a>
\n<\/a><\/p>\n

As in the chart above\u00a0it is not uncommon for the highest peak\u00a0to be immediately next to an area where system performance falls off significantly.\u00a0\u00a0The parameter values representing this peak then\u00a0could be referred to as being too sensitive or not particularly robust.\u00a0 While this might be a very good system we would not want to use the parameter values that put us right at that peak as the probability of failure or at least significantly different results in real trading\u00a0is too great.\u00a0 We would want instead to select parameter values that while still performing well In Sample also had a higher probability of\u00a0performing well Out of Sample because they weren’t as sensitive.\u00a0 This could\u00a0be illustrated by\u00a0where I’ve placed the arrow\u00a0in the above chart.<\/p>\n

While the 3d surface area plots in AmiBroker are fine for visualization of Sensitivity and then to at least some degree\u00a0Robustness with 2 parameter systems, they won’t help when one is trying to understand the Sensitivity of parameter values with systems that have 3 or more parameters.\u00a0 One way to have some idea of how sensitive the parameter values are with systems that have 3 or more parameters is to take a statistically significant number of points and randomly generate values for each of the parameters that represent those points in some percentage range that is plus or\u00a0minus\u00a0from the original point, test those for fitness and then compare the results to the fitness of our original point.\u00a0 For example in the above chart let’s assume that the parameter values for L1 and L2 as represented by where I placed the arrow are at 75 and 70 respectively and we chose for our range to randomly test other points in the +\/- 5% range. We could then test points with values for L1 varying from ~71 – 79 and for L2 varying from ~66 – 74.\u00a0 This would give us data that could then be\u00a0plotted in a different way that showed us how sensitive those parameters are.\u00a0 Below is an example of such a plot using a bar chart to categorize groups of points and their fitness relative to the fitness of our original parameter values found in optimization.<\/p>\n

\"io-sensitivity.png\"<\/a><\/p>\n

<\/a>The top section of the chart shows categories and percentages of points tested and for example the tallest bar shows that 7.3% of the points tested had fitness that was 93% as good as our original point.\u00a0 Also notice that since the fitness of the original point we picked was not the highest peak in the original 3d surface area plot, that some bars in the\u00a0chart above have a higher than 100% value.\u00a0 The bottom section of the bar chart is composed of\u00a0cumulative values from the top section and for example shows that 45% of our tests were less than 93% as good as our original point.\u00a0 While this tool\u00a0may not appear to be quite as useful as the surface area plots, keep in mind that it is valid\u00a0regardless of the number of parameters being optimized.\u00a0<\/p>\n

The above\u00a0are standard shareware features in IO<\/em>.<\/p>\n

Given that unlike Exhaustive Search an Intelligent Optimization methodology will by its nature not examine every possible combination of parameter values, it would be unlikely without some additional influence or direction that the Intelligent Optimization process would have\u00a0picked for parameter values those that were not particularly sensitive.\u00a0 This is because the processes of judging or calculating parameter sensitivity are typically\u00a0performed after the optimization was finished because with Exhaustive Search that is all that is required as we had a chance to view the results of all combinations.<\/p>\n

In order to ensure that parameter sensitivity is taken into account when looking for parameter values with high fitness utilizing an Intelligent Optimizer, it is\u00a0necessary to have a methodology for evaluating how sensitive parameter values are and to have that in turn impact the fitness calculation during the optimization process<\/u> so that the process is\u00a0led to\u00a0a more robust set of parameter values.<\/p>\n

Given that the Intelligent Optimization process as implemented in IO<\/em> sends parameter values to AmiBroker for evaluation by its optimizer and retrieves results back from AmiBroker to determine how it should alter its search pattern in the next generation, this is more straight forward\u00a0then it would first appear.\u00a0 This is accomplished by between one generation of regular optimization and the next looking at the results coming back from AmiBroker and for those points that are worth further examination performing some tests for Sensitivity\u00a0that are not dissimilar to the methodologies used to generate the bar charts above. \u00a0The IO<\/em> options and mechanics\u00a0for this while not difficult for the user to employ are varied and fairly sophisticated and as such rather than discuss all of them here I would recommend for those who are interested that you read the sections on Sensitivity in the full documentation.<\/p>\n

The above\u00a0are advanced\u00a0features in IO<\/em>.<\/p>\n

A\u00a0shareware version of IO<\/em> with full documentation can be found in the AmiBroker Files Section …
\nhttp:\/\/groups.yahoo.com\/group\/amibroker\/files\/IO.zip<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

This page is obsolete Almost all\u00a0who have been trading for more than a short while have come to realize that without additional information, In Sample Optimization results are purely for bragging rights and as such have very little predictive capability for how some system is likely to perform where it counts … Out of Sample.\u00a0 […]<\/p>\n","protected":false},"author":14,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[115],"tags":[],"_links":{"self":[{"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/posts\/953"}],"collection":[{"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/users\/14"}],"replies":[{"embeddable":true,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/comments?post=953"}],"version-history":[{"count":1,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/posts\/953\/revisions"}],"predecessor-version":[{"id":3482,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/posts\/953\/revisions\/3482"}],"wp:attachment":[{"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/media?parent=953"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/categories?post=953"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.amibroker.org\/editable_userkb\/wp-json\/wp\/v2\/tags?post=953"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}