Users' Knowledge Base

Filed by brian_z at 11:54 pm under Ideas (Experimental)
Comments Off on Inverse Return To Mean System

Backfilled vs Real-Time Data issues

If you are not sure what HFAT (High Frequency Automated Trading) is all about, please Google the topic. This post highlights some of the problems you may encounter when venturing into HFAT of stocks. The views expressed here are based on personal experiences and/or may be anecdotal; not everything that happens in real-time trading is easy to explain. If you have technical insight and see inaccuracies, please comment for the benefit of future readers.

Designing and implementing high frequency trading systems is, from a trader’s viewpoint, probably the ultimate experience. To see and hear trades executed every few seconds and see the profits rolling in should give any trader an unprecedented high.

The catch is that to design an HFAT system that works with real money is very different from designing one using local data. The smaller the time frame the greater the impact of small data discrepancies. In sub-minute time frames, your HFAT system may perform very differently with local data than with real-time, raw market data.

A typical problem is that real-time live market data are delayed by up to several hundred milliseconds and that quotes may arrive out of sequence. What you see on your charts may be several quotes after the trade took place. This flawed data is what your trading system is trading and must be designed to work with. The charts you see in AmiBroker are mostly backfilled and/or updated after trading hours. At the end of a trading day, you will have data in your database that have a mixture of backfilled data (time and data errors have been corrected) and raw data (flawed) that were collected during the current day’s trading session. You may also have several lengthy data gaps that were introduced when you shut down the system and/or you lost your data feed.

While the procedure may vary for different data providers, quotes that are received in real time will lag in time. Since bar periods during live data collection are based on your computer clock, quotes may end up in the next bar due to their delayed arrival. The data used to backfill your database come from a different data server and will be time stamped. This allows AmiBroker to correct the position of quotes that were received out of sequence. This process removes the real time delays that were present when the data was received.

Since there are no delays with backfilled data, your backfilled data look ahead by several hundred milliseconds with respect to the data you will eventually be trading.

It is not unusual to develop a system with 5-second backfilled data (where all bad ticks and time-stamp errors have been corrected by the data provider) and obtain Holy Grail performance only to find out that when traded with real-time streaming data (where the data is delayed, contains bad ticks and time-stamp errors), the system is a total failure. The following charts illustrate this problem. The data to the left of the red line is backfilled and the data to the right of the Red line is data collected in real time. White is the equity.

You will not be able to visually judge whether data are backfilled or raw. The differences will only show up by running a trading system on the data; your trading system may be the only way to distinguish between backfilled and raw data. The chart below shows a close up of the data change.

Backfilling the above database and performing another Backtest over the same period produces a very different equity:

There is no guarantee that a system developed on one type of data will perform equally well with the other. When you first encounter a major equity drawdown, you may assume that this was just “a bad day”; after all, all trading systems have them. You may have developed and backtested your system over thousands of trades, covering a period of six months or more. You have been a good student and have used all the recommended methods to validate your trading system. You have tested in- and out-of-sample, applied intelligent optimizations, used Walk-Forward testing, performed Monte-Carlo analysis, and the list goes on. After being so thorough, how could you go wrong? You are ready to trade real money tomorrow and make your first 50% in one day!

The point is that all this effort is wasted time if the data used during development aren’t 100% identical to what you will be trading with.

The best way to develop an HFAT system is to use real live market data. The earlier you change from local or edemo data to real data, the more time you will save, and the more disappointments you will be spared. An HFAT system can never be completed off line, with a local database, or with simulated edemo Data. Its design must always include a significant paper trading and real-money phase.

Another problem when trading your IB paper-trading (simulated) account is that the user does not know the rules Interactive Brokers uses to decide whether an order should be executed or not. These execution criteria may change without warning. This imposes an artificial order to your executions that is unreal; the simulated market conditions will be different from those encountered in real trading. You may well develop a trading system that exploits IB’s way of processing to give you unreal performance, but such a system would fail in real trading.

Also, your paper-trades are not seen by, and cannot influence, the market. When trading real money your orders could be setting a new High or a Low, or if you are trading large sums, you could draw the price up or down. This means that even if your system performs extremely well in simulated trading, this is no guarantee that your system will perform well trading real money.

Using your simulated account to validate your system should never be your final validation before trading for profits; you should always include a real-money evaluation phase in your development plan. Your first real trades should never be to make money; they should be planned to validate your system under varied conditions.

Market behavior is very complex; be prepared for the unexpected and never skip a development step because something works extremely well. For example you might be testing your system using your IB simulated paper-trading account and see your profits skyrocket too fast to follow, perhaps having 90% winners and RARs that are out of this world. When this happens, it is extremely exciting and fun to watch; it is a rare experience that must be appreciated. It suggests that Holy-Grails are possible. But are they? Such favorable trading conditions may last for a few hundred trades, a few hours, or perhaps a few days. This can happen when technical conditions and market behavior are all just perfect for your system. Some unknown factor just made everything work perfectly. When you experience this, you’ll be analyzing your charts, trading log, execution report, etc. for weeks to follow. The fact is that it may never happen again, and you may never know what really happened.

Order and Position Status

IB Position size reporting may be erratic, is always delayed, and may include transient information. If you are trading fast and you use the IB Position Size to determine your next action, this will be a problem. This is especially the case with reversal systems where Covers may be processed before the Buys, and there may be many partial fills. For example, if you are reversing 100 shares, going alternatively Long and Short, you might read position sizes of 0, 100, 200, and even 300 shares. Do not base your system’s action solely on a single reading of the position size; your protective mechanisms will shut down your system many times a day. If a position is not what it is supposed to be on 5 consecutive queries (at quote interval), you may want to close all positions, suspend operation and continue later, or shut down the system and retry later.

Reporting of order status appears more reliable and stable. Usually it seems unnecessary to repeat Order Status queries.

IB Snapshots

Not addressed in this post is the matter of Snapshots however it is extremely important for real-time traders to understand how IB compresses and transmits its data. This topic has been discussed on several forums, for more information on IB data in general please read the following threads:

AmiBroker user group: Interactive Brokers Plug-in dropping volume data
IB’s Discussion Board: Globex Ticks snapshot or reality?
AmiBroker User Group: AB Tick Bar Analysis

The IB Maximum Message Rate

IB has a limit to the maximum rate of messages (order related) you can transmit per second. The rate of queries is not limited. The current limit is 50 messages per second. If you exceed this rate, IB will produce an error code, and if you continue to exceed the message rate, IB will suspend your connection. This, of course, should be prevented at all cost. The message rates are documented here. How to introduce real-time delays measuring in milliseconds is documented in the post on High Precision Interval and Delay Timing.

Internet Delays

Order and Position status is subject to a 50-400 milliseconds Internet delay. This delay will vary with your location and type of Internet connection to IB. You can test this delay by pinging the IB server. To do this type ping gw1.ibllc.com on command in the Start->Run windows (for Windows XP), and click the run button. A window, as shown below, will appear and show you the delays for three consecutive queries (pings) to IB:

If you encounter excessive delays or cannot connect at all, you can get more details about how your connection is routed by running tracert gw1.ibllc.com in the same manner. You may want to browse the Technical FAQ at IB for related items.

Edited by Al Venosa.

Filed by Herman at 8:55 am under Real-Time System Design
Comments Off on High-Frequency Automated Trading (HFAT); Part 1

An unsorted, unqualified list of FAQ’s relating to anything and everything to do with AmiQuote.

The list is dynamic and subject to change without notice.

Q1. Hi, my Amiquote crashes on Windows Vista. Sometimes after Amiquote downloads some tickers data it crashes with following message: “Quote MFC Application has stopped working”. Any suggestions?

A1. We actually tested it and AmiQuote only crashes on Vista when you are using cracked AmiBroker.
If you are using TRIAL, legal AmiBroker or REGISTERED, legal AmiBroker it works fine.

Tomasz Janeczko
amibroker.com

Filed by brian_z at 12:11 am under Data
Comments Off on AmiQuote FAQ’s

Filed by brian_z at 10:38 pm under Backtester
Comments Off on Backtesting Example – NewHighs/NewLows

Filed by brian_z at 9:43 pm under AFL - The Basics
Comments Off on Selected Date Range

Filed by brian_z at 11:13 am under Scans
Comments Off on New Highs/New Lows Indicator (v3)

Before continuing with this post you should carefully read the AmiBroker Help topic for the getPerformanceCounter().

Measuring time is an important aspect of all real-time intraday trading systems. Typical tasks requiring high-resolution timing include:

1. Limiting the message rate to Interactive Brokers (IB) to 50/sec (API Error 100).
2. Inserting small delays before polling IB Status, to allow for Internet delays.
3. Staggering (interlacing) portfolio trades to spread out the action.
4. Measuring and optimizing AFL execution time.
5. Modifying orders after a small delay (to ensure fills).
6. Periodic execution of tasks, for example, Watchlist scanning, Display and Status refresh, calculations based on slow changing variables, etc.
7. Time-Stamping events, for example order placement.
8. Collecting/preprocessing quotes.
9. Overlay live tick-charts on faster and easier to manage 1-minute charts.

Most of these tasks can be accomplished using just three custom timing functions:

1. GetElapsedTime(): A function that returns elapsed time since reset.
2. SetDelay(): A function that returns time left to reach a future time.
3. GetDelayTrigger(): A function that returns a trigger when a delay times out.

This post provides example functions in a demo application. To allow the use of many timers each function requires you to provide a TimerName, which will be used to retrieve timer information. Static variables are Global and can be read from anywhere; this means you have to be careful not to cross-reference the timers by using the same TimerName from different panes or windows. When running multiple copies of the same code you will need to key the TimerNames. For more on how to do this, see Keying Static Variables in the Real-Time AFL Programming category.

The timers below are implemented using the getPerformanceCounter(). This function returns the amount of time elapsed since the computer was last started. Tomasz recently explained this as follows: “The underlying high frequency counter runs all the time since computer start. What ‘reset’ flag really does is to store last value so next time you read it, it gets subtracted from last value giving you the difference. If reset is false, the last value is set to zero, and you get the original number of ‘clock ticks’ since computer start“.

The timers in this post do their own sampling of the underlying high frequency counter, and the getPerformanceCounter() Reset argument is always left set to False.

The getPerformanceCounter() returns values with microsecond resolution; however, the practical accuracy is severely limited by interruptions from the computer’s operating system. Do not expect much better than about 50-millisecond absolute accuracy. Aside from designing your own dedicated trading hardware (to replace the PC) there isn’t much that can be done about this. If you are brave, you can experiment with increasing program priority in your Task Manager window.

Chart refreshes are most often initiated by an arriving quote, but they can also be initiated by mouse clicks, tooltip, and various chart operations. This means that, when market activity is low and things are not happening as fast as you would like, you can force extra AFL executions by clicking on your chart. You can verify this by running the code below and, while clicking rapidly on the chart, observe that the timer counts displayed will update more rapidly.

You can ensure a one-second chart refresh by adding a RequestTimedRefresh(1) to your code. If the frequency of your arriving data is slow, your AFL code may execute only sporadically. Since your code must execute to read your timers, the resolution of your timers will be limited by the chart refresh rate. If your chart refreshes once a second your timing resolution will be one second!

Normally most of the AFL code in an Indicator window executes when your chart refreshes; however, to obtain speed advantages, you may execute non-critical sections (like account information and System Status) of your code less frequently using a timer. You can also execute small sections of code more frequently by placing them inside a well-controlled loop. If you do this, be sure to limit the maximum time your code can spend inside the loop to one second or less.

For fast trading systems, the frequency of AFL executions (chart refreshes) may be slow and this may make it difficult for you to get LMT fills. There is no way to have a program that requires 50 milliseconds per pass to execute 20 times per second.

Considering the interval between AFL executions, it is important to plan the layout of your code so that all events are handled in the most efficient order. If you don’t, transmittance of your order could well be delayed by up to a full second. There are situations where you want to invoke an immediate re-execution of your code. In some cases you might want to do this after placing an order to check order status before the next quote or refresh. Although it should be used sparingly this is possible by calling the RefreshAll():

<p>This function can only be called once a second; calling it faster will not result in more frequent chart refreshes. This means you should only call it when really needed. <p>The code presented below is for demonstration only. The getElapsedTime() lets you measure elapsed time from the moment of Reset. The first argument passes the name you assign to the static timer; this allows you to use the same function to time different events. The second argument is a Reset flag. When this Reset is True, the function samples the underlying high frequency counter and uses it for later reference. When you call the function with the Reset argument set to False, it calculates the elapsed time by subtracting the earlier sampled value from the current value of the Performance Counter. <p>The setDelay() function lets you Start, Read, and Cancel a time delay. The TimerName argument functions as in the getElapsedTime(). Calling the setDelay() with the mSecDelay argument set to a non-zero value will start the Delay timer. Calling it with the mSecDelay argument set to zero will make it return the current count-down time in millseconds. Calling the function with the cancel argument set to True will terminate the delay. <p>The getDelayTrigger() function returns a trigger. This is a signal that is true for only one pass through the code. Triggers are frequently used in real-time trading systems. They are needed to prevent multiple actions when a signal becomes True. <p>To run the code, copy the formula to an Indicator and click Insert. You'll see a chart window like Figure 1 below: 

<p align="center">Figure 1. Result from running the example code. 
<a href='http://www.amibroker.org/userkb/2007/11/10/high-precision-delay-and-interval-timing/timerdisplayjpg/' rel='attachment wp-att-1422' title='timerdisplay.jpg'><img src='http://www.amibroker.org/userkb/wp-content/uploads/2007/11/timerdisplay.jpg' alt='timerdisplay.jpg' /></a>

<p>The example code maintains three timers, T1, T2 and T3. All timing values are expressed in milliseconds. In Figure-1 the Elapsed Time shown is measured from timer Reset. The Delay shown is the time remaining after Start, until the delay times out. The line for Timer T2 shows that its Delay just timed-out and produced a trigger. Timer T3 still has a Delay in progress. Right-click on the chart to open the Param window: 

<p align="center">Figure 2. Param window. 
<a href='http://www.amibroker.org/userkb/2007/11/10/high-precision-delay-and-interval-timing/timerparampng/' rel='attachment wp-att-1423' title='timerparam.png'><img src='http://www.amibroker.org/userkb/wp-content/uploads/2007/11/timerparam.png' alt='timerparam.png' /></a>

<p>If you click one of the timer Resets in the Param window you'll see the ElapsedTime in the corresponding row go to zero, and then start to increment sporadically when your chart refreshes. Without live data this would be at approximately 1-second intervals, as determined by the RequestTimedRefresh(1); <p>If you click Start for one of the timers this will start a delay. You can see how it counts down in the Delay column. Click the timer's Cancel to terminate the Delay. Note that whenever a Delay times out, the word "Trigger" briefly appears in the third column. 


function RefreshAll()
    {
    oAB = CreateObject("Broker.Application");
    oAB.RefreshAll();
    }

function getElapsedTime( TimerName, Reset )
    {
    if( Reset ) 
        {
        TimeRef= GetPerformanceCounter(False);
        StaticVarSet(TimerName,TimeRef);
        }
    TimeRef = Nz(StaticVarGet(TimerName));
    ElapsedTime = GetPerformanceCounter(False) - TimeRef;
    return ElapsedTime;
    }

function setDelay( TimerName, MsecDelay, Cancel )
    {
    HRCounter= GetPerformanceCounter(False);
    if( Cancel ) 
        {
        StaticVarSet("TO"+TimerName,-1);
        StaticVarSet("DS"+TimerName, 0);
        }
    else if( MsecDelay ) 
        {
        StaticVarSet("TO"+TimerName,HRCounter+MsecDelay );
        }
    TimeOutTime = Nz(StaticVarGet("TO"+TimerName));
    DelayCount = Max(0, TimeOutTime - HRCounter );
    StaticVarSet("DC"+Timername, DelayCount);
    return DelayCount;
    }

function getDelayTrigger( TimerName )
    {
    DelayCount = Nz(StaticVarGet("DC"+TimerName));
    DelayState = DelayCount > 0;
    PrevDelayState = Nz(StaticVarGet("DS"+TimerName));
    StaticVarSet("DS"+TimerName, DelayState);
    return DelayState < PrevDelayState;
    }
 
RequestTimedRefresh( 1);

_SECTION_BEGIN("TIMER 1");
Reset1 = ParamTrigger("1 - Reset","RESET");
MSecDelay1 = Param("1 - Delay (mS)",1000,0,10000,10);
if( ParamTrigger("1 - Start", "START") ) setDelay( "Timer1", MSecDelay1, 0 );
if( ParamTrigger("1 - Cancel", "CANCEL") ) setDelay( "Timer1", MSecDelay1, 1 );
_SECTION_END();

_SECTION_BEGIN("TIMER 2");
Reset2 = ParamTrigger("2 - Reset","RESET");
MSecDelay2 = Param("2 - Delay (mS)",4000,0,10000,10);
if( ParamTrigger("2 - Start", "START") ) setDelay( "Timer2", MSecDelay2, 0 );
if( ParamTrigger("2 - Cancel", "CANCEL") ) setDelay( "Timer2", MSecDelay2, 1 );
_SECTION_END();

_SECTION_BEGIN("TIMER 3");
Reset3 = ParamTrigger("3 - Reset","RESET");
MSecDelay3 = Param("3 - Delay (mS)",8000,0,10000,10);
if( ParamTrigger("3 - Start", "START") ) setDelay( "Timer3", MSecDelay3, 0 );
if( ParamTrigger("3 - Cancel", "CANCEL") ) setDelay( "Timer3", MSecDelay3, 1 );
_SECTION_END();

ET1     = getElapsedTime( "Timer1", Reset1 );
ETA1     = setDelay( "Timer1", 0, 0 );
TT1     = getDelayTrigger( "Timer1");

ET2     = getElapsedTime( "Timer2", Reset2 );
ETA2     = setDelay( "Timer2", 0, 0 );
TT2     = getDelayTrigger( "Timer2");

ET3     = getElapsedTime( "Timer3", Reset3 );
ETA3     = setDelay( "Timer3", 0, 0 );
TT3     = getDelayTrigger( "Timer3");

Title = "\nFilename: "+Filename+"\n\n"+
"MilliSeconds Since Computer Startup: "+NumToStr(GetPerformanceCounter(False),1.3)+"\n\n"+
"Timer     ElapsedTime     Delay     Trigger\n\n"+
"T1      "+NumToStr(ET1,10.0)+"     "+NumToStr(ETA1,7.0)+"      "+WriteIf(TT1,"TRIGGER1","")+"\n\n"+
"T2      "+NumToStr(ET2,10.0)+"     "+NumToStr(ETA2,7.0)+"      "+WriteIf(TT2,"TRIGGER2","")+"\n\n"+

Edited by Al Venosa.

Filed by Herman at 9:09 pm under Real-Time AFL Programming
Comments Off on High-Precision Delay and Interval Timing

Filed by brian_z at 11:05 am under Database Management
Comments Off on Setup A Custom Database – Nasdaq

A list of unqualified data resources with a bias towards Real Time equity data (mainly for the American market). Some of the sites do provide data for other financial instruments as well.

The list is dynamic and subject to change without notice.

The list is ordered according to when the author researched the site and it is not sorted by rank or order of merit.

CAUTION – MOST OF THE LINKS CONTAIN COMMERCIAL REFERENCES OR LINK TO COMMERCIAL SITES.

NOTE: THE AUTHOR HAS NO COMMERCIAL AFFILIATIONS OF ANY KIND AND RECEIVES NO GRATUITIES, OR BENEFITS, FROM ANY COMMERCIAL ORGANIZATION OR INDIVIDUAL INVOLVED IN COMMERCIAL ACTIVITIES. NOR IS HE PERSONALLY ENGAGED IN ANY COMMERCIAL ACTIVITIES, RELATED TO TRADING, IN ANY WAY. THE LINKS ARE PROVIDED AS A RESOURCE AND FOR EDUCATIONAL PURPOSES. THEY ARE NOT A RECOMMENDATION, ON THE PART OF THE AUTHOR, NOR DO THEY CONSTITUTE INVESTMENT ADVICE AND SHOULD NOT BE CONSTRUED AS SUCH.

4) Paritech

http://www.paritech.com.au/default-au.htm

A Melbourne (Australian) based provider of Real Time market data for the Australian Stock Exchange.

 Additional Products:

Unlimited real time data for the Australian markets using Paritech’s Pulse software or DDE functionality – Real Time export function – export 50 time series live fields and events out of PULSE into your own application e.g. Excel and other compatible software.

http://www.paritech.com.au/support/pulse/features/dde.asp

3) TechnicalTrends

 http://www.technicaltrends.com/

Indian stock-market data – quality end of day and intra day data for stocks, futures & commodities traded on Indian exchanges (NSE, BSE, NCDEX and MCX). Extensive historical data is available. Intra day data (30 minute) is available for ALL stocks in the NSE.  End of day data is provided in Computrac format (now popular as Metastock format)  while intra day data is in text files easily accessed by all technical analysis software. Commodities data is provided for all contracts listed on the NCDEX & MCX.  Data is available in individual, as well as continuous contracts.

2) ViraTech

http://www.viratechindia.com/index.html

Indian stock-market data – intraday data for active cash stocks, as well as ALL current month Futures Stocks & Indices. Historical data / Continuity / Offline option – with a large intraday history, you have completely continuous authentic data for over 15000 5-minute bars.

1)  eSignal

A reliable vendor for intraday data feed in Amibroker format including International baskets e.g. futures of National Stock Exchange(NSE, India).

Filed by brian_z at 11:08 am under Data Resources
Comments Off on Data Resources – Real Time Equity

An unsorted, unqualified list of FAQ’s on anything and everything to do with Real Time equity data.

The list is dynamic and subject to change without notice.

CAUTION – SOME OF THE LINKS CONTAIN COMMERCIAL REFERENCES, OR LINK TO COMMERCIAL SITES.

NOTE: THE AUTHOR HAS NO COMMERCIAL AFFILIATIONS OF ANY KIND WITH, AND RECEIVES NO GRATUITIES OR BENEFITS FROM, ANY COMMERCIAL ORGANIZATION OR INDIVIDUAL INVOLVED IN COMMERCIAL ACTIVITIES. NOR IS HE PERSONALLY ENGAGED IN ANY COMMERCIAL UNDERTAKINGS, RELATED TO TRADING, IN ANY WAY.  THE LINKS ARE PROVIDED AS A RESOURCE AND FOR EDUCATIONAL PURPOSES ONLY. THEY ARE NOT A RECOMMENDATION, ON THE PART OF THE AUTHOR, NOR DO THEY CONSTITUTE INVESTMENT ADVICE AND SHOULD NOT BE CONSTRUED AS SUCH. 

To be commenced at a future date.

Filed by brian_z at 11:06 am under Data
Comments Off on Data FAQ’s – Real Time Equity