Users' Knowledge Base

When you are designing a real-time trading system, many things can go wrong. This post is intended to alert you to some of the potential pitfalls. However, that is all it can do. Only experience can teach you how to prevent them. Be aware that even the most experienced designers will make some of these mistakes repeatedly.

Since documenting all potential pitfalls with coding examples would consume too much time and space, they are, for now, only briefly commented on. Most of them will trigger a user response of “Oh yeah, that happened to me!”. If you need a more detailed explanation you can post questions in a comment to this post

No rules exist to prove that a trading system is free from coding or logical errors. However, two indicators are fairly reliable in suggesting you may have a problem:

1) Your profits are simply too good to be true. In this case you have no choice but to work through the code line by line, trying to find lines of code that look into the future. If that doesn’t reveal any errors, then you would have to inspect the plotted signals and trade list trade by trade.
2) Your system is very profitable trading Long but not Short, or Short buy not Long. When this happens, you may have an error in either the Long or Short parts of your code, and comparing the two sections will often reveal the problem (this only works for reversal systems). However, it could also be that your code is correct but that your trading principle is overly trend sensitive. This would almost certainly get you in trouble when the trend reverses. In this case no other cure exists than to re-think the basic system.

When designing high-frequency trading systems, i.e., those whose trade durations are in minutes, everything changes, and many traditional procedures fall apart. Internet delays, data delays, bad data (spikes), temporary system freezes (Windows sometimes has a mind of its own!), lagging status reports, TWS problems, etc., all become critical issues that will prevent you from obtaining a close match with the Backtester.

Many of these problems will only surface when you start trading real money. Hence, the final stages of developing a trading system should always involve trading real money. Here is where the Interactive Brokers account simulator (paper-trading account) may be an indispensable tool since you can test your system in real time without committing real dollars. But, since the market does not see your trades, even paper-trading results will differ from trading real money. In general, the faster you trade, the greater your real-trading results will deviate from your backtest results. You should also be aware that commissions play a much greater role on performance of high-frequency trading systems because trade profits are smaller.

No matter how you go about it, troubleshooting a complex trading system will almost always be a tedious and boring job that could keep you busy for several days or weeks. If you find that certain problems continue to resurface, they are likely related to your personal development style, and you may be able to write some code that checks for these specific problems. See the Debugging category for some ideas.

The list below, which is not exhaustive, is presented to caution you that many areas can lead to problems. Some are obvious, while others may be expanded on as needed and time allows.

– High/Low precedence (contrary to EOD where the Backtester is unable to determine which came first, the entry/exit or the high/low, in realtime there can be no ambiguity in price precedence).
– Data Delays (real-time data may be delayed for various reasons and time periods (Internet delays, lack of quotes, packets vs. ticks, etc.).
– Low Liquidity (there may be no-volume trading periods).
– Data Holes (bars with no trades).
– Data Spikes (high spikes without volume may trigger trades).
– Data Padding (a bar without data may be padded).
– Premature Padding (the last bar may be a padded bar).
– Data Accuracy (prices you receive aren’t always accurate).
– Random Slippage (you will rarely get the expected price).
– Breakout slippage (you will rarely get the Breakout price of your system).
– Survivorship Bias (companies that didn’t do well and stopped trading won’t be in your database, i.e., you are working above average stocks).
– Lucky Trades (a series of lucky trades may look like good performance).
– Parameter Over-Optimizing (optimized parameters are rarely stable over time).
– Design Over-Optimizing (frequent testing is like running an optimization and may be leading to false conclusions).
– Out–of-Bound Prices (with PriceBoundChecking turned ON, AmiBroker forces the trade price within the High-Low range, this may hide pricing errors).
– Price Rounding (prices may be rounded or truncated by the broker).
– Wrong Use of >= and <= (when using both <= and >= in the same statement, only the first equal condition will ever be seen).
– Comparing Floating Point Numbers (calculated values can have many decimal places, either round values or use the AlmostEqual()).
– Chart Justification (make sure you are looking at the Last bar!).
– System Mortality (no system will work forever).
– Sharing Trading Systems (sharing systems with other traders may result in over-trading a system).
– Being Duped by a Trend (a rallying ticker may make your system look like the HG (holy grail).
– Tricking AmiBroker (AmiBroker has its limits; it is possible to write esoteric code that will produce wrong results).
– Order Visibility (placing your order for every trader to see may influence the orders they place).
– Making the Market (extreme example: if you place a MKT order during a no-trading period you will change the chart).
– Window/Pane Execution Order (when passing variables between panes or windows do not assume that they execute in a fixed order, more).
– Trading at the Open (order execution at the start/end of day is different from midday because of volatility and data delays).
– IB Data Snap Shots (snapshots are only representative of prices traded).
– Trade Delays (make sure you understand your trade delays when backtesting).
– EOD and Intraday Gaps (There is no time interval in RT gaps).
– Time Zones (make sure your computer and database timezones are properly set).
– Very Short Time-Frames (prices jump and are less contiguous).
– Setting LMT Prices (consider rounding for faster order executions).
– 24-Hour vs. RTH (Regular Trading Hour) Backtesting (extended hours can rarely be traded like RTH due to huge bid/ask spreads and low volume).
– Static Variables Naming (use unique names for your static variables).
– Incorrect Computer Time (computer time offset from market time can cause real problems).
– Look-Ahead Problems (not all look-ahead coding problems are obvious).
– Buy/Sell Precedence in a Loop (be aware that AB and custom AFL loops enforce a Buy/Sell priority).
– RT Candle Discrepancies (RT Candles may be different from later backfills, especially in the opening print).
– Bars Loaded (consider bars-loaded with respect to execution speed and loops).
– Signal lifetime (signal strength quickly decays over bars in high frequency trading).
– SameBarExits (Sell signals may act as a qualifier for Buy signals).
– Designing systems based on High and Low triggers (these may fill in the Backtester but not in real trading). more…
– Using the wrong CommissionMode and/or CommissionAmount can make any system look good, or bad…
– Using zero TradeDelays is OK if you code the delays in your system’s code, else you may be looking into the future.

Edited by Al Venosa

Filed by Herman at 6:25 pm under Real-Time System Design
Comments Off on System-Design Pitfalls

In real-time trading many situations arise when you want to delay action until a specific criterion is met. In AmiBroker you can base your delays in many different ways, the only requirement being that the delay variable increments or decrements. If the selected variable doesn’t revert towards your timeout value (target), your delay function would never time out. In this case you would have to add code to handle that condition. A few variables you might use are:

– RT TimeNumber (Now(4)).
– Elapsed Seconds (redrawaction).
– Real-time Data time-stamp (TimeNum()).
– Tick-count (New Data).
– Volume (Change in Volume).
– Price change (Change in Price).
– Chart Refresh (any AFL Execution).
– Indicator values.

Which of the above variables you would use for your delay depends on the requirements of your trading system. There may be times when you may need to combine several methods to get the required results. For example, if a delay were based on the data-timestamp, it might not time out during a data dropout or a no-trading period. In this case you need to back up your data-timestamp delay with a real-time (seconds) delay.

Delays play a critical role in real-time system design. For example, in real-time systems, signals may have a short lifetime. The signal is strongest when it triggers and than quickly decays until, perhaps after a few bars, it has lost all significance. Letting the order fill at the time when it has lost significance is pure gambling. To prevent this, you can cancel the order after a delay, or decrease the position size proportional to the perceived decay in signal strength (perhaps based on elapsed bars?)

Since in a real-time system the time-lapse between AFL executions can be significant, you should place your LMT price calculations ahead of the ordering code. Calculating the LMT price after the order has been placed postpones order placement until the next AFL execution occurs, i.e., when the next quote arrives; by then, the price has probably changed. Especially during periods of low volume, this could be significant. When these delays would be insignificant in EOD systems, they could make-or-break your system in fast-trading systems.

To ensure frequent AFL execution in the absence of quotes, you can place a RequestTimedRefresh(1) statement at the top of your code, where the variable ‘1’ refers to a 1-second refresh. This guarantees an AFL execution at least once per second.

If your code is lengthy and takes a significant amount of time to execute, you may have to check order status at several places in your code. If changed status demands immediate action, you can force an immediate AFL execution by calling the following function:

Filed by Herman at 5:12 pm under Real-Time AFL Programming
Comments Off on Real-Time Delays

To run Trade-Automation code, you should have the latest software versions of the AmiBroker-Pro and IB-Controller beta and have the AmiBroker Interactive Brokers Data Plugin selected for data source.

To use the AmiBroker Interactive Brokers Controller, you need an Interactive Brokers account. To place simulated orders you can use the TWS-Demo (Login: eDemo, Password: demouser), or, better, you can open a paper-trading account (free). The IB paper trading account is indispensable for any serious testing of your AT systems. The eDemo is very slow and much less adequate than using the TWS paper trading account.

DebugView is an essential tool to trace program execution and create logging files. You may consider creating a custom item in your AmiBroker Tools menu.

To deal with irritating TWS pop-ups common with earlier TWS versions, you may consider a product like PTFB Pro V3.1.0.0 by Technology Lighthouse (just one of many such products – this is not a special endorsement). Lately pop-ups seem to interfere much less, so you may not need this product if you use the most up-to-date version of TWS.

You should consider joining the Amibroker-at forum for discussions about AmiBroker-based Auto-Trading applications. To read about auto-trading in general, you should visit the Elite-Trader and AmiBroker-AT forums. You should visit the Interactive Broker forums to read about the TWS API and other AT topics. Be sure to create an icon for the AmiBroker ReadMeAT.html on your desktop, this file is located in your AmiBroker folder (default location). You might also want to have a copy of the API List of Error messages on your desktop; you will be referring to them frequently.

Edited by Al Venosa

Filed by Herman at 8:43 pm under Real-Time Auto-Trading
Comments Off on Internet Resources for AT

As pointed out in a comment to this series, the Interpretation Window can be a valid alternative to using DebugView. A few points to consider when choosing either method are listed below.

DebugView

Debug information is routed to DebugView using this type of statement:

<p>DBVOn = ParamToggle("DebugView","OFF|ON",0); // Controls all output  <p>if( DBVOn ) _TRACE("# BuyPrice ="+NumToStr(BuyPrice,1.2));  

Any type of information that can be expressed in string format can be send to DebugView. A few DebugView features are:

– A stand-alone program
– Display accumulates multi-pass information
– Output includes timing information
– Output can be manually or auto saved to a file
– Optional scrolling display
– Copy and paste to/from clipboard
– History depth setting (#lines displayed)
– Filter/Highlight wrt keywords
– Always on top
– Find
– Software Clear display
– Low overhead (depending on the number of _TRACE() statements used)

Interpretation Window

Debug information is routed to the Interpretation Window using this type of statement:

<p>BuyPrice = LastValue(C);  <p>if(Status("Action") == actionCommentary)  
<p>{  
<p>// This code executes only when the Interpretation Window is open.  
<p>printf("Buy=%1.2f\n", BuyPrice);  
<p>}  

Some features offered by the Interpretation Window:

– The Interpretation Window is optimized for Chart-Commentaries
– Only displays info for current pass
– No history
– Copy and paste to clipboard
– An open Interpretation Window adds one full execution cycle to the code.

While not all pros and cons were covered it can be seen from the above listing that DebugView is designed for debugging while the Interpretation Window is not.

When running complex real-time trading systems the additional pass through the AFL code may prove to be too costly in terms of execution speed.

Further comments invited.

Filed by Herman at 3:41 pm under Debugging AFL
Comments Off on DebugView vs. Interpretation Window

When you are developing a Real-Time trading system, often the first problem you have to solve is to prevent repeat orders. Repeat orders can occur when a signal remains true over several bars or chart refreshes, and each new bar/refresh sends out a new order. This can result in unintentional pyramiding.

Another ordering problem can occur when your entry and exit prices are too close together with respect to price volatility. In this case, the price volatility can whipsaw you in and out of positions in rapid succession. When this happens, slippage and commissions will quickly erode any profits you may have had. Also, if your system cannot keep up with each signal, you may end up in an opposing trade.

A common mistake when attempting to prevent repeat orders is to wait for order confirmation from the TWS after each partial fill. The problem here is that confirmations from the TWS are always subject to significant delay, and they will often let several repeat orders slip through before the acknowledgment is received.

Another flawed method is to filter or smooth the price arrays. Although this may prevent repeat orders, the lag introduced by this technique will kill most systems.

There is no single best way to prevent repeat orders. Your solution will depend on your personal preferences and the principle of your trading system. Under certain conditions you may want to combine several of the methods shown below. The examples presented here are intended to make you aware of the problems involved and suggest some possible solutions. It is your responsibility to modify the code to suit your particular system’s requirements.

To test the demo codes below, you will need to have the TWS running and connected to the IB eDemo or your paper trading account. To keep the programs below as simple as possible, you have to reset the programs after changing the Transmit ParamToggle from Off to On.

Using OrderIDs to prevent repeat orders

When your program calls the PlaceOrder() or ModifyOrder() function, the IBc returns a unique OrderID. The OrderID uniquely identifies the order you placed so that you can refer to it later to modify, cancel, or check order status. The OrderID only acknowledges that the order was placed; it does not mean that the order was transmitted to the market or was filled.

If an OrderID has been received, this means that the order was placed. You can prevent order repeats by checking whether the OrderID has a value. Then, if you only place orders when the OrderID is empty, you cannot place repeat orders. The first example below lets you explore this procedure.

After an order has been filled or cancelled, you may eventually want to transmit a new order. In this case, you will have to define a rule for clearing the OrderID. This rule can be based on order status, a contrary signal, a time delay, the start of a new bar, or any number of things. Note that after you have cleared an OrderID, you can no longer modify or cancel the order it referred to. Hence, you should only clear an OrderID if you are sure that the OrderID is no longer needed, i.e., the order was confirmed cancelled, filled, or rejected.

The method below demonstrates how to use the OrderID, and because it still allows you to enter and exit (whipsaw) a position within milliseconds, it is only a first step to preventing repeats and whipsaws. To run this demo, Insert the code into a chart pane. When you open the Param window you will see these options:

The Chart Title shows IBc connection, OrderID, and Position status. Note that while Interactive Brokers calls all the messages ‘error’ messages, many are just harmless notifications. For more details on these error messages see API Error Messages.

When you click the Buy button, the Title will display the OrderID for the order placed, Order Status, and the position size if the order was filled. If you watch closely when you place an order, you may see order status change, for example from PreSubmitted to Filled:

In this demo you clear the OrderIDs manually. In a real system it would be cleared when your system is ready to allow the next order to go out.

RequestTimedRefresh( 1 ); 
ibc = GetTradingInterface("IB"); 
Transmit = ParamToggle("Transmit","OFF|ON",0); 
BuyOrderTrigger = ParamTrigger("Place Buy order","BUY"); 
SellOrderTrigger = ParamTrigger("Place Sell order","SELL"); 
Reset = ParamTrigger("Reset Program","RESET PROGRAM"); 
ClearBuyOrderID = ParamTrigger("Clear Buy","CLEAR"); 
ClearSellOrderID = ParamTrigger("Clear Sell","CLEAR"); 
BuyOrderID = StaticVarGetText("BuyOrderID"); 
SellOrderID = StaticVarGetText("SellOrderID"); 


BuyPending = ibc.IsOrderPending(BuyOrderID); 
SellPending = ibc.IsOrderPending(SellOrderID); 
if( BuyOrderTrigger AND BuyOrderID == "" ) 
{ 
BuyOrderID= ibc.PlaceOrder( Name(), "Buy", 100, "MKT", 0, 0, "Day", Transmit); 
StaticVarSetText("BuyOrderID",BuyOrderID); 
SetChartBkColor( colorBrightGreen ); 
} 
if( SellOrderTrigger AND SellORderID == "") 
{ 
SellORderID = ibc.PlaceOrder( Name(), "Sell", 100, "MKT", 0, 0, "Day", Transmit); 
StaticVarSetText("SellOrderID",SellOrderID); 
SetChartBkColor( colorRed ); 
} 
else if( Reset ) 
{ 
StaticVarSetText("BuyOrderID",""); 
if( BuyPending ) ibc.CancelOrder( BuyOrderID ); 
StaticVarSetText("SellOrderID",""); 
if( SellPending ) ibc.CancelOrder( SellOrderID ); 
ibc.CloseAllOpenPositions(); 
} 
else if( ClearBuyOrderID AND BuyOrderID != "" )  
{ 
StaticVarSetText("BuyOrderID",""); 
if( BuyPending ) ibc.CancelOrder( BuyOrderID ); 
} 
else if( ClearSellOrderID AND SellOrderID != "" )  
{ 
StaticVarSetText("SellOrderID",""); 
if( SellPending ) ibc.CancelOrder( SellOrderID ); 
} 
LastTWSMsg = ibc.getLastError( 0 ); 
Title = "\n"+ 
"Last TWS Error Msg: "+LastTWSMsg+"\n"+ 
" BuyOrder Status: "+WriteIf( BuyOrderID != "", BuyOrderID+" "+ibc.GetStatus( BuyORderID, True ),"")+"\n"+ 
" SellOrder Status: "+WriteIf( SellOrderID != "", SellOrderID+" "+ibc.GetStatus( SellORderID, True ),"")+"\n"+ 
" TWS Position Size: "+NumToStr(ibc.GetPositionSize( Name() ),1.0,False); 

Clearing OrderIDs at the Start of a New Bar

In this example the OrderIDs are automatically cleared at the start of a new bar. This limits each type of trade (Buy, Sell, short, or Cover) to one per bar. If you want to place a new order while an order is pending, you may either cancel it first or modify the pending order.
Since the start of a new bar can only be detected when its first quote arrives, it is impossible to get a fill at this time. The earliest time an order can be filled is on the second quote of the bar. How long this takes depends on market volume and whether you use eSignal (trades) or IB data (snapshots). Since the Opening quote can occur at any time during the bar interval, it may take from zero to the full bar interval before the OrderIDs are cleared. To prevent this delay, you can clear the orders with reference to your system’s clock. However, this requires that your system is perfectly synchronized with the market-time (see next example).
The best way to test this example is to use a 1-minute database and set your chart to the 5-minute timeframe. Next, open the Bar-Replay tool, select a range, and set the Step Interval to 1 minute and the Step Rate to 1/Sec.
The chart background will flash Green when a Buy order is placed, Red when a Sell order is placed, and White when a NewBar is started. You can test blocking of repeat orders by manually placing orders in rapid succession. The example code below will only place one Buy and/or Sell order during each bar-period (between White flashes). This method does allow you to take quick profits when both your entry and exit price are hit within one bar period. However you can do this only once per bar.

RequestTimedRefresh( 1 ); 
ibc = GetTradingInterface("IB"); 
Transmit = ParamToggle("Transmit","OFF|ON",0); 
BuyTrigger = ParamTrigger("Place Buy order","BUY"); 
SellTrigger = ParamTrigger("Place Sell order","SELL"); 
Reset = ParamTrigger("Reset All","RESET"); 
PrevTN = StaticVarGet("TimeNumber"); 
TN = LastValue(TimeNum()); 
NewBar = TN != PrevTN; 
StaticVarSet("TimeNumber",TN); 
BuyOrderID = StaticVarGetText("BuyOrderID"); 
SellOrderID = StaticVarGetText("SellOrderID"); 
BuyPending = ibc.IsOrderPending(BuyOrderID); 
SellPending = ibc.IsOrderPending(SellOrderID); 
if( NewBar ) 
{ 
if( NOT BuyPending ) StaticVarSetText("BuyOrderID",""); 
if( NOT SellPending ) StaticVarSetText("SellOrderID",""); 
SetChartBkColor( colorWhite ); 
} 
if( BuyTrigger AND BuyOrderID == "" ) 
{ 
BuyOrderID= ibc.ModifyOrder( BuyOrderID, Name(), "Buy", 100, "MKT", 0, 0, "Day", Transmit); 
StaticVarSetText("BuyOrderID",BuyOrderID); 
SetChartBkColor( colorBrightGreen ) ; 
} 
else if( SellTrigger AND SellOrderID == "" ) 
{ 
SellORderID = ibc.ModifyOrder( SellORderID , Name(), "Sell", 100, "MKT", 0, 0, "Day", Transmit); 
StaticVarSetText("SellOrderID",SellOrderID); 
SetChartBkColor( colorRed ) ; 
} 
else if( Reset ) 
{ 
StaticVarSetText("BuyOrderID",""); 
if( BuyPending ) ibc.CancelOrder( BuyOrderID ); 
StaticVarSetText("SellOrderID",""); 
if( SellPending ) ibc.CancelOrder( SellOrderID ); 
ibc.CloseAllOpenPositions(); 
} 
LastTWSMsg = ibc.getLastError( 0 ); 
BuyStatus = WriteIf( BuyOrderID != "", BuyOrderID+", Status: "+ibc.GetStatus( BuyORderID, True ),""); 
SellStatus= WriteIf( SellOrderID != "", SellOrderID+", Status: "+ibc.GetStatus( SellORderID, True ),""); 
LastBuyTime= Nz(StaticVarGet("LastBuyTime")); 
LastSellTime= Nz(StaticVarGet("LastSellTime")); 
Title = "\n"+ 
"Last TWS Error Msg: "+LastTWSMsg+"\n"+ 
" BuyOrderID: "+BuyStatus+"\n"+ 
" SellOrderID: "+SellStatus+"\n"+ 
"TWS Position Size: "+NumToStr(ibc.GetPositionSize( Name() ),1.0,False); 
Plot(C,"Close",colorBlack,styleBar); 

Clearing OrderIDs after a Delay

This example maintains a delay between same-type orders. It uses a delay-timer for each type of trade and clears the OrderIDs when the delay times out.
If you set the delay to the bar-interval and if your system clock were synchronized with the market, then this method would give similar results to the NewBar method discussed earlier. However, this method is better since it enables you to place your LMT orders before the Open of the bar, thus giving you a one-quote timing advantage. This may not sound like much, but in fast trading, especially during moderate trading volume, this improves your chances of getting LMT fills.
Again, the best way to test this example is to use a 1-minute database and set your chart to the 5-minute timeframe. Next, open the Bar-Replay tool, select a range, and set the Step Interval to 1 minute and the Step Rate to 1/Sec.
Experiment with the parameter options and observe that you cannot place a same-type order before the delay has timed out (see Chart Title).

RequestTimedRefresh( 1 ); 
ibc = GetTradingInterface("IB"); 
Transmit = ParamToggle("Transmit","OFF|ON",0); 
BuyTrigger = ParamTrigger("Place Buy order","BUY"); 
BuyLockoutPeriod = Param("BuyLockoutPeriod",10,1,300,1); 
SellLockoutPeriod = Param("SellLockoutPeriod",10,1,300,1); 
SellTrigger = ParamTrigger("Place Sell order","SELL"); 
Reset = ParamTrigger("Reset All","RESET"); 
BuyOrderID = StaticVarGetText("BuyOrderID"); 
SellOrderID = StaticVarGetText("SellOrderID"); 
BuyPending = ibc.IsOrderPending(BuyOrderID); 
SellPending = ibc.IsOrderPending(SellOrderID); 
RequestTimedRefresh( 1 ); 
NewSecond = Status("redrawaction"); 
if( NewSecond ) 
{ 
BuyCountDown = Max( 0, Nz(StaticVarGet("BuyCountDown"))-1); 
StaticVarSet("BuyCountDown", BuyCountDown); 
if( BuyCountDown == 0 AND NOT BuyPending ) StaticVarSetText("BuyOrderID","");  
SellCountDown = Max( 0, Nz(StaticVarGet("SellCountDown"))-1); 
StaticVarSet("SellCountDown", SellCountDown); 
if( SellCountDown == 0 AND NOT SellPending ) StaticVarSetText("SellOrderID",""); 
} 
BuyCountDown = Nz(StaticVarGet("BuyCountDown")); 
SellCountDown = Nz(StaticVarGet("SellCountDown")); 
if( BuyTrigger AND ( BuyOrderID == "" AND BuyCountDown == 0 ) ) 
{ 
BuyOrderID= ibc.ModifyOrder( BuyOrderID, Name(), "Buy", 100, "MKT", 0, 0, "Day", Transmit); 
StaticVarSetText("BuyOrderID",BuyOrderID); 
StaticVarSet("BuyCountDown", BuyLockoutPeriod); 
SetChartBkColor( colorBrightGreen ); 
} 
else if( SellTrigger AND ( SellOrderID == "" AND SellCountDown == 0 ) ) 
{ 
SellORderID = ibc.ModifyOrder( SellORderID , Name(), "Sell", 100, "MKT", 0, 0, "Day", Transmit); 
StaticVarSetText("SellOrderID",SellOrderID); 
StaticVarSet("SellCountDown", SellLockoutPeriod); 
SetChartBkColor( colorRed ); 
} 
else if( Reset ) 
{ 
StaticVarSetText("BuyOrderID",""); 
if( BuyPending ) ibc.CancelOrder( BuyOrderID ); 
StaticVarSetText("SellOrderID",""); 
if( SellPending ) ibc.CancelOrder( SellOrderID ); 
ibc.CloseAllOpenPositions(); 
} 
LastTWSMsg = ibc.getLastError( 0 ); 
BuyStatus = WriteIf( BuyOrderID != "", BuyOrderID+", Status: "+ibc.GetStatus( BuyORderID, True ),""); 
SellStatus= WriteIf( SellOrderID != "", SellOrderID+", Status: "+ibc.GetStatus( SellORderID, True ),""); 
LastBuyTime= Nz(StaticVarGet("LastBuyTime")); 
LastSellTime= Nz(StaticVarGet("LastSellTime")); 
Title = "\n"+ 
"Last TWS Error Msg: "+LastTWSMsg+"\n"+ 
" BuyOrderID: "+BuyStatus+"\n"+ 
" BuyCountDown: "+NumToStr(BuyCountDown,1.0,False)+" Sec.\n"+ 
" SellOrderID: "+SellStatus+"\n"+ 
" SellCountDown: "+NumToStr(SellCountDown,1.0,False)+" Sec."+"\n"+ 
" TWS Position Size: "+NumToStr(ibc.GetPositionSize( Name() ),1.0,False); 
Plot(C,"Close",colorBlack,styleBar); 

Alternating Trades

If your system trades LMT orders, you can design your orders to alternate between Long and Short without any limitations.
This method is well suited for reversal systems and high speed trading.
If your limit prices are set properly, this method will allow a fast sequence of reversal trades that can be very profitable. Systems like this may make several trades per minute, sometimes for several minutes, during high volatility.
In the following example, Order Status is checked, and an opposite order is only allowed to pass if the previous order has been filled. Note that this demo does not work if the Transmit ParamToggle is turned Off because under that condition no orders are transmitted to the market and can thus never be filled.
Again, the best way to test this example is to use a 1-minute database and set your chart to the 5-minute timeframe. Next open the Bar-Replay tool, select a range, and set the Step Interval to 1 minute and the Step Rate to 1/Sec.

RequestTimedRefresh( 1 ); 
ibc = GetTradingInterface("IB"); 
Transmit = ParamToggle("Transmit","OFF|ON",0); 
Reset = ParamTrigger("Reset All","RESET"); 
BuyTrigger = ParamTrigger("Place Buy order","BUY"); 
SellTrigger = ParamTrigger("Place Sell order","SELL"); 
LastTrade = StaticVarGetText("LastTrade"); 
BuyOrderID = StaticVarGetText("BuyOrderID"); 
SellOrderID = StaticVarGetText("SellOrderID"); 
BuyPending = ibc.IsOrderPending(BuyOrderID); 
SellPending = ibc.IsOrderPending(SellOrderID); 
IBPosSize = ibc.GetPositionSize( Name() ); 
BuyStatus = ibc.GetStatus( BuyORderID, True ); 
SellStatus = ibc.GetStatus( SellORderID, True); 
if( BuyTrigger ) 
{  
if( LastTrade == "Sell" OR LastTrade == "" ) 
{ 
if( SellStatus == "Filled" OR SellStatus == "" ) 
{ 
BuyOrderID= ibc.ModifyOrder( BuyOrderID, Name(), "Buy", 100, "MKT", 0, 0, "Day", Transmit); 
StaticVarSetText("BuyOrderID",BuyOrderID); 
StaticVarSetText("LastTrade", "Buy"); 
SetChartBkColor( colorBrightGreen ) ; 
} 
} 
} 
else if( SellTrigger ) 
{ 
if( LastTrade == "Buy" OR LastTrade == "" ) 
{ 
if( BuyStatus == "Filled" OR BuyStatus == "" ) 
{ 
SellORderID = ibc.ModifyOrder( SellORderID , Name(), "Sell", 100, "MKT", 0, 0, "Day", Transmit); 
StaticVarSetText("SellOrderID",SellOrderID); 
StaticVarSetText("LastTrade", "Sell"); 
SetChartBkColor( colorRed ) ; 
} 
} 
} 
else if( Reset ) 
{ 
StaticVarSetText("BuyOrderID",""); 
if( BuyPending ) ibc.CancelOrder( BuyOrderID ); 
StaticVarSetText("SellOrderID",""); 
if( SellPending ) ibc.CancelOrder( SellOrderID ); 
StaticVarSetText("LastTrade", ""); 
ibc.CloseAllOpenPositions(); 
} 
LastTWSMsg = ibc.getLastError( 0 ); 
//BuyStatus = WriteIf( BuyPending, BuyOrderID+", Status: "+BuyStatus,""); 
//SellStatus= WriteIf( SellPending, SellOrderID+", Status: "+SellStatus,""); 
Title = "\n"+ 
"Last TWS Error Msg: "+LastTWSMsg+"\n"+ 
" BuyOrderID: "+BuyOrderID+" "+BuyStatus+"\n"+ 
" SellOrderID: "+SellOrderID+" "+SellStatus+"\n"+ 
" Last Trade: "+LastTrade+"\n"+ 
" TWS Position Size: "+NumToStr(IBPosSize,1.0,False); 
Plot(C,"Close",colorBlack,styleBar); 

Edited by Al Venosa

Filed by Herman at 12:03 am under Real-Time AFL Programming
Comments Off on Preventing Repeat Orders and Whipsaws

Filed by Herman at 7:01 pm under Debugging AFL
Comments Off on Adding Manual Test Signals

Besides demonstrating the basics of Automated Trading (AT), the code below may function as a diagnostic tool during AT code development. It often happens that things suddenly stop working, and no orders are transmitted. When this happens, and before you start looking for bugs in your code, you can run this code to verify that your interfacing to the TWS is functional.

For orders to be transmitted to the market you must have entered your “Unlock Code” for the IB Controller in the Unlock window that pops up when you click Files -> Enter Unlock Code. You can obtain your code electronically by following the link to the IBc User Agreement. When you have signed and submitted the User Agreement the Unlock Code will be emailed to you within seconds.

The test code below can be executed from an Indicator window and will test your AB->TWS connection by placing orders from the Param window to your eDemo or Paper Trading account:

Order and TWS Status is displayed in the Title:

If you are using IB?s eDemo, orders may be processed slowly enough for you to observe how the orders are processed.

The code below illustrates several basic but very important aspects of Automated Trading, and it is important to fully understand this code before trying more complex programs. The most important concept to understand is that of the Order ID. The IBc returns a unique OrderID for each order placed. This OrderID can subsequently be used to modify, transmit, cancel, and get status for the order. For any AT system to function properly, OrderIDs must be tracked meticulously at all times. Using an expired OrderID, a non-existing one, or one for an order that is already filled, for example, will lead to API errors.

SetChartOptions( 2, chartWrapTitle );
RequestTimedRefresh( 1 );
BuyOrderTrigger = ParamTrigger( "Place Buy order on TWS", "BUY" );
SellOrderTrigger = ParamTrigger( "Place Sell order on TWS", "SELL" );
CancelOrderTrigger = ParamTrigger( "Cancel order on TWS", "CANCEL" );
TransmitTrigger = ParamTrigger( "Transmit Order", "TRANSMIT" );
ibc = GetTradingInterface( "IB" );
OrderID = StaticVarGetText( "OrderID" );
ORderStatus = ibc.GetStatus( ORderID, True );

if ( BuyOrderTrigger )
{
    OrderID = ibc.ModifyOrder( OrderID, Name(), "Buy", 100, "MKT", 0, 0, "Day", TransmitTrigger );
    StaticVarSetText( "OrderID", OrderID );
}

if ( SellOrderTrigger )
{
    OrderID = ibc.ModifyOrder( OrderID, Name(), "Sell", 100, "MKT", 0, 0, "Day", TransmitTrigger );
    StaticVarSetText( "OrderID", OrderID );
}
else
    if ( CancelOrderTrigger )
    {
        OrderID = StaticVarGetText( "OrderID" );
        ibc.CancelOrder( ORderID );
    }
    else
        if ( TransmitTrigger )
        {
            if ( ibc.IsOrderPending( ORderID ) )
                ibc.Transmit( ORderID );
        }

OrderID = StaticVarGetText( "OrderID" );

LastTWSMsg = ibc.getLastError( ORderID );
Title = "\n" +
        "           OrderID: " + ORderID + "\n" +
        "Last TWS Error Msg: " + LastTWSMsg + "\n" +
        "Order Status at IB: " + ORderStatus + "\n" +
        " TWS Position Size: " + NumToStr( ibc.GetPositionSize( Name() ), 1.0, False );

Edited by Al Venosa

Filed by Herman at 12:56 am under System Automation
Comments Off on Testing your AB-IBc-TWS Communication

You can suggest future topics for the Real-Time Auto-Trading category in a comment to this post.

Filed by Herman at 8:49 pm under Real-Time Auto-Trading
Comments Off on Request Real-Time Topics here

When migrating from end-of-day (EOD) to Real-Time (RT) trading, you will stumble upon many surprises, some beneficial and some that will be difficult to overcome. Trading is a very personal activity, and traders differ extensively on how they perceive the critical factors for success. The list below highlights a few areas where you should expect differences from your EOD experience.

1. Appearance of Bars. When you reduce the chart’s Time Frame (TF), the appearance of price-bars changes drastically. For example, in EOD you will rarely find sequential bars with unchanged prices; in the minute time frame, however, this may happen surprisingly often. Reducing the TF to 5 seconds exacerbates the problem even more. Many bars may be simple flat, horizontal lines where the OHLC prices are all equal. This could cause your indicators to drift and generate misleading signals. In addition, as you reduce the TF, the share volume may decline to the point where a bar might only represent a single trade, making it extremely difficult to get a fill at its price.

2. Data Padding. If you trade portfolios and use Foreign() the data you see may have been aligned and padded to your currently selected symbol (see: Automatic Analysis/Settings/check Pad and align all data to reference symbol). In this case the bar-prices you see are not real, have no volume and, of course, cannot be traded.

3. Gaps. When Indicators are used in RT, they do not correct for overnight gaps and intraday no-trading periods, which could cause problems with system performance.

4. Open vs. Close. In RT trading, the previous close is often equal or nearly equal to the current open since they are just one quote apart, while in EOD the entire trading system may be based on the difference between the previous closing price and the current open price.

5. Internet Delays. Internet delays impose much more significant effects in RT compared to EOD. The shorter time frames preclude speedy responses to price changes. Consequently, you will need to anticipate them and ensure your orders are placed if and when the expected move takes place.

6. Volume. Volume is a much more critical factor in RT trading. Many price changes cannot be traded because they might have resulted from just a single trade, which means that in these cases someone got filled before you did!

7. Commissions. When taking more frequent but smaller profits, which is a typical characteristic of RT trading, commissions play a larger role on performance. This means that the percentage of winning trades becomes more important than in EOD trading.

8. Order Execution. Efficient order placement is crucial to success in RT trading. Contrary to EOD trading, experiencing a few seconds or minutes delay can kill your system. Manually calculating order prices in RT becomes difficult if not impossible.

9. Data flaws. RT Data may arrive out of sequence, contain errors, and be corrected at a later time by the data provider. This means that after an EOD Backfill, your bars may look different. This often leads to highly misleading and overly optimistic profit performance by the backtester.

10. OHLC-Timing. In EOD, Open prices occur at the start of the daily-bar interval and Close Prices appear at the end of the bar. In real time (RT), however, the Open may precede the Close by just a fraction of a second, and the actual quotes might have arrived anywhere during the bar?s interval. This can become a problem when your system is timed by your system’s clock. For example, whereas the Backtester may easily enter a trade at the Open and exit at the Close on the same bar, you may not have enough time for this to occur in RT.

11. Data Stability. EOD data are stable, i.e., the OHLC prices never change once the bar has completed. For example, if your system triggers a Short signal when the High price crosses your LMT price, this signal can never disappear. Using RT data, however, the only price that remains stable for the duration of the bar is the Open price, while the HLC prices will constantly change until after the close of the trading bar. Barring data errors, the High and Low prices, by definition, can only go higher or lower, respectively, and cannot retrace to earlier values. However, the Close price will vary throughout the bar duration and may result in multiple signals when it crosses your LMT price multiple times. So, you should exercise care when developing entry or exit signals based on the closing price relative to other prices.

12. Trend Sensitivity. An EOD Trend-Following system is an example of a trend-sensitive trading system. Because RT trades are often much shorter in duration, RT trades have greater trend immunity because the trending component of stock prices diminishes at shorter time frames.

13. Breakouts. In smaller time-frames, RT Breakout systems will seldom give you your breakout price, and slippage will be proportionally greater as you reduce the timeframe. This is because the price tends to jump over your threshold and fill at a worse price.

14. Static Variables. You need to use Static Variables to carry system data from one pass to the next. There is simply no other way to pass parameters from one AFL execution to the next.

15. Random Price Movements. Price movements are subject to random price fluctuations introduced by slow trader response and Internet delays. Such changes are negligible in EOD trading, but as you work with shorter time frames in RT trading, you will gradually start trading basic price volatility that has very little to do with the direction of the market.

16. Sequential Processing. In EOD programming, tasks are usually completed within one pass through the code. However, in RT, programming tasks are completed in many small steps spread out over multiple bars, each step dependent on whether an RT external condition is met.

Edited by Al Venosa

Filed by Herman at 9:01 am under Real-Time Auto-Trading
Comments Off on RT vs EOD Trading

When AmiBroker doesn?t receive data during regular trading hours, the missing period or bar is referred to as a Hole in the data array. Holes can occur when a ticker isn?t trading or when the data feed is interrupted. They can last for any length of time and can occur in any timeframe.

When a bar has no data, the bar is not added to the database. The database record simply skips to the next bar-period. Holes may be filled in during backfill or when the data provider transmits data corrections. When this happens, the database is updated.

When plotting Indicators based on the selected current symbol, holes are skipped to create continuous looking charts. Because of this you cannot visually detect any holes. However, as will be explained below, this is not the case when plotting foreign data.

While AmiBroker never modifies the data it collects and stores in your database, it may fill in and fix holes during operations that require a common time base. Such operations would include several AA functions, the creation of composites, and chart overlays. The process of fixing and padding data to allow these operations is called data alignment.

Data Alignment in the AA

In Automatic Analyzer (AA) operations, data Alignment can be turned On/Off and the Reference Symbol to be used for alignment can be specified. This is explained in the AFL Function Reference quoted below:

?Pad and align to reference symbol

When this is turned on, all symbols’ quotes are padded and aligned to reference symbol. Note: by default this setting is OFF. Use responsibly. It may slow down backtest/exploration/scan and introduce some slight changes to indicator values when your data has holes and holes are filled with previous bar data. The feature is intended to be used when your system uses general market timing (generates global signals based on data and/or indicators calculated using Foreign from ‘reference’ symbol) or when you are creating composites out of unaligned data. Note: if reference symbol does not exist, data won’t be padded.?

Data alignment in Indicators

Foreign data used in Indicator formulas are aligned to the current symbol and not, as is the case in the AA, to the Reference symbol specified in the AA Settings. Depending on the value of the fixup parameter of the Foreign() function, data holes can be handled in any of three ways. This is explained in the AFL Function Reference as follows:

?The last parameter – fixup – accepts following values

• 0 – the holes are not fixed
• 1 – default value – missing data bar OHLC fields are filled using previous bar Close, and volume is set to zero.
• 2 – (old pre-4.90 behaviour) – causes filling the holes in the data with previous O, H, L, C, V values

The fixup parameter of 2 is provided to maintain compatibility with earlier AmiBroker releases and should not be used when developing new code.?

Holes in Real-Time Portfolio-Trading

When executing an RT portfolio trading system from an Indicator window, you typically loop through the portfolio tickers and use the Foreign() function to access data for each ticker as it is selected. Since both the current and the foreign tickers may have holes, there are a variety of hole conditions to deal with. Holes and padded bars may appear and disappear from your chart depending on the value of the fixup parameter and whether the current or foreign ticker has holes. Charts may also change during RT backfills. Since the plotted data are the same data used in your formulas, you have to be cautious about how you use foreign data.

The criteria to detect holes are simple: if the Volume equals null, the data are missing (hole); if the Volume equals zero, the data are padded (filled hole); and if the Volume is greater than zero, the data are true (no hole).

Since foreign data are aligned to the current ticker in Indicators, it can be stated that:

? If the current ticker has no data for a particular bar, the corresponding bar-interval for the foreign array will not be displayed and will not be available for calculations. This blocks foreign data and could result in missed signals.

? If the foreign ticker has no data for a particular bar while the corresponding bar for the current ticker has data, a bar will be added to the foreign array. If fixup is turned On, this bar will be filled with prices from the previous foreign bar. If you use Intraday signals, this could result in extra or duplicate signals.

It is obvious from the above that to design a robust High-Frequency Real-time portfolio trading system that executes in the Indicator window, you must consider the effect of holes on your signals.

Detecting Holes

To detect all holes perfectly, you need a perfect reference ticker, i.e., one that trades every minute of the day for every trading day of the year. If such a perfect ticker existed, you would make this your current ticker so that all foreign data would be aligned to it. Simply checking for zero volume would identify all holes. However, since such a perfect ticker does not exist in reality, if you assume that a ticker is perfect, you are simply transferring the hole problem from one ticker to another, and you wouldn’t be any closer to solving the problem.

A perfect solution would be to use a linear Date Reference Array that has bars for all calendar days. Such an array is documented in the post titled ?Date Calculations? on this site. To apply a linear Date Reference Array to detect holes is beyond the scope of this post, but it may be covered in a later post.

Another reasonably good solution is to use a high volume ticker, like the QQQQ, for your current ticker. The problem is that even the QQQQ has low volume periods and may contain holes.

A slightly better solution is to create a Volume composite for a group of actively traded tickers, perhaps from different markets, and make it your current symbol. For this composite to have a Hole there would have to be a period during which none of the tickers traded. While this is unlikely to happen in longer time frames, it can still happen in very short time frames, making this solution imperfect also. This solution will be used here to create a chart that displays (maps) the existence of holes for an entire WatchList all at once.

The demo code below uses a separate WatchList for the composite and the tickers to be mapped. To prevent a slow response (perceived as lockup by many) when large numbers of tickers are mapped, or when you inadvertently select a very large WatchList, you can set a maximum number of tickers (defaults to 50). You should only use high volume tickers for the composite WatchList since lightly traded tickers are unlikely to add any bars to the composite.

Program Parameters

The Param window for the demo code has the following options:

The Reference Watchlist (0-based) should point to the group of tickers used to create the composite. You should always click UPDATE after changing the Reference Watchlist to create a new composite for the new Watchlist.

The Watchlist to Map selects the watchlist to be tested for holes. Each ticker in this watchlist will add a horizontal line to the chart. This would normally be your portfolio watchlist. Holes are indicated using a digit surrounded by a small circle. The digit is the fixup parameter used in the Foreign() to retrieve that data.

The occurrence of any digit shows the location of a hole. A 0 digit means the bar is Null or Empty, and no bar will show; a digit of 1 means that the bar was padded with only the C price from the previous bar; a digit of 2 means the data was padded with OHLCV (obsolete method!) data from the previous bar.

You can observe how holes are replaced with data from the previous bar. An empty bar is replaced with a bar identical to the preceding bar when you vary the fixup parameter in the Param window.

To demonstrate how a different Reference ticker changes the distribution of holes, you can open your workspace and step through (make current) different tickers in your database. The program checks the cursor position, and if you set Cursor-Selected Chart to SHOW (rather than HIDE), a price chart will be overlain on the chart for the selected ticker line. This allows you to zoom in and inspect prices in the immediate area surrounding the Hole or Padded data.

Note: Since there is no way to detect which pane the cursor is on, the cursor-position sensing will work only if the hole map is run in its own window.

Typical Hole-Maps

I the captures below, a NASDAQ 100 (N100) composite was used as reference for the ticker FISV. Each ticker in the chart (hole-map) below is assigned a horizontal time line on which the existence of holes is indicated with a small circle. Placing your cursor on a hole displays its date and data in the title. The color of the circles matches that of the ticker. The Y-axis numbers are the ticker numbers (redundant).

In the first case, the fixup parameter was set to zero and the bar location with missing bar is left empty (see cursor location). Because the digit is so small it may look like a ?1?, but it is actually a ?0?. The fact that a bar is missing means that the current symbol had data for this bar but the foreign ticker did NOT.

Fixup mode ‘0’

In the next capture a fixup parameter of 1 is used, and the missing bar is filled with the C price from the previous bar while the Volume (not shown) is set to zero.

Fixup mode ‘1’

In the next capture, the fixup parameter is set to 2, which results in the missing bar being Filled with OHLCV values from the previous bar.

Fixup mode ‘2’

In all the above examples, an EOD chart was used. However, it could just as well have been a 1-minute chart (or any other timeframe). Here is the code that produced the above charts:

RequestTimedRefresh(1);
GraphZOrder=True;
GraphXSpace = 10;
RefWLNum = Param("Reference WatchList",0,0,64,1);
MapWLNum = Param("WatchList to Map",8,0,64,1);
ScanTrigger = ParamTrigger("Reference Composite","CREATE COMP");
ShowChart = ParamToggle("Cursor-Selected Chart","HIDE|SHOW");
FixupSelection = ParamList("Fille Type (Fixup)","NO FIXUP (0)|PREVIOUS C to OHL (1)|PREVIOUS OHLCV (2)");
TMax = Param("Max. Number Tickers Scanned",50,1,200,1);
if( FixupSelection == "NO FIXUP (0)" ) FixupFlag = 0;
else if( FixupSelection == "PREVIOUS C to OHL (1)" ) FixupFlag = 1;
else if( FixupSelection == "PREVIOUS OHLCV (2)" ) FixupFlag = 2;
MapTickers = CategoryGetSymbols(categoryWatchlist,MapWLNum);
FirstVisibleBar = Status("FirstVisibleBar");
Lastvisiblebar = Status("LastVisibleBar");
XText = Lastvisiblebar-(Lastvisiblebar-FirstVisibleBar)/10;
if( ScanTrigger )
{
RefTickers=CategoryGetSymbols(categoryWatchlist,RefWLNum);
for( T=0; (symbol=StrExtract(RefTickers,T))!="" AND T < Tmax; T++ )
{
Vt=Nz(Foreign(Symbol,"V",False))>0;
AddToComposite(VT,"~MarketVolume","V",1|2|128);
}
}
else
{
for( T=0; (symbol=StrExtract(MapTickers,T))!=""; T++ )
{
VT1= Foreign(symbol,"V",FixupFlag);
F0 = IsNull(VT1);
F1 = VT1 == 0;
F2 = VT1 == Ref(VT1,-1);
VT = Nz(VT1);
if(T>15) Co = T - int(T/15)*15; else Co = T;
Plot(T,"",colorBlack,styleLine|styleNoLabel);
PlotShapes(IIf(F0,shapeDigit0,IIf(F1,shapeDigit1,IIf(F2,shapeDigit2,shapeNone))),Co,0,T,0);
PlotText(""+symbol,XText,T,Co);
}
}
DT=DateTime();
TickerNumber=round(GetCursorYPosition());
SelectedTicker=StrExtract(MapTickers,TickerNumber);
DateTimeNum=GetCursorXPosition();
if(TickerNumber>15)Co=TickerNumber-int(TickerNumber/15)*15;else Co=TickerNumber;
SetForeign(SelectedTicker,FixupFlag);
if(ShowChart) Plot(C,"",Co,styleBar|styleOwnScale|styleThick);
if( SelectedTicker == "" ) SelectedTicker = Name();
Title="\n"+EncodeColor(Co)+
NumToStr(SelectedValue(DT),formatDateTime)+","+SelectedTicker+",Vol:"+NumToStr(V,1.0,False)+","+
"Open:"+NumToStr(O,1.3)+","+"High:"+NumToStr(H,1.3)+","+"Low:"+NumToStr(L,1.3)+","+"Close:"+NumToStr(C,1.3)+"\n"+
"Previous DateTime:"+NumToStr(Ref(DT,-1),formatDateTime)+"\n"+
"Selected DateTime:"+NumToStr(DT,formatDateTime)+"\n"+
" Next DateTime:"+NumToStr(Ref(DT,1),formatDateTime);

Edited by Al Venosa

Filed by Herman at 5:59 pm under Real-Time AFL Programming
Comments Off on Data Holes in Real-Time Trading