In the batch_transform we were incrementing the trading_days counter if there
is a new day event. Thus with a window_length of 1 and daily bars you will
update the batch_transform on the first day which is correct. But with minutes
you update with the first minute bar of the day which is not correct.
This is fixed by calculating the market_close explicity and seeing whether the
event.dt is on or past it.
I also added a unittest to test the correct behavior of this.
Before the change to the RollingPanel, window_length
specified the number of days that should be in a window.
The previous commit broke this if data was minute resolution.
By passing bar='minute' to the batch_transform we internally
multiply the window_length by 60*6.5 to have a full day.
Also adds a (still rudamentary) test for batch_transform
with minute data.
When setting timeperiod in the talib function it subtracts by 1. We then used this subtracted value to set the window_length in the batch_transform which was then not passing a big enough panel. Ultimately this caused the talib transforms to always return nans.
This also makes the unittest more stringent by explicitly comparing the output of the wrapped TALib moving average to pandas rolling_mean().
Finally, this also allows passing of window_length instead of timeperiod to allow usage of the same interface as before.
With the benchmark returns marked at midnight, the performance packet
for a day was emitted *before* any events for that day were processed.
Fix by expecting benchmarks marked at the market close, for backtests
that use minute data but emit performance results daily, so that the
benchmark handles at the end of day.
TST: Also, add test that exercises the event loop with minutely data,
(with benchmarks that are marked end of day), since that combination
was previously uncovered.
The leading date of the date range was never called with update,
because in the main loop the todays_date variable was
incremented before update was called.
Fix by moving the increment to the next trading day to after the
call to update.
So that TALib is still available, but smooth out the ability to
run tests with some issues that bear investigating.
- Ignore MAVP during tests.
- Temporarily use a "regular" member instead of __doc__ string.
(TODO: look into using `type` to generate the class)
- During tests wait until a window exists.
- moved Order and Blotter to zipline.finance.blotter
- moved order method from AlgoSimulator to Blotter
- eliminated the set_order method in algorithm
- moved blotter to the algorithm
Add a test suite of dividend performance that ensures a holiday
is covered, needed because tests usually go over a random test range,
so holiday coverage isn't covered in basic test.
From @fawce.
Refactor PerformanceTracker, Blotter, and AlgorithmSimulator to
work with handling the end of a bar at the AlgorithmSimulator level
instead of within PerformanceTracker.
- PerforamnceTracker and Blotter are longer generators,
both provide functions to process events instead.
- AlgorithmSimulator calls each from within the loop running
over the data generator.
- Change test_perf_tracker utility to be compatible with change
away from PerformanceTracker as a generator.
Has the effect of:
- Fixing the timing of order emission.
- Allow minutely emission of benchmarks, which was prevented
by the extra grouping previously caused by Blotter.
Minutely emission also depends on work for streaming benchmarks
through performance and risk at a minute granularity.
- Create different benchmark containers in performance
depending on emission rate.
- Add a minute close method which updates algorithm and
benchmark returns, and calculates the risk metrics
depending on those methods.
- Provide fake 0.0 values for annualized metrics like
sharpe, sortino, and information, until we figure out
how they should be treated in the context of minutely
calculation.
*NOTE* This does not fully work without the changes to the
simulation loop by @fawce
The bar timestamps for day start and finish, for NYSE traded stocks,
should be 9:31 AM EST to 4:00 PM EST, for a total of 390 minutes.
Fix starting at 9:30 AM and the creation of 391 bars.
Instead of creating a list of benchmarks in the risk module,
stream benchmarks through the system as events, starting from the
algorithm generator.
Works towards more easily setting arbritrary pricing data as
a a benchmark, as well as working towards live minutely benchmarks.
- Add transaction and order types
- Move TransactionSimulator from trading.py to tradesimulation.py
(only used by other members of the tradesimulation module)
- Make Transaction an independent event, like dividend
- Add Blotter class.
- Flatten the transaction events to be independent of trade bar events
- Make orders into events that reach performance (need to add
handling)
- Issue IDs to orders and tracking each transaction's order id.
- Make volume share slippage fill orders independently, rather than
aggregating them into a single transaction.
- Perf tracker holds orders, serializes them with transactions.
- Order state defined and maintained by order class.
- Minutely emission of orders based on last_modified date.
Also, fix double emission of performance results with the last minute.
Change the perf tracker unit tests so that it doesn't rely on an
'extra' event triggering emission.
Unlike daily, minute emission now emits at the end of the bar in
the PerformanceTracker.transform instead of waiting for the next event.
Prevent an extra performance result with the timestamp of the midnight of
the day from being emitted.
Fix by setting the `saved_dt` value with the dt of the first event,
before entering into the main performance loop, otherwise a performance
result with a midnight timestamp and data from just the first event is
emitted.
The `test_multi_source_as_input_no_start_end` test is designed
to test for a lack of simulation parameters, so make that they are
not passed.
Was unnoticed because sim_params passed as an arg are dropped.
The start and end of the simulation parameters should be 'normalized'
i.e. midnight timestamped.
However, the algorithm tests were using the timestamp of the
first and last trade, which were in market times,
i.e. 9:30 AM and 4:00 PM EST.
Fix passing the sim_params that is used to create the trade_history,
instead of having the sim_params inferred from the source.
(Also may want to consider fixing the logic that infers the date
range from the sources provided.)
Also, add a `num_days` option to `factory.create_simulation_parameters`
so that the a date range that covers the desired number of days is covered.
Since the default sim_params were covering a year, while the test only
supplies 4 values, causing an alignment issue with the record test,
since a years worth of results were returned, but there were only 4 events.
The intraday performance results were emitting all transactions
for the entire day up to that point, instead of the desired transaction
list for the current timestamp.
Add a `dt` parameter to the `to_dict` method of PerformancePeriod so
that the transactions are limited to a specific datetime.
When the parameter is `None`, a todays_performance object will
function as previously with returning all transactions for the day.
# Please enter the commit message for your changes. Lines starting
The treasury_duration member in RiskMetrics is never used except
for in unit tests.
Remove the saving of treasury_duration in preparation for the
move of the choose_treasury method out of the RiskMetrics classes.
Down the line, if we do restore the sanving of treasury_duration,
choose_treasury can return a tuple that includes treasury_duration
instead of just returning the rate.
In the test factory creation of returns, the date creation was using
a timedelta of one day instead of incrementing by trading days.
Working towards changing risk module behavior which would leverage
the trading day map, but tests fail because non-trading days are
created.
Remove `factory.create_returns`, moving uses of that function to us
`factory.create_returns_from_period`, since the number of days input
for `create_returns` was more difficult to use when specifying ranges
over arbirtray dates.
So that the environments' exchange time is used without having to
specify it independently.
Also, moves uses of Delorean.shift for the exchange conversion inside
of environment to use the exchange_dt_to_utc method.
Wires up performance tracker so that when `emission_rate` is set
to `minute`, the performance packets are sent out every minute,
instead of once per day.
Please note, the performance packets that are generated are not
ready for prime time consumption, this patch is merely a step towards
hooking up the ability to inspect minute data.
Known issues:
- The packets do not currently include risk information.
Since we need to consider how this affects the denominators
of the risk calculations.
In preparation of changing how risk metrics are calculated,
this makes the checking of results more tolerant to floating point
rounding, since upcoming changes change the results by a neglible margin.
Converts the risk iterative and batch comparison tests to use
the trading environments next date, instead of just advancing by day,
so that the returns being passed into the RiskMetrics in the unit
test are using the same trading calendar as the internal checks for
trading days.
Fixes a case where empty return periods were being into
`calculate_period_returns`
Clearing the way for the pandas based optimization of the risk module.
Thomas Wiecki