This commit:
* Adds a crypto_benchmark that can create benchmarks for symbols found on POLO
* Changes default trading calendars to OPEN
* Properly computes daily bar data from five minute POLO bars
* Allows trading of one hundredth of a coin, later we plan to integrate per the
ratio of a full coin to its base denomination.
* Updated cython build scripts
* Updated setup.py to to install catalyst package
* Updated momentum example to use catalyst package
* catalyst executable now supports loading pipelines from multiple bundles
- Adds a `map` method to `LabelArray` that maps a unary function over
the categories of a LabelArray, shrinking the underyling codes if
possible.
- Adds a new `.relabel` method to string-dtype classifiers that maps a
unary function over the unique elements of the underlying LabelArray.
This is useful for things like cleaning noisy label data.
I'm not sure what the raw csv pulled from the federal reserve looked like before, but when trying to download fresh treasure data (data not stored in `./zipline`), there is an error that says "Time Period not in list". After checking the raw csv now, it looks like there are 5 header rows rather than just 1, so skipping those rows removes that error.
MAINT: Remove environment as an argument to benchmark source.
To allow the BenchmarkSource class to be more easily used in contexts other than
a TradingAlgorithm, remove the TradingEnvironment as an argument to the
benchmark source.
Instead:
- Pass a benchmark Asset, instead of a bencmark sid; so that the asset_finder
does not need to be passed to the benchmark source.
- Pass the pre-calculated benchmark_returns instead of an env,
which contains the benchmark_returns; a consumer can let the benchmark_returns
stay as the default of `None` when using an asset.
We may want to further refactor and make two different classes, instead of
relying on a combination of existence/non-existence of benchmark_asset and
benchmark_returns. That refactoring should be easier to do with this change.
* BLD: Modify conda label for Travis CI
We've been unnecessarily building packages that are already in the ci channel because conda build isn't respecting the full URL for that channel when using --skip-existing. We updated it to be just the name of the channel.
A TypeError exception was raised with message "Cannot join tz-naive with
tz-aware DatetimeIndex". Removing old unnecessary workaround in
`holidays_at_time` function (Pandas already fixed that before 0.18)
fixes this issue.
BLD: Move setuptools_scm recipe earlier
in build process. We order alphabetically.
BLD: Update to newest Cython version
BLD: Update version of matplotlib
BLD: Pin matplotlib dependency
The only downstream contex that was using batch_order_target_percent
already had all necessary prices, so calling batch_order_target_percent
was wasteful.
This allows us to remove the check for whether the provided dt had a
time of midnight, which was a flimsy way to infer if the data frequency
was 'daily'. Besides the explicit check being preferable, this method
was broken on the futures calendar, since midnight is a valid market
minute.
Added as a subclass of MinuteEquityHistoryTestCase, where the primary
calendar is 'us_futures'.
Notes on modifications to MinuteEquityHistoryTestCase:
- To work on generic calendars, many tests now use set minutes for
window start and end, and check the values on active equity minutes.
- test_minute_regular should test against active equity minutes
- Adapts test_minute_midnight to work with futures calendar
- Use a method of getting the last open minute that works with
calendars that are open at midnight
- Test against Sunday at midnight, since the real intention of this
test is to check that given a non-open minute, we fall back to the
last open minute.
Added as a minimal subclass of DailyEquityHistoryTestCase, swapping out
just the primary calendar. This requires significant modifications to
DailyEquityHistoryTestCase, to allow for a generic primary calendar.
Previously, a dataframe passed into BcolzDailyBarWriter.write that was
missing an expected session between its first and last sessions would be
written incorrectly. Upon converting the dataframe to a ctable, the
values for all days following the gap would be shifted backwards, and
nans would be shifted in at the end.
This commit handles the issue by asserting that the number of rows in
the input table matches the number of sessions in the calendar between
the table's first and last sessions.
Also fixes a test that was mistakenly using minutes_in_range where it
should have been using sessions_in_range (uncovered by this change).
The March, June, September, and December contracts for these futures
contain most of the trading activity, so we exclude the other more
sparsely traded contracts from the chain.
For futures, we need to divide the position’s commission by the
contract size to get a per-unit commission in order to properly update
the position’s cost basis.
Optimize session close lookups in MinuteResampleSessionBarReader:
- Adds `session_closes_in_range` method (along with
`session_opens_in_range`) to TradingCalendar to allow vectorized
retrieval of all values in a range of sessions.
- Improves code path for resampling a single session's worth of data (as
is the case when calling `get_value`), since we don't actually need to
look up the close minute.
When retrieving the open and close for a given session, we only care
about the scalar values, so using DataFrame.at instead of DataFrame.loc
is significantly faster.
Instead of recursively calling `DailyHistoryAggregator.closes` until we
find a non-nan close, we can instead call `load_raw_arrays` once, and
find the value from the returned array.
Should be no functional change.
By making the raise on `if not isinstance` instead of doing a continue on `if
isinstance` (with a raise at the end of the loop if no 'good' conditions were
met'), the function should be more amenable to adding an additional validity
check, after the type check passes.
This is on the path to adding an additional validity checks parameter to
`check_parameters`, e.g. adding an 'is positive' check.
- Instead of maintaining a separate `j` value, set the bounds of the range so
that `i` is the values emitted by the range.
- Change `close_loc` to `prev_close_loc` since the market close location is used
to ensure that the data index stops at the market open if the entire day is
nans.
- Change the setting of `loc` to be done before the loop which check for nans,
instead of setting to the previous close loc at the end of the loop.
This prepares for a separate fix to prevent out of bounds access when the first
session has nans for all minutes.
* ENH: Use regular rounding to calculate order amounts.
We previously tried to prevent accidental over-ordering by truncating
orders down unless they were within 1e-4 of the next higher integer.
Unfortunately, this makes it easy for a sell order to be one share short
of the desired position.
Using regular rounding treats both buys and sells in the same way.
* ENH keep non-rounding behavior consistent, but leave code structured to make easier to override
* DOC make round_order public and describe behavior in docstring
The CFE was closed along with the NYSE in observation of the days of
mourning in honor of the passing of presidents Gerald Ford and Ronald
Reagan. The CFE also observed the closures due to Hurricane Sandy,
along with NYSE. Adding those adhoc holidays to exchange_calendar_cfe
and removing them from cfe.csv in tests. To fit with
USNationalDaysofMourning, also removing the closure in observation of
the day of mourning in honor of the passing of president Nixon in
1994, despite the fact that the exchange did not exist at that time.
Signed-off-by: Maxwell Rounds <maxwell.j.rounds@gmail.com>
Useful for avoiding the need to create Asset objects when sids are
easier to use.
This is based off the existing implementation of StaticAssets, and
StaticAssets is now implemented as a wrapper around StaticSids.
DOC: Clarify changes and fix formatting
DOC: Remove 1.0.3 whatsnew
DOC: More formatting
DOC: Remove superfluous info
DOC: Change deploy command
DOC: Fix typo
DOC: Add mention of winsorize factor
DOC: Change date of release
DOC: Remove date for now
DOC: Change release to development
Capital base is included in the sim params, so we should define the
value there, or use the default.
This change also unifies the default capital base as 1e5, as was
previously defined in algorithm.py.
If we have a test that's being called more than once (i.e. two
test cases, both subclasses of the same base test case, with
different setup but calling the same test), allow the subsequent
calls to re-consume the same params
Make a new decorator, expect_strictly_bounds, which shares logic
with the existing expect_bounded (which checks that inputs fall
INCLUSIVELY between bounds),
When EQUITY_DAILY_BAR_SOURCE_FROM_MINUTE is set, use
EQUITY_MINUTE_BAR_LOOKBACK_DAYS as the default value for
EQUITY_DAILY_BAR_LOOKBACK_DAYS.
Without this, trying to run a minutely backtest in a test setting only
EQUITY_MINUTE_BAR_LOOKBACK_DAYS and EQUITY_DAILY_BAR_SOURCE_FROM_MINUTE
fails because the benchmark creation process makes a daily history call
for the entire period of the backtest, which then fails because the
equity daily bar calendar is shorter than the equity minute bar
calendar.
I can't imagine a circumstance in which you'd want the daily bar
calendar to be shorter than the minute bar calendar when you're sourcing
daily bars from minutes, so this change makes that the default behavior
unless it's explicitly overridden.
Fixes a bug where we'd fail to raise an error if the start/end of a
history window call don't aren't in the loader's calendar.
We were started dropping this error after a previous change swapped out
calls to `index.get_loc` with calls to `index.searchsorted` to avoid
creating hash tables in pandas.
When opening with a new `end_session`, i.e. opening for append, write the new
end session to the metadata.
Fixes an issue where the calendar on minute bar readers did not include the
recently appended day, causing reads on the last values to fail.
According, update append test to read a value, instead of checking table length.
TST: add test case for empty raw events data
BUG: update for python compatibility
MAINT: Simplify assertion for empty events case.
DOC: Add comments on indexer unpacking.
MAINT: move some config to test method
Remove need for a consumer that is editing an existing minute bars directory to
reread the values which should not change from the metadata.
Add a test to the append on new day and truncate, which would be the common
usage of this method.
From pep-0008:
```
Always use a def statement instead of an assignment statement that binds a
lambda expression directly to an identifier.
Yes:
def f(x): return 2*x
No:
f = lambda x: 2*x
The first form means that the name of the resulting function object is
specifically 'f' instead of the generic '<lambda>'. This is more useful for
tracebacks and string representations in general. The use of the assignment
statement eliminates the sole benefit a lambda expression can offer over an
explicit def statement (i.e. that it can be embedded inside a larger expression)
```
This format is intended for storing data for all sids of an asset type,
e.g. equities or futures for a session. bcolz is not used to avoid the overhead
of creating the directories and files for each asset (which numbers around ~8000
for active equities) can be removed since the update is meant to be read at
once, instead of supporting the random access pattern needed by the simulation.
This patch only adds the reader/writer pair, with the management of finding the
paths to delta files and the application of the updates to the bcolz write left
to internal loader code.
Also, the update reader interface is intentionally constrained to the data for
an entire session to allow for an implementation that allows for mid-session updates.
* ENH: Adds support for supplementary asset mappings
- Adds a supplementary_mappings table to the assets.db, to hold point-
in-time mappings of sids to arbitrary categories of values, e.g.
alternative identifiers. This bumps ASSET_DB_VERSION.
- Adds supplementary_map and supplementary_map_by_sid to AssetFinder,
caches of the underlying table that are fully populated on first
access, which map the supplementary values to sids and vice versa,
respectively.
- Adds lookup_by_supplementary_mapping method, which fronts
supplementary_map to query for the asset last known to have held a
value at a given dt.
- Add get_supplementary_field method, which fronts
supplementary_map_by_sid to query for the last known value held by an
asset at a given dt.
For futures that behave like GC, use the latest roll as the back contract when
walking backwards over the window, so that when the front contract is skipped
because it never has more volume between its auto close date and the previous
auto close date, the back contract which did have volume is still used when
making comparisons to construct the chain.
Instead of maintaining a separate index into the sessions index, use the `.freq`
member of the sessions index for decrementing to the current session and finding
the previous session.
When following the release guide, installing from testpypi using the
`-i` flag failed on my, and at least one other's, development machines.
The cause of the failure appears to be that pip would look for packages,
such as `LogBook` or `pandas` on `testpypi`. However many dependencies
do not have versions that meet our version criteria. (e.g. pandas does
not have a version between 0.16.0 and 0.18.0 on testpypi.)
Instead, use `--extra-index-url` so that other packages can use `pypi`
as a fallback server, instead of being limited to `testpypi`.
To support contracts such as `PL` which should roll from F->J->N->V, add the
ability to pass a predicate function to the ordered contract chain contstrution
which returns `True` if the contract is allowed in the chain.
Convert the end minute to the its session label before calling `_active_contract`,
otherwise the volume roll finder's attempt to use the session bar reader fails
due to a non-session label Timestamp.
Fix multiple errors when attempting to generate rolls for futures which do not
roll month to month, e.g. the Eurodollar.
These errors were caused by logic that always incremented from contract to
contract by delivery month, with errors when the next contract was not part of
the quarterly roll chain and thus had not yet begun trading even though the
previous contract had autoclosed. Instead, filter out these contracts and only
allow contracts that have begun trading before the previous contract's autoclose.
This is in lieu of a more explicit specification of quarterly rolls.
Instead of requiring the roll finder to juggle the indices into the ordered
contracts, use a doubly linked list where the nodes element is the contract
with members pointing to the previous and next contracts in the chain.
Besides improving legibility in the roll finder code, this change is on the path
to adding a predicate to exclude contracts from the chain, e.g. contracts in ED
which are not in the roll schedule.
Change test results for primary chain, since new implementaton does not stop at
contract in which has not yet started when constructing the chain.
- Use `expect_bounded` to check inputs.
- Add tests for expected failures from `MACDSignal`.
- Use `float64` instead of `float` in a few places. This prevents
diverging behavior on 32-bit systems.
- Docstring edits.
- Use a RandomState with a seed so that we have repeatible results.
- Use `randint` instead of `random_integers.` `random_integers` is
deprecated.
- Use `parameter_space` to test multiple period lengths.
Previously, if input to the BcolzMinuteBarWriter had the first bar on a
non-trading minute, the next trading session would be considered the
"first day" in the input. Now, we consider the previous trading session
the "first day".
The intention is to correctly associate minutes after official trading
hours on half days with session that closed early, not the following
session (a future improvement here would be to not accept minutes
outside trading hours).
Fix common error condition which was triggered whenever the session at the end
of the prefetched history window was a session where the back contract was
active. When the back contract was the active contract, the next contract for
consideration was the front contract at the end of the window, which
definitionally always has an autoclose after the end of the window.
Instead, just start seeking backwards from the end of the window.
Also prevent lookahead bias in volume rolls, which was caused by the using the
volume for a session to determine whether that session had rolled. Information
that would not have been available at the beginning of the session.
This change makes the volume rolls overly conservative, and may be improved by
looking at vectors of the preceding volume and making the roll off of momentum.
Any DataFrame that's had `.loc` or `.iloc `called on it participates in
a cycle, which means they're not immediately garbage collected when they
go out of scope. This matters for pipeline results because they consume
multiple megabytes per column, which means that a pipeline result with
many columns can hold take up over 100MB. By manually breaking
DataFrame cycles, we can ensure that we never hold multiple pipeline
results in memory at once.
Otherwise, we either raise an exception or filter out all unsafe values.
This addresses an issue where the BcolzMinuteBarWriter would scale up
values to convert to uint32, but the resulting values were too large,
and would be mangled.
Based on the approach we take in the BcolzDailyBarWriter.
This modificaiton to the estimates loader allows the caller to pass
in an equity pricing loader which can then be used to get split data
for sids. That split data is then used to do point-in-time adjustments
of estimates data.
TST: add test for multiple estimates columns
TST: add test for multiple datasets requesting different columns
TST: add blaze versions for all next/previous tests
Protect a case where data is written with a non-zero volume, but a 0/nan for the
OHLC values. The slippage model was relying on a non-zero volume implying that
there was a valid trade price for the corresponding bar. When there was a mismatch,
a transaction with a nan value was created, which would in turn propagate the
nan into portfolio value, which would then cause errors when the portfolio value
was used to size orders during rebalancing.
When data is fixed, can remove.
(Also may want to add behavior to minute bar writer to ensure that 0 volumes
always have corresponding nan ohlc.)
When the following conditions occur,
- a `nan` occurred after a half day (e.g. on the Monday after
Thanksgiving, where the Friday would be a half day.)
-data was written to the span between the early close and where the market close
would have been if it were not an early close session
- a `nan` also occured on the last minute of the early market session.
the exisitng implementation would incorrectly return a `nan` when requesting a
forward filled price.
The steps that caused this error were.
1. Request for `'price'` on the market open of the day after the early close.
2. `nan` is found for that minute
3. `get_last_traded_dt` is called, and finds a volume that occurs after the
early close. e.g. `18:47` when the market close was `18:00`.
4. The minute position for `18:47` is used, when calling
`find_positon_of_minute`, since that value is after the `market_close` the
minute is set to the position of `18:00`` due to the delta logic in
5. Since there is also no data in at `18:00`, a `nan` is returned, even though
there were valid minutes earlier in the session. e.g. a non-zero volume at
`16:47` should have been used, but was not.
Fix by checking the current minute against the minute close when searching for
the last traded minute. If the minute is greater than the market close for the
corresponding day, continue the search until the minute position is within the
trading session.
This could also be fixed by enforcing that only zeros can be written between an
early close and the minute where the close would have been, but this fix allows
the reader to work with existing data.
The rolls are already calculated and assigned to `rolls_by_asset` earlier in the
`load_raw_arrays` method, so remove the duplication.
The change should not affect results.
The use of `slice_indexer` on all market minutes was taking about 110ms on my
development machine.
This change to getting the start and end indices changes the entire `_calendar`
method to take 10ms on the same machine.
Noticed while creating a `HistoryLoader` in a notebook context.
The end date of the last contract with a sufficient start date was being
used for the continuous future overall end date; however the end date of
that contract (which is the last day for which there is data for the
contract) is not necessarily the greatest end date out of all contracts.
It is possible for the furthest out contract to have some, but very
few, trades before it is more actively traded. Which would give it a
start date within in the range of the simulation, but an end date is
earlier than the other contracts which are active during the simulation.
This bug would result in `nan`s when getting the current price because
of the `end_date` check in `get_spot_value`. When the current simulation
time was greater than the `end_date` of the last contract the condition
which guards against attempting to get data for an instrument past its
end date would return a `nan`, even when the current underlying contract
did have data for that date.
Use max end date of all contracts instead of the last one, to ensure
that the continuous future last date is always great enough to allow
access to all contracts with in the chain.
Also, use min start date to accurately mirror the end date behavior.
Use `roll_style` not `roll`.
Also, add test case to cover using the session bar reader `get_value`,
by adding a test which uses `close`, since only `contract` was being
exercised, which does not exercise the session daily bar reader.
Register equities, futures, and continuous futures to an `abc` which
signifies that the type is associable with data, and thus can be used in
a history context.
May want to use this in `check_parameters` for `BarData` methods, but
work would need to be done to make sure the error message still displays
the registered types.
In preparation for using `DataPortal` in notebooks, remove restriction on
the `HistoryLoader` to dates that are monotonically increasing. Notebook
usage of the `DataPortal` is more useful when the end of the history
window can be arbitrary dates without having to restart the notebook kernel.
Due to the implementation of the prefetch and caching logic, the end
date of history calls could previously only increase. e.g. `2016-11-01`,
`2016-11-02`, `2016-11-03`. This pattern was sufficient for backtesting
and live simulations, since the current time of the algorithm only ever increases.
With this change, which resets the underlying sliding window when the
last fetched idx is greater than the
Now calls to history in the same process with end dates such
`2016-11-01`, `2016-10-31`, `2015-11-02` should work.
Allow `ContinuousFuture` when checking for a single "asset".
This could be further improved by:
- Defininng a tuple of the `Asset`-like types OR making
`ContinuousFuture` and `Asset` share a common type (whether that is
`ContinuousFuture` inheriting from `Asset` or making `Asset` and
`ContinuousFuture` share a common type.)
- Make a `history` test which uses `BarData` + `ContinuousFuture`,
instead of just using the history loader directly from tests.
To support using a `DataPortal` and `HistoryLoader` in a notebook, allow
the prefetch length to be configurable, so that it can be set to 0.
Unlike backtesting where the prefetch is useful for repeated history
windows viewed from datetimes which are monotonically increasing by a
small amount, the notebook usage of history windows needs only to
retrieve the exact data needed for the window specified.
This patch also fixes some boundary conditions related to rolls and
adjustments which were uncovered by querying for the adjustments with an
end date near the end of the window.
Rename _get_daily_window_for_sids to _get_daily_window_data.
Rename _get_minute_window_for_assets to _get_minute_window_data.
Rename _get_daily_data to get_daily_spot_value.
Instead of using the difference between the session close of the front
contract before the roll and and the open of back contract on the
beginning of the roll, use the close of both at the end of the session
before the roll.
The closes of the session prior to roll is in lieu of settlement data.
Make `__next__` and `seek` share code instead of seek() calling
`__next__`. This avoids having to make a large number of integer
comparisons and `asanyarray` calls when seeking more than one tick
forward.
There have been cases where the requested start or end date is not in
the history calendar.
Add the beginning and of the calendar to the KeyError to give more
detail to figure out root cause.
Add roll style which takes the volume of the contracts into account.
If the volume moves from the front to the back before the auto close
date, the roll is put at that session.
Also, factors out some of the common logic shared with calendar based rolls.
Match the behavior of the minute bar reader, now that the session and
minute bar readers share a common interface.
isnull is slightly slower than checking against -1; however, n cases
where we check against illiquid trades in a tight loop, volume is
checked which is not using nan. The change here should be marginal with
regards to performance.
Move dates queried near beginning of test data so that the range of data
covered does not extend beyond the beginning of the range.
i.e. the windows were covering 2016-01-25, which had no test data generated.
(Does not matter for the calendar based rolls, but is needed for volume
based rolls.)
Also, make room for having the first roll to be a day before the first auto
close by moving the first contracts auto close date back a day.
In preparation for testing volume rolls.
The last traded dt provided from the session bar reader which resamples
from minutes should provide a dt that is a session label, not one that
is at the minute frequency.
If a KeyError occurred in the adjustment logic, the exception would be
swallowed by the try block, which was intended to just check whether or
not there was an adjustment reader adjusted.
Discovered when some logic in a futures adjustment reader were failing
because of a mismatch of minute and session labels, which resulted in no
adjustments during windows when there should have been.
The minute to session sampling reading was creating two DataFrame
objects, the first to hold the minute data, and then a second returned
by the `DataFrame.groupby` to sample down to sessions.
Instead use the arrays returned by the minute readers `load_raw_arrays`
and implement sampling logic which takes advantage that the minutes
being passed start with the first minute of the first session and end
with the last minute of the last session.
On my machine this takes the tests in `test/test_continuous_futures`
from ~4.0 to about ~0.1 seconds.
Add `.adj('mul')` and `.adj('add')` methods on ContinuousFuture, which
when used with `history`, will calculate and apply adjustments so that
the values are adjusted to account for discounts and premiums during
rolls.
Example usage in an algo:
```
from zipline.api import continuous_future
def initialize(context):
context.cl_add = continuous_future('CL', offset=0, roll='calendar').adj('add')
context.cl_mul = continuous_future('CL', offset=0, roll='calendar').adj('mul')
context.cl = continuous_future('CL', offset=0, roll='calendar')
schedule_function(print_history)
def print_history(context, data):
frame = data.history([context.cl, context.cl_add, context.cl_mul],
['price', 'sid'],
20,
'1d')
print 'unadjusted'
print frame.loc[:, :, context.cl]
print 'adjusted add'
print frame.loc[:, :, context.cl_add]
print 'adjusted mul'
print frame.loc[:, :, context.cl_mul]
```
Include minutes (in addition to the days) in the price encoding for
continuous futures tests.
Need for different values minute to minute arose when working on tests
for adjusted values.
Start making the equity adjustments calculations for the history loader
conform to the same method signature as `load_adjustments` provided by
`SQLiteAdjustmentReader, so that an `AdjustmentReader` interface can
begin to take form.
This prepares for creating a `DispatchAdjustmentReader` which will route
adjustment calculations for equities to the
`HistoryCompatibleUSEquityAdjustmentReader` and continuous futures to a
not yet implemented adjustment reader. All of these readers will share
the `load_adjustments` method.
Limit the perspective offset to 1. There is a possibility that if a
consumer of the AdjustedArrayWindow does not fetch adjustments between
the end of the data window and the vantage points beyond the end of the
window.
Until that case has a solution, e.g. having the consumer of the
AdjustedArrayWindow include the perspective offset when calculating the
query for adjustments, limit the offsets to 1.
- Refactor `test_adjusted_array` to test a range of perspective_offsets in
all tests.
- Make perspective_offset a parameter to `AdjustedArray.traverse`
instead of `AdjustedArray`.
Add a perspective offset to `AdjustedArrayWindow` and `AdjustedArray`,
so that `HistoryLoader` does not need to twiddle with offsets to support
viewing the data from the bar after end of the window, (Which is the
case when a '1d' history window is retrieved in minute mode, which is
explained in the docstring for `HistoryLoader.history`)
Presently, this simplifies the logic in
`HistoryLoader._get_adjustments_in_range`, and other incoming
AdjustmentReader's, (e.g. the roll based adjustment reader for continous
futures.) This patch should also make it easier for history and pipeline
to converge on a singular `load_adjustments` method.
Enable unadjusted history for continuous futures.
The history array is filled by the values for the underlying contracts,
where the contract used changes based on rolls.
e.g., if a `1d` history window was over the range
`2016-01-20` -> `2016-02-29` with contracts with a suffix of `F16` that
rolls at the beginning of the session on `2016-01-26`, `G16` on
`2016-02-26`, and `H16` on `2016-03-26`. The `2016-01-20` ->
`2016-01-25` portion would use the values for `F16', the `2016-01-26` ->
`2016-02-25` portion would use `G16` and the `2016-02-26` ->
`2016-02-29` portion would use `H16`.
Using the same contracts as above, a `1m` history window over the range
(using a timezone of US/Eastern) `2016-01-25 4:00PM` -> `2016-01-25
7:00PM` would fill the `4:00PM` -> `6:00PM` portion with data for `F16`
and the `6:01PM` -> `7:00PM` portion with data for `G16`, since the
beginning of the `2016-01-26` session is `2016-01-25 6:01PM`.
Supports `1d` and `1m`.
Also adds the `sid` field to `history` to assist in showing the active
contract at each dt in the window.
Allows ``__funcname`` to be passed to preprocessors like expect_types
and expect_dtypes to override the name displayed in error messages.
This is useful for providing clearer errors for ``__init__`` and
``__new__`` methods in classes.
Add `chain`field to current, as well as supporting methods in DataPortal
and OrderedContracts.
Enables the following example:
```
from zipline.api import continuous_future
def initialize(context):
context.primary_cl = continuous_future('CL', offset=0, roll='calendar')
schedule_function(print_current_chain)
def print_current_chain(context, data):
chain = data.current_chain(context.primary_cl)
print 'datetime={0}'.format(get_datetime())
print 'primary={0}'.format(chain[0])
print 'secondary={0}'.format(chain[1])
print 'tertiary={0}'.format(chain[2])
```
```
datetime=2015-12-23 14:31:00+00:00
primary=Future(1058201602 [CLG16])
secondary=Future(1058201603 [CLH16])
tertiary=Future(1058201604 [CLJ16])
```
Also:
- make return types of OrderedContracts methods compatible across
architectures. (Noticed while adding `active_chain` method.)
- Add year suffix to future contract names in test data.
Add the ability for an algorithm to request the current contract for a
future chain via `data.current`.
e.g.:
```
data.current(ContinuousFuture('CL', offset=0, roll='calendar'),
'contract')
```
This allows optionally setting the last available dts in the DataPortal
explicitly. If these args aren't provided, we fall back to inferring
these from the underlying readers, which was the previous behavior.
When dispatching to sub readers in dispatch reader, pass along the asset
object, instead of extracting the sid.
The in development reader for continuous futures values besides `sid`
are needed from the `ContinuousFuture` object.
In days_at_time, use a Timedelta instead of a DateOffset. We were
previously using DateOffset to work around a pandas 16 bug even though
it raises a PerformanceWarning. Now that we're on pandas 18, we can use
the much simpler Timedelta construction.
- Use parameter_space instead of `parameterized.expand`.
- Use a timedelta instead of concatenating strings.
- Use a (possibly no-op) scramble function instead of reordering list
literals.
- Use `freeze_dt, unfreeze_dt, re_freeze_dt` instead of `dts[n]`.
- Rename `assert_vectorized_results` to `assert_all_restrictions`.
In favor of a new method `set_restrictions` which takes a Restrictions
object. Calls to `set_do_not_order_list` should raise a deprecation
warning and create an equivalent Restrictions object, with which
`set_restrictions` will be called. For convenience, create a
RestrictionsSet from which the "restrictions" version of a security
list can be accessed
Introducing a WithCreateBarData fixture which allows for the
creation of a BarData using only the `simulation_dt_func` and
`restrictions` params. Assumes that each suite uses the same
`data_portal`, `data_frequency` and `trading_calendar`
The SecurityList implements a non-exposed method
`current_securities(dt)` which SecurityListRestrictions calls to
determine if an asset is restricted. Deprecate the `__iter__` and
`__contains__` methods of security lists in favor of
`current_securities(dt)`
An ABC Restrictions defines a group of restrictions responsible
for returning restriction information for sids on certain dts. An
InMemoryRestrictions is a point-in-time group of such restrictions,
with all restrictions and their dates passed in upon instantiation.
A StaticRestrictedList takes a list of sids, restricting them at all
dates
MAINT: optimization - only look at assets appearing in data
TST: simplify test
DOC: add documentation for checkpoints
MAINT: explicitly cast event date field to datetime
MAINT: add back import
TST: fix indexing to remove setting wtih copy warning
TST: fix quarter normalization test
TST: change test name
BUG: remove arg
BUG: look at dict keys
TST: add test for windowing
MAINT: raise ValueError instead of asserting
TST: add assertion to check windowing
TST: parametrize test over number of quarters forward/back.
BUG: fix adjustment calculation logic for quarter crossovers.
TST: add test for previous quarter windows
BUG: fix bugs in calculating previous windows
BUG: fix missing value for datetime
TST: add test case for missing quarter
`data.loader.ensure_benchmark_data()` was trying to use data after an exception was raised loading it. The code was logging and swallowing exceptions; this re-raises.
With the addition of the truncate function, there are cases where we'll
want to construct a BcolzMinuteBarWriter to call truncate, without
gathering all the metadata. This commit adds a write_metadata arg to its
init, which is True by default. If False is specified, no metadata is
written.
Requires adding logic to truncate to update end_session in metadata to
the truncate date.
`future_chain` will be replaced by the as yet to be implemented method,
`data.current_chain`
Also removing `FutureChain` which will be replaced by another version
which only supports indexing and iteration.
Pandas 0.18 doesn't like having null-ish values in categoricals. Fixing
this properly requires re-thinking the semantics for missing_value on
pipeline terms, so we're punting on that until after we've upgraded to
0.18.
Pandas 0.18 deprecated passing "null-ish" values to pd.categorical. The
expectation, instead, is that you use categorical's native support for
missing data, which means the user will always get NaN's for missing
entries of the categorical.
A follow-up to this change should probably drop support for custom
missing values entirely and to use LabelArray/categorical for integer
data.
In pandas 0.18, the behavior of ``nth()`` changed so that Grouper no
longer can be easily used to recover group labels.
Instead of using the built-in grouper behavior, we use a groupby on two
arrays we build ourselves. This recovers the original behavior, and is
about 2x faster as a bonus.
- Fixes a warning on indexing with a float that ultimately came from
pd.Timedelta.total_seconds(). Adds ``timedelta_to_integral_seconds``
and ``timedelta_to_integral_minutes()`` functions and replaces various
usages of ``int(delta.total_seconds())`` with them.
- Fixes a warnings triggered in ``_create_daily_stats`` from
passing tz-aware datetimes to np.datetime64.
This reverts commit 86c7635b45, reversing
changes made to c77f2b92df.
Some real world cases hit errors with this change, due to the new offset
logic attempting to create Adjustments with invalid parameters.
Will identify exact conditions that cause this error and add as a test
case before remerging.
One year NYSE test that buys a lot triggers 492,963 calls to
minute_to_session_label. Only 98924 ~(390 * 252) make it past the
cache and trigger the heavier computation.
Instead of `HistoryLoader` containing separate adjustment calculation
logic, use `SQLiteAdjustmentReader.load_adjustments`.
This change required the addition of two offset parameters to
`load_adjustments` since the perspective on the data from within
`schedule_function` is skewed from how Pipeline looks at historical
data.
This is working towards creating an `AdjustmentReader` abc which
`SQLiteAdjustmentReader` and a upcoming continuous future adjustment
reader will share.
Remove module scope invocations of `get_calendar('NYSE')`, which cuts
zipline import time in half on my machine. This make the zipline CLI
noticeably more responsive, and it reduces memory consumed at import
time from 130MB to 90MB.
Before:
$ time python -c 'import zipline'
real 0m1.262s
user 0m1.128s
sys 0m0.120s
After:
$ time python -c 'import zipline'
real 0m0.676s
user 0m0.536s
sys 0m0.132s
This reverts commit 5b1aa5ec55.
The paradigm is: we're calculating a new capital base for the
performance period. We are therefore using the total
portfolio_value, not just the cash, to calculate the
difference from the specified target as the algorithm
has meaningful holdings.
They're not meaningful, and they cause warnings from numpy.
Implemented in terms of a new preprocessor, `expect_bounded`, which
takes a tuple of `upper_bound` and `lower_bound`.
test_resample now fully covers the resample module.
Fix a bug exposed by increased coverage, where daily aggregation on
`high` would return `nan` for an asset instead of 1) during the
course of day `1d` history was called on non-consecutive minutes and 2)
either, a) the value for the previously inspected dt was `nan` or b)
there were only `nan`s between the previous and current dt.
`low` had a similar bug which was only triggered if the value for the
previously inspected dt was `nan`.
If subclasses have additional capital change information that
is required to correctly calculate the target values for
cash capital changes, it can now be provided via
"portfolio_value_adjustment".
Increase coverage on `ReindexSessionBarReader` so that all methods which
are considered part of the interface are covered by `test_resample`.
Fix bug in `get_value`, exposed by increased coverage, where the
`NoDataOnDate` exception was bubbling from the bcolz reader all the way
up when a session which was a holidy on the underlying reader was passed
to the reindex reader. (The reindex reader should return nan/0 in that
case.)
Also, move location of data index exceptions so that they are agnostic
to bcolz/us_equity_pricing; since the exception is now used by the
resample module to fix aforementioned bug.
BarData now takes the trading calendar as a parameter.
can_trade now checks if the asset’s exchange is open at the current or
next market minute (defined by the given trading calendar).
Remove special handling for the last session of an asset, which was
moving the last traded back a session.
If the asset has data on a session, `get_last_traded_dt` should always
return that session if it is the parameter to the method.
No longer auto-updates its internal as-of date, instead requires an explicit
as-of date from the consumer.
Take a static list of contracts (instead of needing an assetfinder).
Instead of the as_of method, the user-facing API now lets you pass in an
offset, which is defined as an integral number of sessions.
Add direct coverage on last_available_dt.
Also move reader creation into the instance fixture.
This patch attempted to add coverage on `get_last_traded_dt`, but in doing
so, revealed a bug in `BcolzDailyBarReader.get_last_traded_dt` when
requesting the last trading session of an asset.
When that is fixed, the skip can be removed.
Add a test to directly cover the first_trading_day method via the
`test_resample` suite. (The lack of coverage was exposed when testing
against real data.)
Also, refactor resample bar tests so that session bar reader is set up
in instance fixture.
`1d` history calls were failing on key errors when using the
`us_futures` calendar, because of timestamps occuring before a midnight
would present the wrong midnight (i.e. the midnight before the session,
instead of the following midnight, which is the label for the current
session.)
Tests will follow when bringing up coverage on resample and data portal
modules.
For example, to prevent `alpha` from internally calling
`beta` a second time, pass the previously-calculated
`beta` value in.
Requires empyrical 0.1.10 from pypi
Combine the equity and future readers into asset dispatch readers, so
that simulations that use both asset types can access data for each.
This patch enables `history` for future assets in algorithms; however,
it does not add extra coverage in the `test_data_portal` or `test_history`
to cover future assets. Those tests will follow, however putting this in
separately since it shows that the wrapping of the readers in the asset
dispatch reader does not break existing equity strategies.
Add `AssetDispatchSessionBarReader` and corresponding minute and session
bar version of that reader.
This reader routes requests to the appropriate reader based on the asset
type of the requested sids.
`load_raw_array` in the dispatch reader batches the sid by asset type
and then interleaves the results in the out arrays, so that the arrays
data corresponds with sids in the order that sids are passed to the
method, to meet the expected behavior of `load_raw_arrays`.
The dispatch redaer is intended for use by the data portal when using
both future and equities. The dispatch reader will also be passed to the
to the `HistoryLoader`s contained within the data portal, where the
batched `load_raw_arrays` will be used.
Also, BUG:
- Fix the return of `MinuteResampleSessionBarReader.load_raw_arrays` to
match all other readers.
- Use the input dt for the `MinuteResampleSessionBarReader.load_raw_arrays`
as a session label, instead of a minute dt, since it is a session bar
reader.
(Both of these bugs where discovered when using the resample reader for
future data in the dispatch tests.)
Working towards history results which contain mixed asset types, add
a reader which makes `load_raw_arrays` return results indexed on the
session/minute ranges specified by the specified `trading_calendar`
instead of the calendar of the backing reader.
This reader will be used to make Equity readers align with Future
readers. It is intended for use as part of another reader (which will
dispatch queries based on asset type and then recombined results) which
will be passed to the `[Minute|Session]HistoryLoaders in the data portal.
For scaling up pricing data before writing to bcolz, the writer now
accepts a dict mapping each sid to the ratio to use. It still accepts a
single ratio as default_ohlc_ratio, which is used as a fallback if no
mapping exists for a given sid. The default is OHLC_RATIO (1000).
This allows better handling of futures pricing data, where the required
precision across root symbols is not consistent.
The daily/session bar reader's `spot_price` took the same parameters and
returned the same kind of output as the minute bar reader's `get_value`.
Standardize on one method to make a common interface, which may be
formally factored out in a later patch; to help enable writing reader
implementations or mixins which can be agnostic to the bar frequency.
Use the equity calendar to write equity data, even when the simulation
calendar has been set to a different calendar.
Discovered when writing a test that used a calendar for future asset
data, but also wrote equity data.
We were mistakenly using the minute_per_day field.
We now expose from the metadata object the version from which the
metadata was read. This allows a new test that verifies the version is
read correctly.
The new TradingCalendar method is called `minute_index_to_session_labels`.
It takes a DatetimeIndex of in-order market minutes and returns a
DatetimeIndex of the corresponding sessions.
The new method is approximately 100x faster than mapping
`minute_to_session_label` over a large DatetimeIndex.
In the data portal, remove methods that make a distinction between
future and equity asset type. Instead rely on the pricing reader
dispatching.
In support of incoming work which will upsample equity history arrays to
the larger future calendar.
Also, remove perf tracker tests which were using an equity
reader/writer, to be added back in later.
* First pass.
* Improvements and fixes
- Update usages of BcolzMinuteBarWriter
- Updates with rebuilt example data
- Expose calendar from BcolzMinuteBarMetadata instead of calendar_name
- Keep market_opens and market_closes in metadata for compatibility
* Store start_session and end_session in minute bcolz metadata
- start_session replaces first_trading_day
- Add end_session to limit to correct days
* For last_available_dt, get last close from calendar to maintain tz
* Bumps version and handles earlier versionson read
* Rebuilt example data on python 3
* Indicate metadata fields that are deprecated
Implement a `SessionBarReader` which uses a minute bar reader as a
backing source, resampling the minute bars into the box around the
corresponding session data.
Also, add future/CME test cases to resample suite.
Adds a new ``downsample`` method to all computable terms. Computable
terms (Filters, Factors, and Classifiers) can be downsampled to yearly,
quarterly, monthly, or weekly frequency.
The result of ``term.downsample`` is a new term of the same
family (Filter/Factor/Classifier) as ``term``. The downsampled term
computes by delegating to the original term; repeatedly calling its
``compute`` method with length-1 date ranges.
Downsampled terms take advantage of a new ``compute_extra_rows`` Term
method, which allows terms to dynamically request that additional extra
rows of themselves be computed based on the dates for which they're
being computed. This ensures, for example, that a monthly-downsampled
term always computes at the start of a month, even when a
naively-calculated pipeline window would end in the middle of the month.
- Split out extra_rows handling into an `ExecutionPlan` subclass.
`ExecutionPlan` now requires the dates and calendar against which a
set of terms will be computed, and now defers to a term's
`compute_extra_rows` method when deciding how many extra rows are
required to compute for that term. This will allow downsampled terms
to request enough extra rows to guarantee that we can maintain consistent
calculation dates.
As a consequence of the above, `TermGraph` now only deals with logical
dependencies, not with metadata surrounding extra row calculations.
This means that TermGraph can be used to generate dependency
visualizations in interactive contexts where we don't yet have a
calendar or start/end dates.
- Refactored test_{filter,factor,classifier} to use check_terms instead
of run_graph. This makes it easier to make changes to TermGraph,
since the testing interface is now to simply provide a dict of terms.
- Refactored BasePipelineTestCase to use fixtures to create an asset
finder. This fixes a potential leak of the test's asset db, which was
not being explicitly cleaned up.
- Refactored test_technical to use BasePipelineTestCase.
- Added a new special term, `InputDates()`, which can be used to request
date labels for inputs. Like `AssetExists`, `InputDates` is provided
in the initial workspace by default.
- Added a default (failing) `_compute` method to `AssetExists` which
provides a more useful error than AttributeError.
Replace `DailyBarReader` with `SessionBarReader`.
This was intended to go with the patch that added the `SessionBarReader`
abstract base class.
Also, added `trading_calendar` property decorator.
* MAINT: Use TradingCalendar objects for bundles
Instead of trading days, opens, and closes, register now takes a
TradingCalendar object, along with a start_session and end_session. The
ingest function is now passed these values instead as well.
* Accept calendar name in addition to the actual object
* Updates bundles documentation for changes
* Fix typo in docs
* Use class formatting
* Force start_session and end_session within the bounds of the calendar
* Use UTC timestamps in test_core
* Document Trading Calendar API in appendix.rst
In preparation for making a resampling session bar reader, make an
abstract base class with the methods currently used by consumers of the
reader; which should assist in making a drop-in replacement of the daily
bar reader.
While pulling out the interface, it does appear that `spot_price` and
the minute bar reader's `get_value` are the same method but by different
names, showing that there may be room for having both `MinuteBarReader`
and `SessionBarReader` sharing a common `BarReader` interface.
Also, move `DailyHistoryAggregator` to `resample` module, so that tools
for converting from minute to session bars are collocated.
This patch is in preparation of adding a daily bar reader which
resamples minute data, which will be located in the `resample` module
and share the test cases and expected results in `test_resample`.
* BUG: Fixes asset writer to the select the latest asset to hold a sid
When constructing the asset_info dataframe, we were previously taking
the first symbol/sid pair to include, when we should be taking the most
recent.
* Ensure groups are sorted by increasing end_date
* Updates test_lookup_symbol_change_ticker to also cover asset_name
# Event 0 is SESSION_START which always happens at 00:00.
ts,event_type=events[1]
pass
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
fredfortier