If we have a test that's being called more than once (i.e. two
test cases, both subclasses of the same base test case, with
different setup but calling the same test), allow the subsequent
calls to re-consume the same params
When EQUITY_DAILY_BAR_SOURCE_FROM_MINUTE is set, use
EQUITY_MINUTE_BAR_LOOKBACK_DAYS as the default value for
EQUITY_DAILY_BAR_LOOKBACK_DAYS.
Without this, trying to run a minutely backtest in a test setting only
EQUITY_MINUTE_BAR_LOOKBACK_DAYS and EQUITY_DAILY_BAR_SOURCE_FROM_MINUTE
fails because the benchmark creation process makes a daily history call
for the entire period of the backtest, which then fails because the
equity daily bar calendar is shorter than the equity minute bar
calendar.
I can't imagine a circumstance in which you'd want the daily bar
calendar to be shorter than the minute bar calendar when you're sourcing
daily bars from minutes, so this change makes that the default behavior
unless it's explicitly overridden.
From pep-0008:
```
Always use a def statement instead of an assignment statement that binds a
lambda expression directly to an identifier.
Yes:
def f(x): return 2*x
No:
f = lambda x: 2*x
The first form means that the name of the resulting function object is
specifically 'f' instead of the generic '<lambda>'. This is more useful for
tracebacks and string representations in general. The use of the assignment
statement eliminates the sole benefit a lambda expression can offer over an
explicit def statement (i.e. that it can be embedded inside a larger expression)
```
To support contracts such as `PL` which should roll from F->J->N->V, add the
ability to pass a predicate function to the ordered contract chain contstrution
which returns `True` if the contract is allowed in the chain.
To support using a `DataPortal` and `HistoryLoader` in a notebook, allow
the prefetch length to be configurable, so that it can be set to 0.
Unlike backtesting where the prefetch is useful for repeated history
windows viewed from datetimes which are monotonically increasing by a
small amount, the notebook usage of history windows needs only to
retrieve the exact data needed for the window specified.
This patch also fixes some boundary conditions related to rolls and
adjustments which were uncovered by querying for the adjustments with an
end date near the end of the window.
The minute to session sampling reading was creating two DataFrame
objects, the first to hold the minute data, and then a second returned
by the `DataFrame.groupby` to sample down to sessions.
Instead use the arrays returned by the minute readers `load_raw_arrays`
and implement sampling logic which takes advantage that the minutes
being passed start with the first minute of the first session and end
with the last minute of the last session.
On my machine this takes the tests in `test/test_continuous_futures`
from ~4.0 to about ~0.1 seconds.
Enable unadjusted history for continuous futures.
The history array is filled by the values for the underlying contracts,
where the contract used changes based on rolls.
e.g., if a `1d` history window was over the range
`2016-01-20` -> `2016-02-29` with contracts with a suffix of `F16` that
rolls at the beginning of the session on `2016-01-26`, `G16` on
`2016-02-26`, and `H16` on `2016-03-26`. The `2016-01-20` ->
`2016-01-25` portion would use the values for `F16', the `2016-01-26` ->
`2016-02-25` portion would use `G16` and the `2016-02-26` ->
`2016-02-29` portion would use `H16`.
Using the same contracts as above, a `1m` history window over the range
(using a timezone of US/Eastern) `2016-01-25 4:00PM` -> `2016-01-25
7:00PM` would fill the `4:00PM` -> `6:00PM` portion with data for `F16`
and the `6:01PM` -> `7:00PM` portion with data for `G16`, since the
beginning of the `2016-01-26` session is `2016-01-25 6:01PM`.
Supports `1d` and `1m`.
Also adds the `sid` field to `history` to assist in showing the active
contract at each dt in the window.
This allows optionally setting the last available dts in the DataPortal
explicitly. If these args aren't provided, we fall back to inferring
these from the underlying readers, which was the previous behavior.
Introducing a WithCreateBarData fixture which allows for the
creation of a BarData using only the `simulation_dt_func` and
`restrictions` params. Assumes that each suite uses the same
`data_portal`, `data_frequency` and `trading_calendar`
- Fixes a warning on indexing with a float that ultimately came from
pd.Timedelta.total_seconds(). Adds ``timedelta_to_integral_seconds``
and ``timedelta_to_integral_minutes()`` functions and replaces various
usages of ``int(delta.total_seconds())`` with them.
- Fixes a warnings triggered in ``_create_daily_stats`` from
passing tz-aware datetimes to np.datetime64.
The daily/session bar reader's `spot_price` took the same parameters and
returned the same kind of output as the minute bar reader's `get_value`.
Standardize on one method to make a common interface, which may be
formally factored out in a later patch; to help enable writing reader
implementations or mixins which can be agnostic to the bar frequency.
Use the equity calendar to write equity data, even when the simulation
calendar has been set to a different calendar.
Discovered when writing a test that used a calendar for future asset
data, but also wrote equity data.
* First pass.
* Improvements and fixes
- Update usages of BcolzMinuteBarWriter
- Updates with rebuilt example data
- Expose calendar from BcolzMinuteBarMetadata instead of calendar_name
- Keep market_opens and market_closes in metadata for compatibility
* Store start_session and end_session in minute bcolz metadata
- start_session replaces first_trading_day
- Add end_session to limit to correct days
* For last_available_dt, get last close from calendar to maintain tz
* Bumps version and handles earlier versionson read
* Rebuilt example data on python 3
* Indicate metadata fields that are deprecated
Implement a `SessionBarReader` which uses a minute bar reader as a
backing source, resampling the minute bars into the box around the
corresponding session data.
Also, add future/CME test cases to resample suite.
Change the mock minute data to no longer use an increasing arange, so
that a days worth of minute data can be summed and fit inside of a
uint32.
This change was required because of working on new test data that looked
like [0, 100, 200, 0, ] which was resulting in a daily rollup of 0 data,
when the coverage needed a non-0 value.
Also, factor out the resampling function, with an eye on a making it
easier to convert from minute bars to daily bars during ingest/load
processes.
When adding fixtures for futures data, there will be a need for multiple
calendars in the fixture ecosystem. e.g. a test that includes both
equities and futures would need an overall calendar which encompasses
both equities and futures; however, the test data for equities should
still still be limited to the bounds set by the NYSE calendar.
Make the fixtures that setup trading calendars and values dervied from
the trading calendar (e.g. trading sessions) accept an iterable of
calendars which need to be created, then populate those values into a
dict keyed by the calendar name.
Change `WithNYSETradingDays` to include sessions in the name,
since we are moving to session as the name for the 'day' unit.
Provide `trading_days` which is really "NYSE trading sessions` on
`WithTradingSessions` for backwards compatibility.
Andrew Daniels