Referencing context.account in handle_data() can block updates to account,
portfolio and performance metrics, which can cause unintended changes in
financial performance. Additionally, since context.account metrics are not
updating frequenctly enough (daily, rather than minutely) algos which base
decisions on these metrics can have undesired behavior.
To address this we do not base our decision to update performance on the
state of account_needs_update.
When we call get_account() from the market and minute close methods, we
set the performance_needs_update parameter to False since we just updated
performance.
_get_account() is renamed to _update_account() since the method does not
return account information.
Also remove redundant asset_needs_update = True lines.
By having both the trade simulation main loop route events to "process"
methods based on event type and the process methods also checking event
type, there was some duplicated effort in doing that comparison many
times.
A particular case where this was noted in profiling was for the
`process_event` function which was checking if the type was not a trade
and returning early, when in a larger universe of stocks the value
returned False 99% of the time.
Instead provide separate process functions specific to each type,
e.g. e.g. `process_trade` and `process_transaction` and route traffic to
those functions in tradesimulation.
For a universe of 160 stocks on both no-op algo and an algo that rebuys
its universe every day, saw about a 10% increase locally.
Also:
- Add process_benchmark to blotter since internal subclass relies on
logic on benchmark, this allows the internal process_trade to be a
`pass`.
- Add warning on unrecoginzed event types.
The risk containers that are actually used for reports use the
'cumulative' style container which has an index of days, not minutes.
The minute containers and copying of data etc. were causing an expanding
memory footprint.
The intraday_risk_metrics is being removed since the values are not
used; cumulative risk metrics with the last value updated to the latest
close has been used for some time.
Before the removal of intraday_risk_metrics, the position trackers
passing of benchmark returns to the cumulative risk metrics needs to no
longer depend on the calculations done by the intraday stats. So instead
use the all_benchmark_returns stored in the tracker directly.
Remove use of defaultdict for orders_by_modified, which was causing an
empty list to be added every time to_dict was called with a specified
dt.
Nnoticed in the minute emission case when hunting another memory leak,
every simulation minute a new Timestamp and list was created and never
let go.
on the number of per-tick update that occur since they were duplicated
per each PerformancePeriod. Also opens up the path to cythonizing the
entire object
Previously the class SerializeableZiplineObject was used to
house basic __setstate__ and __getstate__ methods. It wasn't
really doing much that was helpful, so it is now gone.
For use in a function that wraps de-serialization, to call instead of
creating the OrderedDicts from a module outside of the object.
So that the other module does not need to the internals of this object,
also to ensure that the cythonized OrderedDict is used, when available.
This need should be superseded with serialization versioning.
When calculate_positions_value used np.dot, the return type was a
np.float64. Which allows the use of 0.0 in division to not raise an
exception.
Fix by expliciting creating an np.float64 with 0 value.
Add a set_positions method so that the serialization process can rebuild
from just the positions, since the last_sale and amounts are derivable
from those values.
Also, use the private naming convention for last sale price and amount
members, so that those members are ignored by the serialization process.
based OrderedDict. Series/ndarray can only be sped up so much because
they weren't designed for fast iterative mutations.
This also cut down on the # of intermediate Series being generated
during perf stat generation. Things like s[s > 0] will create a new
Series for s > 0.
Moved cython to requirements.txt and added cyordereddict
Alleviates bottleneck caused re-indexing into a pd.Series during a tight
loop, by keeping track of the index value into the underlying `.values`
in a lookup table.
Based on suggestion from @dalejung
Instead of checking the positions indexes every time either
`_position_amounts` or `_position_last_sale_prices` is updated, check
and grow the individual Series on each update.
This gain with this patch is by reducing the following bottlenecks:
- Checking both vectors when only one is updated.
- Using try/except to trigger the growth, instead of incurring the cost
of checking the Index contains on every update.
In testing this change results in about a 33% speedup of the
`update_last_sale` algorithm when run with a buy and hold algorithm with
160 equities, resulting in a 20% speedup overall.
Previously the last sale price was not correctly being set on
positions when the transaction arrived before the trade event.
The last sale price was defaulted to zero and never updated. This resulted
in one holding stocks that were bough >>0 and now had value 0 from
the perspective of returns. The returns would display correctly again
when the next trade of that security happened. For most securities trading is
frequent enough that there's no issue, but for some illiquid ones it took
hours to fix itself.
Updated test_perf_tracking:TestPerformanceTracker.test_minute_tracker
This test was based on assuming that last_sale_price was zero,
allowing the sharpe ratio to be calculated. The sharpe ratio can no longer
be calculated for this specific tested scenario and the test has been changed
accordingly.
jfkirk