#. View the Compressed Pricing Data in Redis

::

redis-cli keys "SPY_*" redis-cli get "<key like SPY2019-01-08minute>"

Run Backtests with the Algorithm Runner API ===========================================

Run a backtest with the latest pricing data:

.. code-block:: python

import analysisengine.algorunner as algo_runner import analysisengine.plottrading_history as plot runner = algo_runner.AlgoRunner('SPY') # run the algorithm with the latest 200 minutes: df = runner.latest() print(df[['minute', 'close']].tail(5)) plot.plottradinghistory( title=( f'SPY - ${df["close"].iloc[-1]} at: ' f'{df["minute"].iloc[-1]}'), df=df) # start a full backtest with: # runner.start()

Extract from Redis API ======================

Once fetched, you can extract datasets from the redis cache with:

.. code-block:: python

import analysisengine.extract as aeextract print(ae_extract.extract('SPY'))

Extract Latest Minute Pricing for Stocks and Options ====================================================

.. code-block:: python

import analysisengine.extract as aeextract print(ae_extract.extract( 'SPY', datasets=['minute', 'tdcalls', 'tdputs']))

Extract Historical Data

.. code-block:: python

import analysisengine.extract as aeextract print(ae_extract.extract( 'AAPL', datasets=['minute', 'daily', 'financials', 'earnings', 'dividends'], date='2019-02-15'))

Additional Extraction APIs ==========================

Running the Full Stack Locally for Backtesting and Live Trading Analysis ========================================================================

While not required for backtesting, running the full stack is required for running algorithms during a live trading session. Here is a video on how to deploy the full stack locally using docker compose and the commands from the video.

.. image:: https://asciinema.org/a/220487.png :target: https://asciinema.org/a/220487?autoplay=1 :alt: Running the Full Stack Locally for Backtesting and Live Trading Analysis

#. Start Workers, Backtester, Pricing Data Collection, Jupyter, Redis and Minio

::

./compose/start.sh

#. Check the Docker Containers

::

docker ps -a

#. View for dataset collection logs

::

logs-dataset-collection.sh

::

logs-workers.sh

#. Verify Pricing Data is in Redis

::

redis-cli keys "*"

::

./compose/start.sh -c

Run a Custom Minute-by-Minute Intraday Algorithm Backtest and Plot the Trading History ======================================================================================

With pricing data in redis, you can start running backtests a few ways:

Here is a video showing how to run it:

.. image:: https://asciinema.org/a/220498.png :target: https://asciinema.org/a/220498?autoplay=1 :alt: Running an Algorithm with Live Intraday Pricing Data

Write the Trading Performance Report to a Local File

::

run-algo-report-to-file.sh -t SPY -b 60 -a /opt/sa/analysisengine/mocks/examplealgo_minute.py # run-algo-report-to-file.sh -t <TICKER> -b <NUMDAYSBACK> -a <CUSTOMALGOMODULE> # run on specific date ranges with: # -s <start date YYYY-MM-DD> -n <end date YYYY-MM-DD>

Write the Trading Performance Report to Minio (s3)

::

run-algo-report-to-s3.sh -t SPY -b 60 -a /opt/sa/analysisengine/mocks/examplealgo_minute.py

Run and Publish Trading History for a Custom Algorithm ======================================================

Write the Trading History to a Local File

::

run-algo-history-to-file.sh -t SPY -b 60 -a /opt/sa/analysisengine/mocks/examplealgo_minute.py

Write the Trading History to Minio (s3)

::

run-algo-history-to-s3.sh -t SPY -b 60 -a /opt/sa/analysisengine/mocks/examplealgo_minute.py

Developing on AWS =================

If you are comfortable with AWS S3 usage charges, then you can run just with a redis server to develop and tune algorithms. This works for teams and for archiving datasets for disaster recovery.

Environment Variables

Export these based off your AWS IAM credentials and S3 endpoint.

::

export AWSACCESSKEY_ID="ACCESS" export AWSSECRETACCESS_KEY="SECRET" export S3_ADDRESS=s3.us-east-1.amazonaws.com

Extract and Publish to AWS S3 =============================

::

./tools/backup-datasets-on-s3.sh -t TICKER -q YOURBUCKET -k ${S3ADDRESS} -r localhost:6379

Publish to Custom AWS S3 Bucket and Key =======================================

::

extractloc=s3://YOURBUCKET/TICKER-latest.json ./tools/backup-datasets-on-s3.sh -t TICKER -e ${extract_loc} -r localhost:6379

Backtest a Custom Algorithm with a Dataset on AWS S3 ====================================================

::

backtestloc=s3://YOURBUCKET/TICKER-latest.json customalgomodule=/opt/sa/analysisengine/mocks/examplealgo_minute.py sa -t TICKER -a ${S3ADDRESS} -r localhost:6379 -b ${backtestloc} -g ${customalgomodule}

Fetching New Pricing Tradier Every Minute with Kubernetes =========================================================

::

kubectl -f apply /opt/sa/k8/datasets/pulltradierper_minute.yml

Run a Distributed 60-day Backtest on SPY and Publish the Trading Report, Trading History and Algorithm-Ready Dataset to S3 ==========================================================================================================================

Publish backtests and live trading algorithms to the engine's workers for running many algorithms at the same time. Once done, the algorithm will publish results to s3, redis or a local file. By default, the included example below publishes all datasets into minio (s3) where they can be downloaded for offline backtests or restored back into redis.

.. note:: Running distributed algorithmic workloads requires redis, minio, and the engine running

::

numdaysback=60 ./tools/run-algo-with-publishing.sh -t SPY -b ${numdaysback} -w

Run a Local 60-day Backtest on SPY and Publish Trading Report, Trading History and Algorithm-Ready Dataset to S3 ================================================================================================================

::

numdaysback=60 ./tools/run-algo-with-publishing.sh -t SPY -b ${numdaysback}

Or manually with:

::

ticker=SPY numdaysback=60 use_date=$(date +"%Y-%m-%d") ds_id=$(uuidgen | sed -e 's/-//g') tickerdataset="${ticker}-${usedate}${dsid}.json" echo "creating ${ticker} dataset: ${ticker_dataset}" extractloc="s3://algoready/${tickerdataset}" historyloc="s3://algohistory/${tickerdataset}" reportloc="s3://algoreport/${tickerdataset}" backtestloc="s3://algoready/${tickerdataset}" # same as the extract_loc processedloc="s3://algoprocessed/${tickerdataset}" # archive it when done startdate=$(date --date="${numdays_back} day ago" +"%Y-%m-%d") echo "" echo "extracting algorithm-ready dataset: ${extract_loc}" echo "sa -t SPY -e ${extractloc} -s ${startdate} -n ${use_date}" sa -t SPY -e ${extractloc} -s ${startdate} -n ${use_date} echo "" echo "running algo with: ${backtest_loc}" echo "sa -t SPY -p ${historyloc} -o ${reportloc} -b ${backtestloc} -e ${processedloc} -s ${startdate} -n ${usedate}" sa -t SPY -p ${historyloc} -o ${reportloc} -b ${backtestloc} -e ${processedloc} -s ${startdate} -n ${usedate}

Jupyter on Kubernetes =====================

::

./k8/jupyter/run.sh ceph dev

Kubernetes - Analyze and Tune Algorithms from a Trading History ===============================================================

Kubernetes Job - Export SPY Datasets and Publish to Minio =========================================================

::

function ssheng() { pod_name=$(kubectl get po | grep ae-engine | grep Running |tail -1 | awk '{print $1}') echo "logging into ${pod_name}" kubectl exec -it ${pod_name} bash } ssheng # once inside the container on kubernetes source /opt/venv/bin/activate sa -a minio-service:9000 -r redis-master:6379 -e s3://backups/SPY-$(date +"%Y-%m-%d") -t SPY

View Algorithm-Ready Datasets

With the AWS cli configured you can view available algorithm-ready datasets in your minio (s3) bucket with the command:

::

aws --endpoint-url http://localhost:9000 s3 ls s3://algoready

View Trading History Datasets

With the AWS cli configured you can view available trading history datasets in your minio (s3) bucket with the command:

::

aws --endpoint-url http://localhost:9000 s3 ls s3://algohistory

View Trading History Datasets

With the AWS cli configured you can view available trading performance report datasets in your minio (s3) bucket with the command:

::

aws --endpoint-url http://localhost:9000 s3 ls s3://algoreport

Advanced - Running Algorithm Backtests Offline ==============================================

Run a Offline Custom Algorithm Backtest with an Algorithm-Ready File

::

# extract with: sa -t SPY -e file:/tmp/SPY-latest.json sa -t SPY -b file:/tmp/SPY-latest.json -g /opt/sa/analysisengine/mocks/examplealgo_minute.py

Run the Intraday Minute-by-Minute Algorithm and Publish the Algorithm-Ready Dataset to S3

::

sa -t SPY -g /opt/sa/analysisengine/mocks/examplealgo_minute.py -e s3://algoready/SPY-$(date +"%Y-%m-%d").json

::

sa -d -t SPY -g /opt/sa/analysisengine/mocks/examplealgo_minute.py -e s3://algoready/SPY-$(date +"%Y-%m-%d").json

Extract Algorithm-Ready Datasets ================================

With pricing data cached in redis, you can extract algorithm-ready datasets and save them to a local file for offline historical backtesting analysis. This also serves as a local backup where all cached data for a single ticker is in a single local file.

Extract an Algorithm-Ready Dataset from Redis and Save it to a File

::

sa -t SPY -e ~/SPY-latest.json

Create a Daily Backup

::

sa -t SPY -e ~/SPY-$(date +"%Y-%m-%d").json

Validate the Daily Backup by Examining the Dataset File

::

sa -t SPY -l ~/SPY-$(date +"%Y-%m-%d").json

Validate the Daily Backup by Examining the Dataset File

::

sa -t SPY -l ~/SPY-$(date +"%Y-%m-%d").json

Restore Backup to Redis

Use this command to cache missing pricing datasets so algorithms have the correct data ready-to-go before making buy and sell predictions.

.. note:: By default, this command will not overwrite existing datasets in redis. It was built as a tool for merging redis pricing datasets after a VM restarted and pricing data was missing from the past few days (gaps in pricing data is bad for algorithms).

::

sa -t SPY -L ~/SPY-$(date +"%Y-%m-%d").json

Fetch

.. code-block:: python

from analysis_engine.fetch import fetch d = fetch(ticker='SPY') for k in d['SPY']: print(f'dataset key: {k}\nvalue {d["SPY"][k]}\n')

Backfill Historical Minute Data from IEX Cloud ==============================================

::

fetch -t TICKER -F PASTDATE -g iexmin # example: # fetch -t SPY -F 2019-02-07 -g iex_min

.. list-table:: :header-rows: 1

Getting Started ===============

This section outlines how to get the Stock Analysis stack running locally with:

• Redis
• Minio (S3)
• Stock Analysis engine
• Jupyter