ml-finance-python

notebook.ipynb

(11668B)

 {
  "cells": [
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "#Exercises: Spearman Rank Correlation\n",
     "\n",
     "## Lecture Link\n",
     "\n",
     "This exercise notebook refers to this lecture. Please use the lecture for explanations and sample code.\n",
     "\n",
     "https://www.quantopian.com/lectures#Spearman-Rank-Correlation\n",
     "\n",
     "Part of the Quantopian Lecture Series:\n",
     "\n",
     "* [www.quantopian.com/lectures](https://www.quantopian.com/lectures)\n",
     "* [github.com/quantopian/research_public](https://github.com/quantopian/research_public)"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false,
     "deletable": true,
     "editable": true
    },
    "outputs": [],
    "source": [
     "import numpy as np\n",
     "import pandas as pd\n",
     "import scipy.stats as stats\n",
     "import matplotlib.pyplot as plt\n",
     "import math"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "#Exercise 1: Finding Correlations of Non-Linear Relationships\n",
     "\n",
     "##a. Traditional (Pearson) Correlation\n",
     "\n",
     "Find the correlation coefficient for the relationship between `x` and `y`."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false,
     "deletable": true,
     "editable": true
    },
    "outputs": [],
    "source": [
     "n = 100\n",
     "x = np.linspace(1, n, n)\n",
     "y = x**5\n",
     "\n",
     "#Your code goes here"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "# b. Spearman Rank Correlation\n",
     "\n",
     "Find the Spearman rank correlation coefficient for the relationship between `x` and `y` using the `stats.rankdata` function and the formula \n",
     "\n",
     "$$r_S = 1 - \\frac{6 \\sum_{i=1}^n d_i^2}{n(n^2 - 1)}$$\n",
     "\n",
     "where $d_i$ is the difference in rank of the `i`th pair of `x` and `y` values."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false,
     "deletable": true,
     "editable": true
    },
    "outputs": [],
    "source": [
     "#Your code goes here"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "Check your results against scipy's Spearman rank function. `stats.spearmanr`"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false,
     "deletable": true,
     "editable": true
    },
    "outputs": [],
    "source": [
     "# Your code goes here"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "#Exercise 2: Limitations of Spearman Rank Correlation\n",
     "\n",
     "##a. Lagged Relationships\n",
     "\n",
     "First, create a series `b` that is identical to `a` but lagged one step (`b[i] = a[i-1]`). Then, find the Spearman rank correlation coefficient of the relationship between `a` and `b`.\n"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false,
     "deletable": true,
     "editable": true
    },
    "outputs": [],
    "source": [
     "n = 100\n",
     "a = np.random.normal(0, 1, n)\n",
     "\n",
     "#Your code goes here"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "##b. Non-Monotonic Relationships\n",
     "\n",
     "First, create a series `d` using the relationship $d=10c^2 - c + 2$. Then, find the Spearman rank rorrelation coefficient of the relationship between `c` and `d`."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false,
     "deletable": true,
     "editable": true
    },
    "outputs": [],
    "source": [
     "n = 100\n",
     "c = np.random.normal(0, 2, n)\n",
     "\n",
     "#Your code goes here"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "#Exercise 3: Real World Example\n",
     "\n",
     "##a. Factor and Forward Returns\n",
     "\n",
     "Here we'll define a simple momentum factor (model). To evaluate it we'd need to look at how its predictions correlate with future returns over many days. We'll start by just evaluating the Spearman rank correlation between our factor values and forward returns on just one day.\n",
     "\n",
     "Compute the Spearman rank correlation between factor values and 10 trading day forward returns on 2015-1-2.\n",
     "\n",
     "For help on the pipeline API, see this tutorial: https://www.quantopian.com/tutorials/pipeline"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false,
     "deletable": true,
     "editable": true
    },
    "outputs": [],
    "source": [
     "#Pipeline Setup\n",
     "from quantopian.research import run_pipeline\n",
     "from quantopian.pipeline import Pipeline\n",
     "from quantopian.pipeline.data.builtin import USEquityPricing\n",
     "from quantopian.pipeline.factors import CustomFactor, Returns, RollingLinearRegressionOfReturns\n",
     "from quantopian.pipeline.classifiers.morningstar import Sector\n",
     "from quantopian.pipeline.filters import QTradableStocksUS\n",
     "from time import time\n",
     "\n",
     "#MyFactor is our custom factor, based off of asset price momentum\n",
     "\n",
     "class MyFactor(CustomFactor):\n",
     "        \"\"\" Momentum factor \"\"\"\n",
     "\n",
     "        inputs = [USEquityPricing.close] \n",
     "        window_length = 60\n",
     "\n",
     "        def compute(self, today, assets, out, close):       \n",
     "            out[:] = close[-1]/close[0]\n",
     "            \n",
     "universe = QTradableStocksUS()\n",
     "\n",
     "pipe = Pipeline(\n",
     "    columns = {\n",
     "            'MyFactor' : MyFactor(mask=universe),\n",
     "    },\n",
     "    screen=universe\n",
     ")\n",
     "\n",
     "start_timer = time()\n",
     "results = run_pipeline(pipe, '2015-01-01', '2015-06-01')\n",
     "end_timer = time()\n",
     "results.fillna(value=0);\n",
     "\n",
     "print \"Time to run pipeline %.2f secs\" % (end_timer - start_timer)\n",
     "\n",
     "my_factor = results['MyFactor']"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false,
     "deletable": true,
     "editable": true
    },
    "outputs": [],
    "source": [
     "n = len(my_factor)\n",
     "\n",
     "asset_list = results.index.levels[1].unique()\n",
     "prices_df = get_pricing(asset_list, start_date='2015-01-01', end_date='2016-01-01', fields='price')\n",
     "\n",
     "# Compute 10-day forward returns, then shift the dataframe back by 10\n",
     "forward_returns_df = prices_df.pct_change(10).shift(-10)\n",
     "\n",
     "# The first trading day is actually 2015-1-2\n",
     "single_day_factor_values = my_factor['2015-1-2']\n",
     "\n",
     "# Because prices are indexed over the total time period, while the factor values dataframe\n",
     "# has a dynamic universe that excludes hard to trade stocks, each day there may be assets in \n",
     "# the returns dataframe that are not present in the factor values dataframe. We have to filter down\n",
     "# as a result.\n",
     "single_day_forward_returns = forward_returns_df.loc['2015-1-2'][single_day_factor_values.index]\n",
     "\n",
     "#Your code goes here"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "##b. Rolling Spearman Rank Correlation\n",
     "\n",
     "Repeat the above correlation for the first 60 days in the dataframe as opposed to just a single day. You should get a time series of Spearman rank correlations. From this we can start getting a better sense of how the factor correlates with forward returns.\n",
     "\n",
     "What we're driving towards is known as an information coefficient. This is a very common way of measuring how predictive a model is. All of this plus much more is automated in our open source alphalens library. In order to see alphalens in action you can check out these resources:\n",
     "\n",
     "A basic tutorial:\n",
     "https://www.quantopian.com/tutorials/getting-started#lesson4\n",
     "\n",
     "An in-depth lecture:\n",
     "https://www.quantopian.com/lectures/factor-analysis"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false,
     "deletable": true,
     "editable": true
    },
    "outputs": [],
    "source": [
     "rolling_corr = pd.Series(index=None, data=None)\n",
     "\n",
     "#Your code goes here"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "##b. Rolling Spearman Rank Correlation\n",
     "\n",
     "Plot out the rolling correlation as a time series, and compute the mean and standard deviation."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
     "collapsed": false,
     "deletable": true,
     "editable": true
    },
    "outputs": [],
    "source": [
     "# Your code goes here"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "---\n",
     "\n",
     "Congratulations on completing the Spearman rank correlation exercises!\n",
     "\n",
     "As you learn more about writing trading models and the Quantopian platform, enter a daily [Quantopian Contest](https://www.quantopian.com/contest). Your strategy will be evaluated for a cash prize every day.\n",
     "\n",
     "Start by going through the [Writing a Contest Algorithm](https://www.quantopian.com/tutorials/contest) tutorial."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {
     "deletable": true,
     "editable": true
    },
    "source": [
     "*This presentation is for informational purposes only and does not constitute an offer to sell, a solic\n",
     "itation to buy, or a recommendation for any security; nor does it constitute an offer to provide investment advisory or other services by Quantopian, Inc. (\"Quantopian\"). Nothing contained herein constitutes investment advice or offers any opinion with respect to the suitability of any security, and any views expressed herein should not be taken as advice to buy, sell, or hold any security or as an endorsement of any security or company.  In preparing the information contained herein, Quantopian, Inc. has not taken into account the investment needs, objectives, and financial circumstances of any particular investor. Any views expressed and data illustrated herein were prepared based upon information, believed to be reliable, available to Quantopian, Inc. at the time of publication. Quantopian makes no guarantees as to their accuracy or completeness. All information is subject to change and may quickly become unreliable for various reasons, including changes in market conditions or economic circumstances.*"
    ]
   }
  ],
  "metadata": {
   "kernelspec": {
    "display_name": "Python 2",
    "language": "python",
    "name": "python2"
   },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
     "version": 2
    },
    "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython2",
    "version": "2.7.12"
   }
  },
  "nbformat": 4,
  "nbformat_minor": 0
 }