ml-finance-python
python scripts for finance machine learning
git clone https://9o.is/git/ml-finance-python.git
notebook.ipynb
(11671B)
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {},
6 "source": [
7 "## Pipeline API"
8 ]
9 },
10 {
11 "cell_type": "markdown",
12 "metadata": {},
13 "source": [
14 "The Pipeline API is a powerful tool for cross-sectional analysis of asset data. It allows us to define a set of calculations on multiple data inputs and analyze a large amount of assets at a time. Some common uses for the Pipeline API include: \n",
15 "\n",
16 "* Selecting assets based on filtering rules \n",
17 "* Ranking assets based on a scoring function \n",
18 "* Calculating portfolio allocations \n",
19 "\n",
20 "To begin, let's import the Pipeline class and create a function that returns an empty pipeline. Putting our pipeline definition inside a function helps us keep things organized as our pipeline grows in complexity. This is particularly helpful when transferring data pipelines between Research and the IDE."
21 ]
22 },
23 {
24 "cell_type": "code",
25 "execution_count": 1,
26 "metadata": {},
27 "outputs": [],
28 "source": [
29 "# Pipeline class\n",
30 "from quantopian.pipeline import Pipeline\n",
31 "\n",
32 "def make_pipeline():\n",
33 " # Create and return an empty Pipeline\n",
34 " return Pipeline()"
35 ]
36 },
37 {
38 "cell_type": "markdown",
39 "metadata": {},
40 "source": [
41 "To add an output to our pipeline we need to include a reference to a dataset, and specify the computations we want to carry out on that data. For example, we will add a reference to the `close` column from the `USEquityPricing` dataset. Then, we can define our output to be the latest value from this column as follows:"
42 ]
43 },
44 {
45 "cell_type": "code",
46 "execution_count": 2,
47 "metadata": {},
48 "outputs": [],
49 "source": [
50 "# Import Pipeline class and USEquityPricing dataset\n",
51 "from quantopian.pipeline import Pipeline\n",
52 "from quantopian.pipeline.data import EquityPricing\n",
53 "from quantopian.pipeline.domain import US_EQUITIES\n",
54 "\n",
55 "def make_pipeline():\n",
56 " # Get latest closing price\n",
57 " close_price = EquityPricing.close.latest\n",
58 "\n",
59 " # Return Pipeline containing latest closing price\n",
60 " return Pipeline(\n",
61 " columns={\n",
62 " 'close_price': close_price,\n",
63 " },\n",
64 " domain=US_EQUITIES,\n",
65 " )"
66 ]
67 },
68 {
69 "cell_type": "markdown",
70 "metadata": {},
71 "source": [
72 "The Pipeline API also provides a number of built-in calculations, some of which are computed over trailing windows of data. For example, the following code imports the sentdex `sentiment` dataset and defines an output as the 3 day moving average of its `sentiment_signal` column:"
73 ]
74 },
75 {
76 "cell_type": "code",
77 "execution_count": 3,
78 "metadata": {},
79 "outputs": [],
80 "source": [
81 "# Import Pipeline class and datasets\n",
82 "from quantopian.pipeline import Pipeline\n",
83 "from quantopian.pipeline.data import EquityPricing\n",
84 "from quantopian.pipeline.domain import US_EQUITIES\n",
85 "from quantopian.pipeline.data.sentdex import sentiment\n",
86 "\n",
87 "# Import built-in moving average calculation\n",
88 "from quantopian.pipeline.factors import SimpleMovingAverage\n",
89 "\n",
90 "\n",
91 "def make_pipeline():\n",
92 " # Get latest closing price\n",
93 " close_price = EquityPricing.close.latest\n",
94 "\n",
95 " # Calculate 3 day average the sentiment signal\n",
96 " sentiment_score = SimpleMovingAverage(\n",
97 " inputs=[sentiment.sentiment_signal],\n",
98 " window_length=3,\n",
99 " )\n",
100 "\n",
101 " # Return Pipeline containing close_price\n",
102 " # and sentiment_score\n",
103 " return Pipeline(\n",
104 " columns={\n",
105 " 'close_price': close_price,\n",
106 " 'sentiment_score': sentiment_score,\n",
107 " }\n",
108 " )"
109 ]
110 },
111 {
112 "cell_type": "markdown",
113 "metadata": {},
114 "source": [
115 "### Universe Selection"
116 ]
117 },
118 {
119 "cell_type": "markdown",
120 "metadata": {},
121 "source": [
122 "An important part of developing a strategy is defining the set of assets that we want to consider trading in our portfolio. We usually refer to this set of assets as our trading universe. \n",
123 "\n",
124 "A trading universe should be as large as possible, while also excluding assets that aren't appropriate for our portfolio. For example, we want to exclude stocks that are illiquid or difficult to trade. Quantopian's `QTradableStocksUS` universe offers this characteristic. We can set `QTradableStocksUS` as our trading universe using the screen parameter of our pipeline constructor. We can also screen for only those stocks that have a valid, non nan, sentiment_mean:"
125 ]
126 },
127 {
128 "cell_type": "code",
129 "execution_count": 4,
130 "metadata": {},
131 "outputs": [],
132 "source": [
133 "# Import Pipeline class and datasets\n",
134 "from quantopian.pipeline import Pipeline\n",
135 "from quantopian.pipeline.data import EquityPricing\n",
136 "from quantopian.pipeline.domain import US_EQUITIES\n",
137 "from quantopian.pipeline.data.sentdex import sentiment\n",
138 "\n",
139 "# Import built-in moving average calculation\n",
140 "from quantopian.pipeline.factors import SimpleMovingAverage\n",
141 "\n",
142 "# Import built-in trading universe\n",
143 "from quantopian.pipeline.filters import QTradableStocksUS\n",
144 "\n",
145 "\n",
146 "def make_pipeline():\n",
147 " # Create a reference to our trading universe\n",
148 " base_universe = QTradableStocksUS()\n",
149 "\n",
150 " # Get latest closing price\n",
151 " close_price = EquityPricing.close.latest\n",
152 "\n",
153 " # Calculate 3 day average of sentiment scores\n",
154 " sentiment_score = SimpleMovingAverage(\n",
155 " inputs=[sentiment.sentiment_signal],\n",
156 " window_length=3,\n",
157 " )\n",
158 "\n",
159 " # Return Pipeline containing close_price and\n",
160 " # sentiment_score that has our trading universe as screen\n",
161 " return Pipeline(\n",
162 " columns={\n",
163 " 'close_price': close_price,\n",
164 " 'sentiment_score': sentiment_score,\n",
165 " },\n",
166 " screen=base_universe & sentiment_score.notnan(),\n",
167 " domain=US_EQUITIES,\n",
168 " )"
169 ]
170 },
171 {
172 "cell_type": "markdown",
173 "metadata": {},
174 "source": [
175 "Now that our pipeline definition is complete, we can execute it over a specific period of time using `run_pipeline`. The output will be a pandas DataFrame indexed by date and asset, with columns corresponding to the outputs we added to our pipeline definition:"
176 ]
177 },
178 {
179 "cell_type": "code",
180 "execution_count": 5,
181 "metadata": {},
182 "outputs": [
183 {
184 "data": {
185 "application/vnd.jupyter.widget-view+json": {
186 "model_id": "",
187 "version_major": 2,
188 "version_minor": 0
189 },
190 "text/plain": []
191 },
192 "metadata": {},
193 "output_type": "display_data"
194 },
195 {
196 "data": {
197 "text/html": [
198 "<b>Pipeline Execution Time:</b> 2 Minutes, 36.61 Seconds"
199 ],
200 "text/plain": [
201 "<IPython.core.display.HTML object>"
202 ]
203 },
204 "metadata": {},
205 "output_type": "display_data"
206 },
207 {
208 "data": {
209 "text/html": [
210 "<div>\n",
211 "<table border=\"1\" class=\"dataframe\">\n",
212 " <thead>\n",
213 " <tr style=\"text-align: right;\">\n",
214 " <th></th>\n",
215 " <th></th>\n",
216 " <th>close_price</th>\n",
217 " <th>sentiment_score</th>\n",
218 " </tr>\n",
219 " </thead>\n",
220 " <tbody>\n",
221 " <tr>\n",
222 " <th rowspan=\"10\" valign=\"top\">2014-01-02 00:00:00+00:00</th>\n",
223 " <th>Equity(2 [HWM])</th>\n",
224 " <td>10.63</td>\n",
225 " <td>6.000000</td>\n",
226 " </tr>\n",
227 " <tr>\n",
228 " <th>Equity(24 [AAPL])</th>\n",
229 " <td>561.16</td>\n",
230 " <td>-1.666667</td>\n",
231 " </tr>\n",
232 " <tr>\n",
233 " <th>Equity(62 [ABT])</th>\n",
234 " <td>38.34</td>\n",
235 " <td>4.000000</td>\n",
236 " </tr>\n",
237 " <tr>\n",
238 " <th>Equity(67 [ADSK])</th>\n",
239 " <td>50.32</td>\n",
240 " <td>6.000000</td>\n",
241 " </tr>\n",
242 " <tr>\n",
243 " <th>Equity(76 [TAP])</th>\n",
244 " <td>56.15</td>\n",
245 " <td>6.000000</td>\n",
246 " </tr>\n",
247 " <tr>\n",
248 " <th>Equity(88 [ACI])</th>\n",
249 " <td>4.44</td>\n",
250 " <td>6.000000</td>\n",
251 " </tr>\n",
252 " <tr>\n",
253 " <th>Equity(114 [ADBE])</th>\n",
254 " <td>59.87</td>\n",
255 " <td>5.000000</td>\n",
256 " </tr>\n",
257 " <tr>\n",
258 " <th>Equity(122 [ADI])</th>\n",
259 " <td>50.93</td>\n",
260 " <td>6.000000</td>\n",
261 " </tr>\n",
262 " <tr>\n",
263 " <th>Equity(128 [ADM])</th>\n",
264 " <td>43.41</td>\n",
265 " <td>4.000000</td>\n",
266 " </tr>\n",
267 " <tr>\n",
268 " <th>Equity(161 [AEP])</th>\n",
269 " <td>46.74</td>\n",
270 " <td>6.000000</td>\n",
271 " </tr>\n",
272 " </tbody>\n",
273 "</table>\n",
274 "</div>"
275 ],
276 "text/plain": [
277 " close_price sentiment_score\n",
278 "2014-01-02 00:00:00+00:00 Equity(2 [HWM]) 10.63 6.000000\n",
279 " Equity(24 [AAPL]) 561.16 -1.666667\n",
280 " Equity(62 [ABT]) 38.34 4.000000\n",
281 " Equity(67 [ADSK]) 50.32 6.000000\n",
282 " Equity(76 [TAP]) 56.15 6.000000\n",
283 " Equity(88 [ACI]) 4.44 6.000000\n",
284 " Equity(114 [ADBE]) 59.87 5.000000\n",
285 " Equity(122 [ADI]) 50.93 6.000000\n",
286 " Equity(128 [ADM]) 43.41 4.000000\n",
287 " Equity(161 [AEP]) 46.74 6.000000"
288 ]
289 },
290 "execution_count": 5,
291 "metadata": {},
292 "output_type": "execute_result"
293 }
294 ],
295 "source": [
296 "# Import run_pipeline method\n",
297 "from quantopian.research import run_pipeline\n",
298 "\n",
299 "# Execute pipeline created by make_pipeline\n",
300 "# between start_date and end_date\n",
301 "pipeline_output = run_pipeline(\n",
302 " make_pipeline(),\n",
303 " start_date='2014-01-01',\n",
304 " end_date='2017-1-1'\n",
305 ")\n",
306 "\n",
307 "# Display last 10 rows\n",
308 "pipeline_output.head(10)"
309 ]
310 },
311 {
312 "cell_type": "markdown",
313 "metadata": {},
314 "source": [
315 "In the next lesson we will formalize the strategy our algorithm will use to select assets to trade. Then, we will use a factor analysis tool to evaluate the predictive power of our strategy over historical data."
316 ]
317 }
318 ],
319 "metadata": {
320 "kernelspec": {
321 "display_name": "Python 3.5",
322 "language": "python",
323 "name": "py35"
324 },
325 "language_info": {
326 "codemirror_mode": {
327 "name": "ipython",
328 "version": 3
329 },
330 "file_extension": ".py",
331 "mimetype": "text/x-python",
332 "name": "python",
333 "nbconvert_exporter": "python",
334 "pygments_lexer": "ipython3",
335 "version": "3.5.9"
336 }
337 },
338 "nbformat": 4,
339 "nbformat_minor": 2
340 }