ml-finance-python
python scripts for finance machine learning
git clone https://9o.is/git/ml-finance-python.git
03_word2vec.ipynb
(202053B)
1 {
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {},
6 "source": [
7 "# How to train your own word vector embeddings"
8 ]
9 },
10 {
11 "cell_type": "markdown",
12 "metadata": {},
13 "source": [
14 "Many tasks require embeddings or domain-specific vocabulary that pre-trained models based on a generic corpus may not represent well or at all. Standard word2vec models are not able to assign vectors to out-of-vocabulary words and instead use a default vector that reduces their predictive value.\n",
15 "\n",
16 "E.g., when working with industry-specific documents, the vocabulary or its usage may change over time as new technologies or products emerge. As a result, the embeddings need to evolve as well. In addition, corporate earnings releases use nuanced language not fully reflected in Glove vectors pre-trained on Wikipedia articles.\n",
17 "\n",
18 "We will illustrate the word2vec architecture using the keras library that we will introduce in more detail in the next chapter and the more performant gensim adaptation of the code provided by the word2vec authors. "
19 ]
20 },
21 {
22 "cell_type": "markdown",
23 "metadata": {},
24 "source": [
25 "To illustrate the word2vec network architecture, we use the TED talk dataset with aligned English and Spanish subtitles that we first introduced in chapter 13. \n",
26 "\n",
27 "This notebook contains the code to tokenize the documents and assign a unique id to each item in the vocabulary. We require at least five occurrences in the corpus and keep a vocabulary of 31,300 tokens."
28 ]
29 },
30 {
31 "cell_type": "markdown",
32 "metadata": {
33 "slideshow": {
34 "slide_type": "fragment"
35 }
36 },
37 "source": [
38 "## Imports"
39 ]
40 },
41 {
42 "cell_type": "code",
43 "execution_count": 1,
44 "metadata": {
45 "ExecuteTime": {
46 "end_time": "2018-12-10T06:52:40.090697Z",
47 "start_time": "2018-12-10T06:52:39.318699Z"
48 },
49 "slideshow": {
50 "slide_type": "fragment"
51 }
52 },
53 "outputs": [],
54 "source": [
55 "from time import time\n",
56 "from collections import Counter\n",
57 "from pathlib import Path\n",
58 "import pandas as pd\n",
59 "import numpy as np\n",
60 "from numpy.linalg import norm\n",
61 "from scipy.spatial.distance import cdist, cosine\n",
62 "\n",
63 "import matplotlib.pyplot as plt\n",
64 "\n",
65 "from tensorflow.keras.models import Model\n",
66 "from tensorflow.keras.layers import Input, Dense, Reshape, Dot, Embedding\n",
67 "from tensorflow.keras.preprocessing.sequence import skipgrams, make_sampling_table\n",
68 "from tensorflow.keras.callbacks import Callback, TensorBoard\n",
69 "\n",
70 "from gensim.models import Word2Vec, KeyedVectors\n",
71 "from gensim.models.word2vec import LineSentence\n",
72 "from sklearn.decomposition import IncrementalPCA"
73 ]
74 },
75 {
76 "cell_type": "markdown",
77 "metadata": {
78 "slideshow": {
79 "slide_type": ""
80 }
81 },
82 "source": [
83 "### Settings"
84 ]
85 },
86 {
87 "cell_type": "code",
88 "execution_count": 2,
89 "metadata": {
90 "ExecuteTime": {
91 "end_time": "2018-12-10T06:52:40.096261Z",
92 "start_time": "2018-12-10T06:52:40.091941Z"
93 },
94 "slideshow": {
95 "slide_type": "fragment"
96 }
97 },
98 "outputs": [],
99 "source": [
100 "plt.style.use('ggplot')\n",
101 "pd.set_option('float_format', '{:,.2f}'.format)\n",
102 "%matplotlib inline\n",
103 "np.random.seed(42)"
104 ]
105 },
106 {
107 "cell_type": "code",
108 "execution_count": 3,
109 "metadata": {
110 "ExecuteTime": {
111 "end_time": "2018-12-10T06:52:40.534840Z",
112 "start_time": "2018-12-10T06:52:40.530045Z"
113 }
114 },
115 "outputs": [],
116 "source": [
117 "PROJECT_DIR = Path().cwd()"
118 ]
119 },
120 {
121 "cell_type": "code",
122 "execution_count": 4,
123 "metadata": {
124 "ExecuteTime": {
125 "end_time": "2018-12-10T06:52:45.285698Z",
126 "start_time": "2018-12-10T06:52:45.283889Z"
127 },
128 "slideshow": {
129 "slide_type": "fragment"
130 }
131 },
132 "outputs": [],
133 "source": [
134 "LANGUAGES = ['en', 'es']\n",
135 "language_dict = dict(zip(LANGUAGES, ['English', 'Spanish']))"
136 ]
137 },
138 {
139 "cell_type": "markdown",
140 "metadata": {
141 "slideshow": {
142 "slide_type": "slide"
143 }
144 },
145 "source": [
146 "## TED2013 Corpus Statistics"
147 ]
148 },
149 {
150 "cell_type": "markdown",
151 "metadata": {},
152 "source": [
153 "The inputs are produced by the [preprocessing](01_preprocessing.ipynb) notebook that needs to run first."
154 ]
155 },
156 {
157 "cell_type": "code",
158 "execution_count": 5,
159 "metadata": {
160 "ExecuteTime": {
161 "end_time": "2018-12-10T06:52:46.172123Z",
162 "start_time": "2018-12-10T06:52:46.170203Z"
163 },
164 "slideshow": {
165 "slide_type": "fragment"
166 }
167 },
168 "outputs": [],
169 "source": [
170 "SOURCE = 'TED'\n",
171 "LANGUAGE = 'en'"
172 ]
173 },
174 {
175 "cell_type": "code",
176 "execution_count": 6,
177 "metadata": {
178 "ExecuteTime": {
179 "end_time": "2018-12-10T06:52:47.829019Z",
180 "start_time": "2018-12-10T06:52:47.760797Z"
181 },
182 "slideshow": {
183 "slide_type": "fragment"
184 }
185 },
186 "outputs": [
187 {
188 "name": "stdout",
189 "output_type": "stream",
190 "text": [
191 "<class 'pandas.io.pytables.HDFStore'>\n",
192 "File path: vocab/TED/vocab.h5\n",
193 "/en/tokens frame (shape->[3130266,3])\n",
194 "/en/vocab frame (shape->[46176,1]) \n",
195 "/es/tokens frame (shape->[2938546,3])\n",
196 "/es/vocab frame (shape->[74739,1]) \n"
197 ]
198 }
199 ],
200 "source": [
201 "with pd.HDFStore(Path('vocab', SOURCE, 'vocab.h5')) as store:\n",
202 " print(store.info())"
203 ]
204 },
205 {
206 "cell_type": "code",
207 "execution_count": 7,
208 "metadata": {
209 "ExecuteTime": {
210 "end_time": "2018-12-10T06:52:56.753472Z",
211 "start_time": "2018-12-10T06:52:56.687010Z"
212 },
213 "slideshow": {
214 "slide_type": "slide"
215 }
216 },
217 "outputs": [
218 {
219 "name": "stdout",
220 "output_type": "stream",
221 "text": [
222 "# words: 2,710,455\n"
223 ]
224 },
225 {
226 "data": {
227 "text/html": [
228 "<div>\n",
229 "<style scoped>\n",
230 " .dataframe tbody tr th:only-of-type {\n",
231 " vertical-align: middle;\n",
232 " }\n",
233 "\n",
234 " .dataframe tbody tr th {\n",
235 " vertical-align: top;\n",
236 " }\n",
237 "\n",
238 " .dataframe thead th {\n",
239 " text-align: right;\n",
240 " }\n",
241 "</style>\n",
242 "<table border=\"1\" class=\"dataframe\">\n",
243 " <thead>\n",
244 " <tr style=\"text-align: right;\">\n",
245 " <th></th>\n",
246 " <th>word_count</th>\n",
247 " <th>freq</th>\n",
248 " <th>n_words</th>\n",
249 " <th>vocab_size</th>\n",
250 " <th>coverage</th>\n",
251 " </tr>\n",
252 " </thead>\n",
253 " <tbody>\n",
254 " <tr>\n",
255 " <th>0</th>\n",
256 " <td>127548</td>\n",
257 " <td>1</td>\n",
258 " <td>127548</td>\n",
259 " <td>1</td>\n",
260 " <td>0.05</td>\n",
261 " </tr>\n",
262 " <tr>\n",
263 " <th>1</th>\n",
264 " <td>90702</td>\n",
265 " <td>1</td>\n",
266 " <td>90702</td>\n",
267 " <td>2</td>\n",
268 " <td>0.08</td>\n",
269 " </tr>\n",
270 " <tr>\n",
271 " <th>2</th>\n",
272 " <td>74494</td>\n",
273 " <td>1</td>\n",
274 " <td>74494</td>\n",
275 " <td>3</td>\n",
276 " <td>0.11</td>\n",
277 " </tr>\n",
278 " <tr>\n",
279 " <th>3</th>\n",
280 " <td>70274</td>\n",
281 " <td>1</td>\n",
282 " <td>70274</td>\n",
283 " <td>4</td>\n",
284 " <td>0.13</td>\n",
285 " </tr>\n",
286 " <tr>\n",
287 " <th>4</th>\n",
288 " <td>64387</td>\n",
289 " <td>1</td>\n",
290 " <td>64387</td>\n",
291 " <td>5</td>\n",
292 " <td>0.16</td>\n",
293 " </tr>\n",
294 " <tr>\n",
295 " <th>893</th>\n",
296 " <td>5</td>\n",
297 " <td>1755</td>\n",
298 " <td>8775</td>\n",
299 " <td>17026</td>\n",
300 " <td>0.98</td>\n",
301 " </tr>\n",
302 " <tr>\n",
303 " <th>894</th>\n",
304 " <td>4</td>\n",
305 " <td>2484</td>\n",
306 " <td>9936</td>\n",
307 " <td>19510</td>\n",
308 " <td>0.98</td>\n",
309 " </tr>\n",
310 " <tr>\n",
311 " <th>895</th>\n",
312 " <td>3</td>\n",
313 " <td>3831</td>\n",
314 " <td>11493</td>\n",
315 " <td>23341</td>\n",
316 " <td>0.99</td>\n",
317 " </tr>\n",
318 " <tr>\n",
319 " <th>896</th>\n",
320 " <td>2</td>\n",
321 " <td>6822</td>\n",
322 " <td>13644</td>\n",
323 " <td>30163</td>\n",
324 " <td>0.99</td>\n",
325 " </tr>\n",
326 " <tr>\n",
327 " <th>897</th>\n",
328 " <td>1</td>\n",
329 " <td>16013</td>\n",
330 " <td>16013</td>\n",
331 " <td>46176</td>\n",
332 " <td>1.00</td>\n",
333 " </tr>\n",
334 " </tbody>\n",
335 "</table>\n",
336 "</div>"
337 ],
338 "text/plain": [
339 " word_count freq n_words vocab_size coverage\n",
340 "0 127548 1 127548 1 0.05\n",
341 "1 90702 1 90702 2 0.08\n",
342 "2 74494 1 74494 3 0.11\n",
343 "3 70274 1 70274 4 0.13\n",
344 "4 64387 1 64387 5 0.16\n",
345 "893 5 1755 8775 17026 0.98\n",
346 "894 4 2484 9936 19510 0.98\n",
347 "895 3 3831 11493 23341 0.99\n",
348 "896 2 6822 13644 30163 0.99\n",
349 "897 1 16013 16013 46176 1.00"
350 ]
351 },
352 "execution_count": 7,
353 "metadata": {},
354 "output_type": "execute_result"
355 }
356 ],
357 "source": [
358 "with pd.HDFStore(Path('vocab', SOURCE, 'vocab.h5')) as store:\n",
359 " df = store['{}/vocab'.format(LANGUAGE)]\n",
360 "\n",
361 "wc = df['count'].value_counts().sort_index(ascending=False).reset_index()\n",
362 "wc.columns = ['word_count', 'freq']\n",
363 "wc['n_words'] = wc.word_count.mul(wc.freq)\n",
364 "\n",
365 "wc['corpus_share'] = wc.n_words.div(wc.n_words.sum())\n",
366 "wc['coverage'] = wc.corpus_share.cumsum()\n",
367 "wc['vocab_size'] = wc.freq.cumsum()\n",
368 "\n",
369 "print('# words: {:,d}'.format(wc.n_words.sum()))\n",
370 "(wc\n",
371 " .loc[:, ['word_count', 'freq', 'n_words', 'vocab_size', 'coverage']]\n",
372 " .head()\n",
373 " .append(wc\n",
374 " .loc[:, ['word_count', 'freq', 'n_words', 'vocab_size', 'coverage']]\n",
375 " .tail()))"
376 ]
377 },
378 {
379 "cell_type": "code",
380 "execution_count": 8,
381 "metadata": {
382 "ExecuteTime": {
383 "end_time": "2018-12-10T06:52:58.961174Z",
384 "start_time": "2018-12-10T06:52:58.951088Z"
385 },
386 "scrolled": true,
387 "slideshow": {
388 "slide_type": "slide"
389 }
390 },
391 "outputs": [
392 {
393 "data": {
394 "text/plain": [
395 "count 898.00\n",
396 "mean 0.09\n",
397 "std 0.30\n",
398 "min 0.00\n",
399 "50% 0.02\n",
400 "75% 0.04\n",
401 "95% 0.31\n",
402 "96% 0.37\n",
403 "97% 0.49\n",
404 "98% 0.62\n",
405 "99% 1.63\n",
406 "99.9% 3.49\n",
407 "max 4.71\n",
408 "Name: word_count, dtype: float64"
409 ]
410 },
411 "execution_count": 8,
412 "metadata": {},
413 "output_type": "execute_result"
414 }
415 ],
416 "source": [
417 "wc.word_count.div(wc.n_words.sum()).mul(100).describe(percentiles=[.5, .75, .95, .96, .97, .98, .99, .999])"
418 ]
419 },
420 {
421 "cell_type": "markdown",
422 "metadata": {
423 "slideshow": {
424 "slide_type": "slide"
425 }
426 },
427 "source": [
428 "## Evaluation: Analogies"
429 ]
430 },
431 {
432 "cell_type": "markdown",
433 "metadata": {},
434 "source": [
435 "The dimensions of the word and phrase vectors do not have an explicit meaning. However, the embeddings encode similar usage as proximity in the latent space in a way that carries over to semantic relationships. This results in the interesting properties that analogies can be expressed by adding and subtracting word vectors.\n",
436 "\n",
437 "Just as words can be used in different contexts, they can be related to other words in different ways, and these relationships correspond to different directions in the latent space. Accordingly, there are several types of analogies that the embeddings should reflect if the training data permits.\n",
438 "\n",
439 "The word2vec authors provide a list of several thousand relationships spanning aspects of geography, grammar and syntax, and family relationships to evaluate the quality of embedding vectors (see directory [analogies](data/analogies))."
440 ]
441 },
442 {
443 "cell_type": "code",
444 "execution_count": 9,
445 "metadata": {
446 "ExecuteTime": {
447 "end_time": "2018-12-10T06:53:02.611652Z",
448 "start_time": "2018-12-10T06:53:02.535004Z"
449 },
450 "slideshow": {
451 "slide_type": "fragment"
452 }
453 },
454 "outputs": [],
455 "source": [
456 "df = pd.read_csv(Path('data', 'analogies', 'analogies-en.txt'), header=None, names=['category'], squeeze=True)\n",
457 "categories = df[df.str.startswith(':')]\n",
458 "analogies = df[~df.str.startswith(':')].str.split(expand=True)\n",
459 "analogies.columns = list('abcd')"
460 ]
461 },
462 {
463 "cell_type": "code",
464 "execution_count": 10,
465 "metadata": {
466 "ExecuteTime": {
467 "end_time": "2018-12-10T06:53:02.750899Z",
468 "start_time": "2018-12-10T06:53:02.724987Z"
469 },
470 "slideshow": {
471 "slide_type": "fragment"
472 }
473 },
474 "outputs": [
475 {
476 "data": {
477 "text/html": [
478 "<div>\n",
479 "<style scoped>\n",
480 " .dataframe tbody tr th:only-of-type {\n",
481 " vertical-align: middle;\n",
482 " }\n",
483 "\n",
484 " .dataframe tbody tr th {\n",
485 " vertical-align: top;\n",
486 " }\n",
487 "\n",
488 " .dataframe thead th {\n",
489 " text-align: right;\n",
490 " }\n",
491 "</style>\n",
492 "<table border=\"1\" class=\"dataframe\">\n",
493 " <thead>\n",
494 " <tr style=\"text-align: right;\">\n",
495 " <th></th>\n",
496 " <th>category</th>\n",
497 " <th>a</th>\n",
498 " <th>b</th>\n",
499 " <th>c</th>\n",
500 " <th>d</th>\n",
501 " </tr>\n",
502 " </thead>\n",
503 " <tbody>\n",
504 " <tr>\n",
505 " <th>1</th>\n",
506 " <td>: capital-common-countries</td>\n",
507 " <td>athens</td>\n",
508 " <td>greece</td>\n",
509 " <td>baghdad</td>\n",
510 " <td>iraq</td>\n",
511 " </tr>\n",
512 " <tr>\n",
513 " <th>2</th>\n",
514 " <td>: capital-common-countries</td>\n",
515 " <td>athens</td>\n",
516 " <td>greece</td>\n",
517 " <td>bangkok</td>\n",
518 " <td>thailand</td>\n",
519 " </tr>\n",
520 " <tr>\n",
521 " <th>3</th>\n",
522 " <td>: capital-common-countries</td>\n",
523 " <td>athens</td>\n",
524 " <td>greece</td>\n",
525 " <td>beijing</td>\n",
526 " <td>china</td>\n",
527 " </tr>\n",
528 " <tr>\n",
529 " <th>4</th>\n",
530 " <td>: capital-common-countries</td>\n",
531 " <td>athens</td>\n",
532 " <td>greece</td>\n",
533 " <td>berlin</td>\n",
534 " <td>germany</td>\n",
535 " </tr>\n",
536 " <tr>\n",
537 " <th>5</th>\n",
538 " <td>: capital-common-countries</td>\n",
539 " <td>athens</td>\n",
540 " <td>greece</td>\n",
541 " <td>bern</td>\n",
542 " <td>switzerland</td>\n",
543 " </tr>\n",
544 " </tbody>\n",
545 "</table>\n",
546 "</div>"
547 ],
548 "text/plain": [
549 " category a b c d\n",
550 "1 : capital-common-countries athens greece baghdad iraq\n",
551 "2 : capital-common-countries athens greece bangkok thailand\n",
552 "3 : capital-common-countries athens greece beijing china\n",
553 "4 : capital-common-countries athens greece berlin germany\n",
554 "5 : capital-common-countries athens greece bern switzerland"
555 ]
556 },
557 "execution_count": 10,
558 "metadata": {},
559 "output_type": "execute_result"
560 }
561 ],
562 "source": [
563 "df = pd.concat([categories, analogies], axis=1)\n",
564 "df.category = df.category.ffill()\n",
565 "df = df[df['a'].notnull()]\n",
566 "df.head()"
567 ]
568 },
569 {
570 "cell_type": "code",
571 "execution_count": 11,
572 "metadata": {
573 "ExecuteTime": {
574 "end_time": "2018-12-10T06:53:03.107617Z",
575 "start_time": "2018-12-10T06:53:03.068779Z"
576 },
577 "slideshow": {
578 "slide_type": "slide"
579 }
580 },
581 "outputs": [],
582 "source": [
583 "analogy_cnt = df.groupby('category').size().sort_values(ascending=False).to_frame('n')\n",
584 "analogy_example = df.groupby('category').first()"
585 ]
586 },
587 {
588 "cell_type": "code",
589 "execution_count": 12,
590 "metadata": {
591 "ExecuteTime": {
592 "end_time": "2018-12-10T06:53:05.515262Z",
593 "start_time": "2018-12-10T06:53:05.500372Z"
594 },
595 "slideshow": {
596 "slide_type": "slide"
597 }
598 },
599 "outputs": [
600 {
601 "data": {
602 "text/html": [
603 "<div>\n",
604 "<style scoped>\n",
605 " .dataframe tbody tr th:only-of-type {\n",
606 " vertical-align: middle;\n",
607 " }\n",
608 "\n",
609 " .dataframe tbody tr th {\n",
610 " vertical-align: top;\n",
611 " }\n",
612 "\n",
613 " .dataframe thead th {\n",
614 " text-align: right;\n",
615 " }\n",
616 "</style>\n",
617 "<table border=\"1\" class=\"dataframe\">\n",
618 " <thead>\n",
619 " <tr style=\"text-align: right;\">\n",
620 " <th></th>\n",
621 " <th>n</th>\n",
622 " <th>a</th>\n",
623 " <th>b</th>\n",
624 " <th>c</th>\n",
625 " <th>d</th>\n",
626 " </tr>\n",
627 " <tr>\n",
628 " <th>category</th>\n",
629 " <th></th>\n",
630 " <th></th>\n",
631 " <th></th>\n",
632 " <th></th>\n",
633 " <th></th>\n",
634 " </tr>\n",
635 " </thead>\n",
636 " <tbody>\n",
637 " <tr>\n",
638 " <th>: capital-world</th>\n",
639 " <td>8556</td>\n",
640 " <td>abuja</td>\n",
641 " <td>nigeria</td>\n",
642 " <td>accra</td>\n",
643 " <td>ghana</td>\n",
644 " </tr>\n",
645 " <tr>\n",
646 " <th>: city-in-state</th>\n",
647 " <td>4242</td>\n",
648 " <td>chicago</td>\n",
649 " <td>illinois</td>\n",
650 " <td>houston</td>\n",
651 " <td>texas</td>\n",
652 " </tr>\n",
653 " <tr>\n",
654 " <th>: gram6-nationality-adjective</th>\n",
655 " <td>1640</td>\n",
656 " <td>albania</td>\n",
657 " <td>albanian</td>\n",
658 " <td>argentina</td>\n",
659 " <td>argentinean</td>\n",
660 " </tr>\n",
661 " <tr>\n",
662 " <th>: gram7-past-tense</th>\n",
663 " <td>1560</td>\n",
664 " <td>dancing</td>\n",
665 " <td>danced</td>\n",
666 " <td>decreasing</td>\n",
667 " <td>decreased</td>\n",
668 " </tr>\n",
669 " <tr>\n",
670 " <th>: gram8-plural</th>\n",
671 " <td>1332</td>\n",
672 " <td>banana</td>\n",
673 " <td>bananas</td>\n",
674 " <td>bird</td>\n",
675 " <td>birds</td>\n",
676 " </tr>\n",
677 " <tr>\n",
678 " <th>: gram3-comparative</th>\n",
679 " <td>1332</td>\n",
680 " <td>bad</td>\n",
681 " <td>worse</td>\n",
682 " <td>big</td>\n",
683 " <td>bigger</td>\n",
684 " </tr>\n",
685 " <tr>\n",
686 " <th>: gram4-superlative</th>\n",
687 " <td>1122</td>\n",
688 " <td>bad</td>\n",
689 " <td>worst</td>\n",
690 " <td>big</td>\n",
691 " <td>biggest</td>\n",
692 " </tr>\n",
693 " <tr>\n",
694 " <th>: gram5-present-participle</th>\n",
695 " <td>1056</td>\n",
696 " <td>code</td>\n",
697 " <td>coding</td>\n",
698 " <td>dance</td>\n",
699 " <td>dancing</td>\n",
700 " </tr>\n",
701 " <tr>\n",
702 " <th>: gram1-adjective-to-adverb</th>\n",
703 " <td>992</td>\n",
704 " <td>amazing</td>\n",
705 " <td>amazingly</td>\n",
706 " <td>apparent</td>\n",
707 " <td>apparently</td>\n",
708 " </tr>\n",
709 " <tr>\n",
710 " <th>: gram9-plural-verbs</th>\n",
711 " <td>870</td>\n",
712 " <td>decrease</td>\n",
713 " <td>decreases</td>\n",
714 " <td>describe</td>\n",
715 " <td>describes</td>\n",
716 " </tr>\n",
717 " <tr>\n",
718 " <th>: currency</th>\n",
719 " <td>866</td>\n",
720 " <td>algeria</td>\n",
721 " <td>dinar</td>\n",
722 " <td>angola</td>\n",
723 " <td>kwanza</td>\n",
724 " </tr>\n",
725 " <tr>\n",
726 " <th>: gram2-opposite</th>\n",
727 " <td>812</td>\n",
728 " <td>acceptable</td>\n",
729 " <td>unacceptable</td>\n",
730 " <td>aware</td>\n",
731 " <td>unaware</td>\n",
732 " </tr>\n",
733 " <tr>\n",
734 " <th>: family</th>\n",
735 " <td>506</td>\n",
736 " <td>boy</td>\n",
737 " <td>girl</td>\n",
738 " <td>brother</td>\n",
739 " <td>sister</td>\n",
740 " </tr>\n",
741 " <tr>\n",
742 " <th>: capital-common-countries</th>\n",
743 " <td>506</td>\n",
744 " <td>athens</td>\n",
745 " <td>greece</td>\n",
746 " <td>baghdad</td>\n",
747 " <td>iraq</td>\n",
748 " </tr>\n",
749 " </tbody>\n",
750 "</table>\n",
751 "</div>"
752 ],
753 "text/plain": [
754 " n a b c \\\n",
755 "category \n",
756 ": capital-world 8556 abuja nigeria accra \n",
757 ": city-in-state 4242 chicago illinois houston \n",
758 ": gram6-nationality-adjective 1640 albania albanian argentina \n",
759 ": gram7-past-tense 1560 dancing danced decreasing \n",
760 ": gram8-plural 1332 banana bananas bird \n",
761 ": gram3-comparative 1332 bad worse big \n",
762 ": gram4-superlative 1122 bad worst big \n",
763 ": gram5-present-participle 1056 code coding dance \n",
764 ": gram1-adjective-to-adverb 992 amazing amazingly apparent \n",
765 ": gram9-plural-verbs 870 decrease decreases describe \n",
766 ": currency 866 algeria dinar angola \n",
767 ": gram2-opposite 812 acceptable unacceptable aware \n",
768 ": family 506 boy girl brother \n",
769 ": capital-common-countries 506 athens greece baghdad \n",
770 "\n",
771 " d \n",
772 "category \n",
773 ": capital-world ghana \n",
774 ": city-in-state texas \n",
775 ": gram6-nationality-adjective argentinean \n",
776 ": gram7-past-tense decreased \n",
777 ": gram8-plural birds \n",
778 ": gram3-comparative bigger \n",
779 ": gram4-superlative biggest \n",
780 ": gram5-present-participle dancing \n",
781 ": gram1-adjective-to-adverb apparently \n",
782 ": gram9-plural-verbs describes \n",
783 ": currency kwanza \n",
784 ": gram2-opposite unaware \n",
785 ": family sister \n",
786 ": capital-common-countries iraq "
787 ]
788 },
789 "execution_count": 12,
790 "metadata": {},
791 "output_type": "execute_result"
792 }
793 ],
794 "source": [
795 "analogy_cnt.join(analogy_example)"
796 ]
797 },
798 {
799 "cell_type": "markdown",
800 "metadata": {
801 "slideshow": {
802 "slide_type": "slide"
803 }
804 },
805 "source": [
806 "## `word2vec` - skipgram Architecture using Keras"
807 ]
808 },
809 {
810 "cell_type": "markdown",
811 "metadata": {
812 "slideshow": {
813 "slide_type": "slide"
814 }
815 },
816 "source": [
817 "### Settings"
818 ]
819 },
820 {
821 "cell_type": "code",
822 "execution_count": 11,
823 "metadata": {
824 "ExecuteTime": {
825 "end_time": "2018-12-10T06:53:11.962382Z",
826 "start_time": "2018-12-10T06:53:11.957411Z"
827 },
828 "slideshow": {
829 "slide_type": "fragment"
830 }
831 },
832 "outputs": [],
833 "source": [
834 "NGRAMS = 3 # Longest ngram in text\n",
835 "FILE_NAME = 'ngrams_{}'.format(NGRAMS) # Input to use\n",
836 "MIN_FREQ = 5\n",
837 "SAMPLING_FACTOR = 1e-4\n",
838 "WINDOW_SIZE = 5\n",
839 "EMBEDDING_SIZE = 200\n",
840 "EPOCHS = 1\n",
841 "BATCH_SIZE = 50\n",
842 "\n",
843 "# Set up validation\n",
844 "VALID_SET = 10 # Random set of words to get nearest neighbors for\n",
845 "VALID_WINDOW = 150 # Most frequent words to draw validation set from\n",
846 "NN = 10 # Number of nearest neighbors for evaluation\n",
847 "\n",
848 "valid_examples = np.random.choice(VALID_WINDOW, size=VALID_SET, replace=False)"
849 ]
850 },
851 {
852 "cell_type": "code",
853 "execution_count": 12,
854 "metadata": {
855 "ExecuteTime": {
856 "end_time": "2018-12-10T06:53:13.843939Z",
857 "start_time": "2018-12-10T06:53:13.841581Z"
858 },
859 "slideshow": {
860 "slide_type": "fragment"
861 }
862 },
863 "outputs": [],
864 "source": [
865 "path = Path('keras', SOURCE, LANGUAGE, FILE_NAME).resolve()\n",
866 "tb_path = path / 'tensorboard'\n",
867 "if not tb_path.exists():\n",
868 " tb_path.mkdir(parents=True, exist_ok=True)"
869 ]
870 },
871 {
872 "cell_type": "markdown",
873 "metadata": {
874 "slideshow": {
875 "slide_type": "slide"
876 }
877 },
878 "source": [
879 "### Build Data Set"
880 ]
881 },
882 {
883 "cell_type": "markdown",
884 "metadata": {
885 "slideshow": {
886 "slide_type": "fragment"
887 }
888 },
889 "source": [
890 "#### Tokens to ID\n",
891 "\n",
892 "1. Extract the top *n* most common words to learn embeddings\n",
893 "2. Index these *n* words with unique integers\n",
894 "3. Create an `{index: word}` dictionary\n",
895 "4. Replace the *n* words with their index, and a dummy value `UNK` elsewhere"
896 ]
897 },
898 {
899 "cell_type": "code",
900 "execution_count": 15,
901 "metadata": {
902 "ExecuteTime": {
903 "end_time": "2018-12-10T06:54:02.709606Z",
904 "start_time": "2018-12-10T06:54:02.705898Z"
905 },
906 "slideshow": {
907 "slide_type": "slide"
908 }
909 },
910 "outputs": [],
911 "source": [
912 "def build_data(language, ngrams=1):\n",
913 " file_path = PROJECT_DIR / 'vocab' / SOURCE / language / 'ngrams_{}.txt'.format(ngrams)\n",
914 " words = file_path.read_text().split()\n",
915 " \n",
916 " # Get (token, count) tuples for tokens meeting MIN_FREQ \n",
917 " token_counts = [t for t in Counter(words).most_common() if t[1] >= MIN_FREQ]\n",
918 " tokens, counts = list(zip(*token_counts))\n",
919 " \n",
920 " # create id-token dicts & reverse dicts\n",
921 " id_to_token = pd.Series(tokens, index=range(1, len(tokens) + 1)).to_dict()\n",
922 " id_to_token.update({0: 'UNK'})\n",
923 " token_to_id = {t:i for i, t in id_to_token.items()}\n",
924 " data = [token_to_id.get(word, 0) for word in words]\n",
925 " return data, token_to_id, id_to_token"
926 ]
927 },
928 {
929 "cell_type": "code",
930 "execution_count": 16,
931 "metadata": {
932 "ExecuteTime": {
933 "end_time": "2018-12-10T06:54:05.230482Z",
934 "start_time": "2018-12-10T06:54:03.089009Z"
935 },
936 "slideshow": {
937 "slide_type": "fragment"
938 }
939 },
940 "outputs": [],
941 "source": [
942 "data, token_to_id, id_to_token = build_data(LANGUAGE, ngrams=NGRAMS)"
943 ]
944 },
945 {
946 "cell_type": "code",
947 "execution_count": 17,
948 "metadata": {
949 "ExecuteTime": {
950 "end_time": "2018-12-10T06:54:07.383676Z",
951 "start_time": "2018-12-10T06:54:07.381927Z"
952 },
953 "slideshow": {
954 "slide_type": "slide"
955 }
956 },
957 "outputs": [],
958 "source": [
959 "vocab_size = len(token_to_id)"
960 ]
961 },
962 {
963 "cell_type": "code",
964 "execution_count": 18,
965 "metadata": {
966 "ExecuteTime": {
967 "end_time": "2018-12-10T06:54:07.562714Z",
968 "start_time": "2018-12-10T06:54:07.558075Z"
969 },
970 "slideshow": {
971 "slide_type": "fragment"
972 }
973 },
974 "outputs": [
975 {
976 "data": {
977 "text/plain": [
978 "17381"
979 ]
980 },
981 "execution_count": 18,
982 "metadata": {},
983 "output_type": "execute_result"
984 }
985 ],
986 "source": [
987 "vocab_size"
988 ]
989 },
990 {
991 "cell_type": "code",
992 "execution_count": 19,
993 "metadata": {
994 "ExecuteTime": {
995 "end_time": "2018-12-10T06:54:14.901599Z",
996 "start_time": "2018-12-10T06:54:14.745194Z"
997 },
998 "slideshow": {
999 "slide_type": "fragment"
1000 }
1001 },
1002 "outputs": [
1003 {
1004 "data": {
1005 "text/plain": [
1006 "(0, 17380)"
1007 ]
1008 },
1009 "execution_count": 19,
1010 "metadata": {},
1011 "output_type": "execute_result"
1012 }
1013 ],
1014 "source": [
1015 "min(data), max(data)"
1016 ]
1017 },
1018 {
1019 "cell_type": "code",
1020 "execution_count": 20,
1021 "metadata": {
1022 "ExecuteTime": {
1023 "end_time": "2018-12-10T06:54:18.499601Z",
1024 "start_time": "2018-12-10T06:54:17.631605Z"
1025 },
1026 "slideshow": {
1027 "slide_type": "fragment"
1028 }
1029 },
1030 "outputs": [],
1031 "source": [
1032 "s = pd.Series(data).value_counts().reset_index()\n",
1033 "s.columns = ['id', 'count']\n",
1034 "s['token'] = s.id.map(id_to_token)"
1035 ]
1036 },
1037 {
1038 "cell_type": "code",
1039 "execution_count": 21,
1040 "metadata": {
1041 "ExecuteTime": {
1042 "end_time": "2018-12-10T06:54:18.508607Z",
1043 "start_time": "2018-12-10T06:54:18.500964Z"
1044 },
1045 "slideshow": {
1046 "slide_type": "slide"
1047 }
1048 },
1049 "outputs": [
1050 {
1051 "data": {
1052 "text/html": [
1053 "<div>\n",
1054 "<style scoped>\n",
1055 " .dataframe tbody tr th:only-of-type {\n",
1056 " vertical-align: middle;\n",
1057 " }\n",
1058 "\n",
1059 " .dataframe tbody tr th {\n",
1060 " vertical-align: top;\n",
1061 " }\n",
1062 "\n",
1063 " .dataframe thead th {\n",
1064 " text-align: right;\n",
1065 " }\n",
1066 "</style>\n",
1067 "<table border=\"1\" class=\"dataframe\">\n",
1068 " <thead>\n",
1069 " <tr style=\"text-align: right;\">\n",
1070 " <th></th>\n",
1071 " <th>id</th>\n",
1072 " <th>count</th>\n",
1073 " <th>token</th>\n",
1074 " </tr>\n",
1075 " </thead>\n",
1076 " <tbody>\n",
1077 " <tr>\n",
1078 " <th>0</th>\n",
1079 " <td>1</td>\n",
1080 " <td>127548</td>\n",
1081 " <td>the</td>\n",
1082 " </tr>\n",
1083 " <tr>\n",
1084 " <th>1</th>\n",
1085 " <td>2</td>\n",
1086 " <td>90702</td>\n",
1087 " <td>and</td>\n",
1088 " </tr>\n",
1089 " <tr>\n",
1090 " <th>2</th>\n",
1091 " <td>3</td>\n",
1092 " <td>74494</td>\n",
1093 " <td>to</td>\n",
1094 " </tr>\n",
1095 " <tr>\n",
1096 " <th>3</th>\n",
1097 " <td>4</td>\n",
1098 " <td>70274</td>\n",
1099 " <td>of</td>\n",
1100 " </tr>\n",
1101 " <tr>\n",
1102 " <th>4</th>\n",
1103 " <td>5</td>\n",
1104 " <td>64387</td>\n",
1105 " <td>a</td>\n",
1106 " </tr>\n",
1107 " <tr>\n",
1108 " <th>5</th>\n",
1109 " <td>6</td>\n",
1110 " <td>56578</td>\n",
1111 " <td>that</td>\n",
1112 " </tr>\n",
1113 " <tr>\n",
1114 " <th>6</th>\n",
1115 " <td>0</td>\n",
1116 " <td>51254</td>\n",
1117 " <td>UNK</td>\n",
1118 " </tr>\n",
1119 " <tr>\n",
1120 " <th>7</th>\n",
1121 " <td>7</td>\n",
1122 " <td>49621</td>\n",
1123 " <td>i</td>\n",
1124 " </tr>\n",
1125 " <tr>\n",
1126 " <th>8</th>\n",
1127 " <td>8</td>\n",
1128 " <td>47404</td>\n",
1129 " <td>in</td>\n",
1130 " </tr>\n",
1131 " <tr>\n",
1132 " <th>9</th>\n",
1133 " <td>9</td>\n",
1134 " <td>46931</td>\n",
1135 " <td>it</td>\n",
1136 " </tr>\n",
1137 " </tbody>\n",
1138 "</table>\n",
1139 "</div>"
1140 ],
1141 "text/plain": [
1142 " id count token\n",
1143 "0 1 127548 the\n",
1144 "1 2 90702 and\n",
1145 "2 3 74494 to\n",
1146 "3 4 70274 of\n",
1147 "4 5 64387 a\n",
1148 "5 6 56578 that\n",
1149 "6 0 51254 UNK\n",
1150 "7 7 49621 i\n",
1151 "8 8 47404 in\n",
1152 "9 9 46931 it"
1153 ]
1154 },
1155 "execution_count": 21,
1156 "metadata": {},
1157 "output_type": "execute_result"
1158 }
1159 ],
1160 "source": [
1161 "s.sort_values('count', ascending=False).head(10)"
1162 ]
1163 },
1164 {
1165 "cell_type": "code",
1166 "execution_count": 22,
1167 "metadata": {
1168 "ExecuteTime": {
1169 "end_time": "2018-12-10T06:54:18.551156Z",
1170 "start_time": "2018-12-10T06:54:18.509623Z"
1171 },
1172 "slideshow": {
1173 "slide_type": "fragment"
1174 }
1175 },
1176 "outputs": [
1177 {
1178 "name": "stderr",
1179 "output_type": "stream",
1180 "text": [
1181 "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/ipykernel_launcher.py:1: FutureWarning: The signature of `Series.to_csv` was aligned to that of `DataFrame.to_csv`, and argument 'header' will change its default value from False to True: please pass an explicit value to suppress this warning.\n",
1182 " \"\"\"Entry point for launching an IPython kernel.\n"
1183 ]
1184 }
1185 ],
1186 "source": [
1187 "s.sort_values('id').token.dropna().to_csv(tb_path / 'meta.tsv', index=False)"
1188 ]
1189 },
1190 {
1191 "cell_type": "markdown",
1192 "metadata": {
1193 "slideshow": {
1194 "slide_type": "slide"
1195 }
1196 },
1197 "source": [
1198 "#### Analogies to ID"
1199 ]
1200 },
1201 {
1202 "cell_type": "code",
1203 "execution_count": 23,
1204 "metadata": {
1205 "ExecuteTime": {
1206 "end_time": "2018-12-10T06:54:23.842956Z",
1207 "start_time": "2018-12-10T06:54:23.690010Z"
1208 },
1209 "slideshow": {
1210 "slide_type": "fragment"
1211 }
1212 },
1213 "outputs": [],
1214 "source": [
1215 "df = pd.read_csv(Path('data', 'analogies', 'analogies-{}.txt'.format(LANGUAGE)), \n",
1216 " header=None, squeeze=True)\n",
1217 "categories = df[df.str.startswith(':')]\n",
1218 "analogies = df[~df.str.startswith(':')].str.split(expand=True)\n",
1219 "analogies.columns = list('abcd')"
1220 ]
1221 },
1222 {
1223 "cell_type": "code",
1224 "execution_count": 24,
1225 "metadata": {
1226 "ExecuteTime": {
1227 "end_time": "2018-12-10T06:54:24.456585Z",
1228 "start_time": "2018-12-10T06:54:24.436606Z"
1229 },
1230 "slideshow": {
1231 "slide_type": "slide"
1232 }
1233 },
1234 "outputs": [
1235 {
1236 "data": {
1237 "text/html": [
1238 "<div>\n",
1239 "<style scoped>\n",
1240 " .dataframe tbody tr th:only-of-type {\n",
1241 " vertical-align: middle;\n",
1242 " }\n",
1243 "\n",
1244 " .dataframe tbody tr th {\n",
1245 " vertical-align: top;\n",
1246 " }\n",
1247 "\n",
1248 " .dataframe thead th {\n",
1249 " text-align: right;\n",
1250 " }\n",
1251 "</style>\n",
1252 "<table border=\"1\" class=\"dataframe\">\n",
1253 " <thead>\n",
1254 " <tr style=\"text-align: right;\">\n",
1255 " <th></th>\n",
1256 " <th>a</th>\n",
1257 " <th>b</th>\n",
1258 " <th>c</th>\n",
1259 " <th>d</th>\n",
1260 " </tr>\n",
1261 " </thead>\n",
1262 " <tbody>\n",
1263 " <tr>\n",
1264 " <th>1</th>\n",
1265 " <td>athens</td>\n",
1266 " <td>greece</td>\n",
1267 " <td>baghdad</td>\n",
1268 " <td>iraq</td>\n",
1269 " </tr>\n",
1270 " <tr>\n",
1271 " <th>2</th>\n",
1272 " <td>athens</td>\n",
1273 " <td>greece</td>\n",
1274 " <td>bangkok</td>\n",
1275 " <td>thailand</td>\n",
1276 " </tr>\n",
1277 " <tr>\n",
1278 " <th>3</th>\n",
1279 " <td>athens</td>\n",
1280 " <td>greece</td>\n",
1281 " <td>beijing</td>\n",
1282 " <td>china</td>\n",
1283 " </tr>\n",
1284 " <tr>\n",
1285 " <th>4</th>\n",
1286 " <td>athens</td>\n",
1287 " <td>greece</td>\n",
1288 " <td>berlin</td>\n",
1289 " <td>germany</td>\n",
1290 " </tr>\n",
1291 " <tr>\n",
1292 " <th>5</th>\n",
1293 " <td>athens</td>\n",
1294 " <td>greece</td>\n",
1295 " <td>bern</td>\n",
1296 " <td>switzerland</td>\n",
1297 " </tr>\n",
1298 " </tbody>\n",
1299 "</table>\n",
1300 "</div>"
1301 ],
1302 "text/plain": [
1303 " a b c d\n",
1304 "1 athens greece baghdad iraq\n",
1305 "2 athens greece bangkok thailand\n",
1306 "3 athens greece beijing china\n",
1307 "4 athens greece berlin germany\n",
1308 "5 athens greece bern switzerland"
1309 ]
1310 },
1311 "execution_count": 24,
1312 "metadata": {},
1313 "output_type": "execute_result"
1314 }
1315 ],
1316 "source": [
1317 "analogies.head()"
1318 ]
1319 },
1320 {
1321 "cell_type": "code",
1322 "execution_count": 25,
1323 "metadata": {
1324 "ExecuteTime": {
1325 "end_time": "2018-12-10T06:54:26.357446Z",
1326 "start_time": "2018-12-10T06:54:26.263768Z"
1327 },
1328 "slideshow": {
1329 "slide_type": "fragment"
1330 }
1331 },
1332 "outputs": [
1333 {
1334 "data": {
1335 "text/plain": [
1336 "0.332703213610586"
1337 ]
1338 },
1339 "execution_count": 25,
1340 "metadata": {},
1341 "output_type": "execute_result"
1342 }
1343 ],
1344 "source": [
1345 "analogies_id = analogies.apply(lambda x: x.map(token_to_id))\n",
1346 "analogies_id.notnull().all(1).sum()/len(analogies_id)"
1347 ]
1348 },
1349 {
1350 "cell_type": "markdown",
1351 "metadata": {
1352 "slideshow": {
1353 "slide_type": "slide"
1354 }
1355 },
1356 "source": [
1357 "### Generate Sampling Probabilities\n",
1358 "\n",
1359 "There is an alternative, faster scheme than the traditional SoftMax loss function called [Noise Contrastive Estimation (NCE)](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf).\n",
1360 "\n",
1361 "Instead of getting the softmax probability for all possible context words, randomly sample 2-20 possible context words and evaluate the probability only for these."
1362 ]
1363 },
1364 {
1365 "cell_type": "markdown",
1366 "metadata": {
1367 "slideshow": {
1368 "slide_type": "slide"
1369 }
1370 },
1371 "source": [
1372 "**SAMPLING_FACTOR**: used for generating the `sampling_table` argument for `skipgrams`. \n",
1373 "\n",
1374 "`sampling_table[i]` is the probability of sampling the word i-th most common word in a dataset\n",
1375 "\n",
1376 "The sampling probabilities are generated according\n",
1377 "to the sampling distribution used in word2vec:\n",
1378 "\n",
1379 "$p(\\text{word}) = \\min(1, \\frac{\\sqrt{\\frac{\\text{word frequency}}{\\text{sampling factor}}}}{\\frac{\\text{word frequency}}{\\text{sampling factor}}}$"
1380 ]
1381 },
1382 {
1383 "cell_type": "code",
1384 "execution_count": 26,
1385 "metadata": {
1386 "ExecuteTime": {
1387 "end_time": "2018-12-10T06:57:06.462506Z",
1388 "start_time": "2018-12-10T06:57:05.855693Z"
1389 },
1390 "slideshow": {
1391 "slide_type": "slide"
1392 }
1393 },
1394 "outputs": [
1395 {
1396 "data": {
1397 "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8leWd///X2XKSnGyQsCTsSwhGdkSoKW4IgsVtRi8BZ7T9Wq226kz7bTt1nHYej371VzvtzPy0Vvpl6taq4KVtUawKarFUBdlkkX0JS4AEQkKWk+ScnOX7xzk5JCQh21muhM/z8eAB5z738s5N8sl1rvu6r9sSDAYRQgjRd1kTHUAIIURsSaEXQog+Tgq9EEL0cVLohRCij5NCL4QQfZwUeiGE6OOk0AshRB8nhV4IIfo4KfRCCNHH2RMdIExuzxVCiO6xdLSCKYWekydPJjpCm3JycigvL090jFZMzQXmZjM1F5ibzdRcYG62eObKy8vr1HrSdSOEEH2cFHohhOjjpNALIUQfl9A+eqXUzcDNWutExhBCGCIYDNLQ0EAgEMBiufg1xrKyMjweT5ySdV60cwWDQaxWK8nJyR2ek/YktNBrrVcBq4D7E5lDCGGGhoYGHA4HdnvHpclut2Oz2eKQqmtikcvn89HQ0EBKSkq3tpeuGyGEMQKBQKeK/KXGbrcTCAS6vb0UeiGEMbrbNXEp6Mm5kUIvhBB9nDGFPmjgRRUhxKVp7dq1zJ49m6KiIp599tlW73s8Hh588EGKiopYuHAhx48fB6CiooLbb7+d/Px8Hn/88XjHbpcxhV5mQRBCmMDv9/P444/zyiuvsHbtWlauXMn+/ftbrLN8+XIyMzP59NNPuf/++3nyyScBSE5O5kc/+hE//vGPExG9XeYU+mD3LzQIIUS0fPHFF4wcOZIRI0aQlJTErbfeyurVq1uss2bNGu68804Avva1r/HJJ58QDAZJTU1l5syZOJ3ORERvlzmXt3twRVkI0fcEVvwPwePF7b9vsRAMdq0nwDJsFNZFFx/NXVpa2mIOmdzcXL744ot217Hb7WRkZFBZWUn//v27lCdezGnRS6EXQhigrV8eF4546eovmEQz587YXnbihBCx1VHL22634/P5on7c3NzcFrPpnjp1ikGDBrW5Tl5eHj6fj+rqavr16xf1LNGS0Ba91nqV1voBQProhRBGmDJlCsXFxRw7dgyv18tbb73FvHnzWqwzb9483njjDQD+/Oc/U1RUZPQ9ANJHL4QQzdjtdp544gmWLFlCIBDgrrvuoqCggF/84hdMnjyZefPmsWjRIh599FGKiorIysriueeei2x/xRVXUFNTg9fr5f3332f58uWMGzcugV8RWAzpawqe+HIHlv45ic7RijzcoOtMzWZqLjA3W7xz1dXVkZqa2ql1Y9V101OxytXWuQlfEO7wo4Q5F2Ol60YIIWLCnEIvXTdCCBETxhT64I5NiY4ghBB9kjGFniMHE51ACCH6JHMK/YDBiU4ghBB9kjmFPndYohMIIUSfZE6hb/QmOoEQQgDdn6YY4Omnn6aoqIjZs2fz8ccfR5Z/73vfY9KkSVx//fXx+BJaMKfQ+xoTnUAIIXo0TfH+/ftZuXIlf/nLX3j11Vf513/9V/x+PwBKKV599dW4fz0Qo0KvlHIppbYopRZ2eqNGKfRCiMTryTTFq1ev5rbbbsPpdDJ8+HBGjhwZmfly1qxZZGVlxf3rgU5OgaCUegFYCJzWWk9otnw+8DRgA36rtX4q/Na/ALpLSbzyhCkhxHm/3VxGcWVDu+9bujFN8ah+yXzzikEXXacn0xSXlpYyY8aMFtuWlpZ2KWMsdHaum5eAZ4HfNS1QStmAXwNzgRJgk1LqbSAP2A0kdymJp75LqwshRCz0ZJrizmybCJ0q9FrrdUqpkRcsvhI4qLU+DKCUWgHcCqQBLqAQqFdKvau17vi2V3lmrBCimY5a3iZOU5ybm8uJEycuum0i9GT2yiHA8WavS4CZWuuHAZRSXwfK2yvySqkHgAcAtNYkWyEjx7xJzex2OzmSq0tMzWZqLjA3W7xzlZWVYbd3vix1Zd3OuuKKKyguLubEiRPk5uby9ttvs3Tp0hbHmj9/Pn/4wx+YNWsWq1atYvbs2TgcDhYsWMBDDz3Egw8+SGlpKcXFxcyYMQObzQYQ+bs7uZ1OZ7f/L3pyltr6PBL53KK1fuliG2utlwHLmrZrqDqHV2bv6zRTc4G52UzNBeZmi3cuj8cTKYYdieXslU888QR33XVXZJrisWPH8rOf/SwyTbFSikcffZSZM2dGpin2+XyMHTuWW265hdmzZ2Oz2XjyyScJBoP4fD6+/e1vs379eioqKpg8eTLf//73Wbx4caczeTyeVv8Xza8lXExPCn0J0Pwup6HAyXbWbVPzJ0wFPe1fdBFCiHiaM2cOc+bMabHsBz/4QeTfycnJLFu27MLNAPjud7/LI4880mp58znr460nhX4TkK+UGgWcABYBS7qyA631KmAVcD9S6IUQIiY6NY5eKbUcWA8UKKVKlFL3aa19wMPAamAPoLXWu7qdRIZXCiFETHR21E2bHUla63eBd7t7cOm6EUKI2EvoM2Obd900NPpJS2QYIYToo4yZ68YiLXohhIiJhLbom3fdBLwy140QQsRCQlv0WutVWusHAAKN3i7PWyGEELEQi2mK29vniy++SFFREUOGDKGioiImX48xXTfBIDInvRAi4WIxTfHF9jljxgxWrFjB0KFDY/Y1GVPo/RYLMpZeCJFosZim+GL7nDBhAsOGxfYJe8b00QctFqiugvTMREYSQhjiy611VJ/zt/t+d6YpzsiyMWFa6kXXidU0xR3tM5aMGV4ZxAoNdYmMI4QQMZmmOBBoPbdjPKcvTmihby5gsUCDzEkvhAjpqOXd26Yp7mifsWRMH30Ai7TohRAJN2XKFIqLizl27Bher5e33nqLefPmtVhn3rx5vPHGGwD8+c9/pqioCIvFwrx581i5ciUej4djx45RXFzM1KlTO7XPWDKmjz5gsRJsqG9z7mMhhIgXu93OE088wZIlSyLTFBcUFPCLX/wiMk3xokWLePTRRykqKopMUwxQUFDALbfcwnXXXReZprhp2uW29gnw/PPP89xzz3HmzBluuOEGrr/+en75y19G9WuyGDJ2Pfizh/+Nf5w5DOucmxOdpQWZJ7zrTM1mai4wN1u8c9XV1ZGaevEumyaxnI++J2KVq61zE77A22H72Jium/LkLKhzJzqGEEL0OcYUeqvFCnW1iY4hhBB9jjGF/tOcCdKiF0KIGDDmYuw11fsI2mR4pRBCRJsxN0x9mHU53y7fmcg4QgjRJxnTdQNAvYyjF0KIaDOq0Ne6ZVIzIURife9732PSpElcf/31Xd52x44dXHPNNRQVFfHjH/84MiXCf/7nfzJ9+nTmzp3L3Llz+eijj6Id+6KMKvQ7HAMTHUEIcYlTSvHqq692a9vHHnuMX/7yl3zyyScUFxezdu3ayHv3338/H3zwAR988AFz5syJVtxOMarQ73UOloePCCESatasWWRlZbVYduTIEe6++27mz5/P7bffzsGDB1ttV1ZWRk1NDTNmzMBisXDHHXfw/vvvxyv2RRkzqRlAo8UGnnpI7tydcUKIvmvdunWcOXOm3fe7M03xgAEDuPrqq7uc5Yc//CFPPfUUo0ePZuvWrTz22GORuW6alJaWkpubG3ndfIpiCD1J6s0332TSpEn85Cc/afXLJJaMGV6ZZg0AwdCc9FLohRCGcLvdbNmyhW9961uRZV5v66fhXWx643vuuYd//ud/xmKx8B//8R/89Kc/5b/+679iF/oCxgyvzEmCs84sqK6EgbkdbSqE6OM6annHa66bQCBARkYGH3zwQYvlfr+f+fPnA6HZLO+55x5OnToVeb/5VMQDBgyILL/77ru59957Y567OWP66LNTbJx1ZkL1uURHEUKIiPT0dIYNG8aqVauAUMt9165d2Gy2yMXVH/zgBwwaNIi0tDQ2b95MMBjkzTff5MYbbwRC/fdN3nvvvcjMlfFiTB99tsvBAWcmwfoqmapYCJEw3/72t1m/fj0VFRVMnz6d73//+zz77LM89thjPP300/h8Pm699VYuv/zyVtv+7Gc/47vf/S4NDQ1cd911kSGaTzzxBLt378ZisTB06FB+/vOfx/VrMqfQpzqpTkrDW19GcqLDCCEuWU1zy1+oM0MuJ0+ezLp161p1Kf3qV7+KSrbuMqfrJi1U3s+6W1/kEEII0X3GFPqcDCcAZxraf+q7EEKIrjOm0GcnOwCokAa9EJcsuWGyfT05N8YU+qzk0OWCikZjIgkh4sxqtRr5eMBE8/l8WK3dr43G3DDltFtIDXip8NsSGUkIkUDJyck0NDTg8XgiNxu1x+l04vF44pSs86KdKxgMYrVaSU7u/jAVY26YIgj9AvWcDSYlMpIQIoEsFgspKSmdWlceqN55xvSTBIOQbfFSaXEmOooQQvQpRhX6/vYAFdbO/TYXQgjROcYUeoBgkpMzyf3w1dYmOooQQvQZxhT6YDBIMCl0seHo8bIO1hZCCNFZxhR6vw9uGuUCoPx0RYLTCCFE32FMoS8+4GHo8ND0xCfLaxKcRggh+g5jCr2vMUh6ZhouXwOlbrlhQgghosWY2Ss9ntDtvYMCtZwKGBNLCCF6PWNa9PXuAAB5Ni+nrPIoQSGEiBZjCr0rPRQlL9lCuSMDr0dmNxNCiGgwptCfPhXql8/tl0LAYqXsaEmCEwkhRN8Q9c5wpdRlwD8BOcBHWuulXdl+xNABcLqRw8fOMGzc6GjHE0KIS06nCr1S6gVgIXBaaz2h2fL5wNOADfit1voprfUe4EGllBX4n66Ecdf6GTZyCNYthzl+rr4rmwohhGhHZ7tuXgLmN1+glLIBvwYWAIXAYqVUYfi9W4BPgI+6Eqaq0k9SSjKDvFWUhC/OCiGE6JlOFXqt9TrgwttVrwQOaq0Pa629wArg1vD6b2utrwLu7kqY0pJGAPJtdeygH355AIEQQvRYT/rohwDHm70uAWYqpa4F/g5wAu+2t7FS6gHgAQCtNQD9+rvIyclh2ohs1p1IprGqlkEFY3sQsefsdjs5OTkJzdAWU3OBudlMzQXmZjM1F5ibzcRcPSn0bT3+Jai1/hj4uKONtdbLgGVN2zmTLVRX11FeXs7ATCecgJ17i0nOzupBxJ4z8SECYG4uMDebqbnA3Gym5gJzs8UzV15eXqfW68nwyhJgWLPXQ4GTXdmBUupmpdQygOQUK/V14Zum8gYAUFYp0xULIURP9aRFvwnIV0qNAk4Ai4AlXdlB80cJpqRacdf4Acgc0J803zH2nfWysAcBhRBCdLJFr5RaDqwHCpRSJUqp+7TWPuBhYDWwB9Ba613dDZKcYqG+PtSit1qtzLRWsIlsuUNWCCF6qFMteq314naWv8tFLrh2RCl1M3Cz1hpnMvgaQ7NY2h0Wpg/N5KOTyRzafYjLpl7W3UMIIcQlL6FTIGitV2mtHwBIcoZmr2zqp580pQCn38vHu7vU7S+EEOICxsx1k5waitJU6NP7ZXC5v5ztnlQCAbl5Sgghuiuhhb7FqJvkUIu+of58UZ8+wMEpZz+K9x5OTEAhhOgDjOm6cThDw/Ib6oOR92dfNRFr0M/fdhxLTEAhhOgDjOm6sVrBkWRp0aLP7J/FlMYyPm5Ip9HbmMB0QgjRexnTdRMMBklJaVnoARaMzaLSkc7mz7YnJKMQQvR2CX04a/MbpoLBIM4Ua4uuG4BpsyaRdWgbHx+p4isJSSmEEL2bMV03wWCQ5BRrqxa93WGnyFHFZtsgKs+cTVA6IYTovYwq9ACehiCN3pat+vkzx+Kz2nn7L9J9I4QQXWVUoW8q9iePt5z2YPjYEeR5KtnqTkpENCGE6NWMuRhbU1PDZZNSADhX4W+17rWZHo44c6T7RgghusiYcfS7du0iOcVKZj8b1edaF/qJowcC8OXOQ/ENKYQQvZwxXTdNUl1WzlX4W12UHTdxHNneaj484k5QMiGE6J2MKfQnT4YmLxsxNtQPv3t7fYv37XY7c9PcbHPmUnr0RNzzCSFEb2VMoe/fvz8AOQNDQ/tPHG3E62nZqr/hqkKswQBrNuyLez4hhOitjLkYW1VVBYDFYqFgQjIAlRdclB0wZBDTGkv5qD5DpkQQQohOMuZibGVlZWT5yPxQ983Gda374+fnZ3HOkcZfPtoYp5RCCNG7GdN1A+dvmkpKOh9r86cti/3UWZMY7TnDm6V2fI2+uOYTQojeyKhCX15eHvn3TXdkAnCqpJH9uxoiy+12O3eMSuZ0Uiab12+Le0YhhOhtjC30NpuFr85JA2Dflw3s332+2M/86lRyvNX830NBas5Vxz2nEEL0JkYV+kOHWt4M1S/HzpyvpQOwb2cDZ0+HumrsDjvfn5JORVI6r/x5U9xzCiFEb2JUoe/Xr1+rZalpNorCLfvP1tZGll829TKme0/yPkPkUYNCCHERxgyvTEtLw+1u+67X/jl28gudAJw4dn7Cs4dvmoQt4OeFz47i88mFWSGEaIsxwyvT0tKora1td90xBaGx9VvX10VG5/QflMM3MivY4czlLx/KcEshhGiLMV036enpFy30jiQL/QfYAHj/T1WR5QsWfIVRnjP8vtSJu7r97YUQ4lJlTKFvatE3tdbbctV1ob56XyN8vi5U1O0OOw9OG0C1w8WLb22IS1YhhOhNjCr0Pp8Pj8fT7joWi4Wv3RkaX3/6lI9Vr5/D7w8yfsp4buIEH1iH8s47f4tXZCGE6BWMKvQQegDJxVitFhb8fWbk9btvVlFT7ec+dQ2TPKd4obI/h3YfjGlWIYToTYwp9OnpofHyHRV6ALvdwkKVydCRDgA+fq+GY8V+Hro+H5e/gZ9/Xk51ZVUHexFCiEuDMYW+qUXf3hDLC1ksFqbOdJHVP3SBdtcX9RzYm8EPJzg548jg3/+0g3p3XczyCiFEb2FMoU9NTcVisXS60DeZPTedGV91AVBdFeDIoaHcl+bnsHMAT73+OYFAoIM9CCFE32bMDVNWq5XU1NSLDrFsz+AhDm76+0xyh4a6coKeYXzTPpjjzmGsWvVJdEMLIUQvY0/kwbXWq4BVwP0ALperyy36Jja7hSuKXJwq8bL501CXze32HD5y2xm/fS8Fk8dHK7YQQvQqxnTdQM8KfZPcoUncfFcWYy8LTZkwx5bF/r2D2fH5gWhEFEKIXseoQt/RNAhdcdmkFK4oSo28PnpkACeOnI7KvoUQojcxqtB7PB4aGhqoq4vOaJmm1r3FUQbA5xvsuGtlJI4Q4tJiVKFvGmK5a9euqO534d8VUBs4jcNi5U9vn6XB3f7dt0II0dcYVehnzpwJQFVV9G92Wrx4HHXBUjJt6bz5VmnU9y+EEKYyqtA7HA7y8vLYvXt3TPZ/16LxVPmqybRl8sbre2JyDCGEMI1RhR7OT4VQWVkZk/3ftjCb+kAjyeRKsRdCXBKMK/RN3TcrV66Myf7TMl3ccEMS1X43yeSyYvk+/H5/TI4lhBAmMK7QZ2VlAaHJzaqrq2NyjP4DMrnz7wdS5a/BZR3E8tdPUXW248nUhBCiNzKu0AMsWLAAgNOnYzfuPcnpYPFdedT4z5BpS2PtBz4++8u+mB1PCCESJSaFXil1m1Lqf5RSbyml5nV1+5EjRwLw7rvv0tjYGO14ETabjSVL8klJP4kVOHtmEPr1vTE7nhBCJEKnC71S6gWl1Gml1JcXLJ+vlNqnlDqolPoRgNZ6pdb6fuDrwF1dDeVwOCJj6pcuXdrVzbvshpsKmXV1kNqAhxQG88prR6UrRwjRZ3SlRf8SML/5AqWUDfg1sAAoBBYrpQqbrfJv4fe77Bvf+AYuV2j64ddee+2iz5KNhoF5/bnjjv5U+avItGWy7kM/r6+QC7VCiN6v04Vea70OqLhg8ZXAQa31Ya21F1gB3KqUsiilfg68p7Xe2p1gFouFxYsXA1BeXs6LL74Y82LvcDj4hyUjsCSdBCDVMojnlx5i77ajMT2uEELEUk+nKR4CHG/2ugSYCTwC3ABkKqXGaq1/c+GGSqkHgAcAtNbk5OS0eYCf/OQn/PSnP6W2tpaSkhKmTp3aw8gd+/r9V1PvbmDZ83vIsKVzYF8mX+w5xEMPTcduT+jMzhF2u73dc5ZopmYzNReYm83UXGBuNhNz9bRqWdpYFtRaPwM8c7ENtdbLgGVN25SXl7e77re+9S1efvll/vSnPzF06FAslrYOG313LxnGug8PUnU2hzRrNr//v0cYMKiMWdcWxOX4F5OTk8PFzlkimZrN1FxgbjZTc4G52eKZKy8vr1Pr9XTUTQkwrNnrocDJHu6zFafTyYwZMwDYsGFDtHd/UX+3aBbzbk2lyh+af+dM2SBeefUY1eeiM52yEELEWk8L/SYgXyk1SimVBCwC3u7sxs0fJdiRSZMmhQ64aVNMh1y2xZmcxD8sGcGg3NP4g0Ey7Rn8dbWPj96NzZw8QggRTV0ZXrkcWA8UKKVKlFL3aa19wMPAamAPoLXWnZ5jWGu9Smv9QGfWtdlskf75pUuXxr3YA1x59ThuvjMDdyA0v31dTR6vLz/D1s8Oxj2LEEJ0liXWI1kuRil1M3Cz1vr+kyc71+Pz/PPP43a7sVqtfOc734l5f317/W0Hd5ewcbuddGsyAFV+N1dfaWP42MExzdNRLhOYms3UXGBuNlNzgbnZEtBH32ERTOgUCF1p0Te57777GDx4MIFAgF/96lcxmbu+M8YWDmXJ4sEMGFiGJ+gn0+Zi+5ZkXnvtgDzYRAhhFCPnuunInXfeSXZ2NgAvv/wy7733XsKyzLqugDsWZWNzhj6RpNsG8ME79az7QKZSEEKYIaGFvisXY5uzWCzcfffdzJ8fulH3wIEDvPTSS9GO1yU33VbIdfMdVPtD95RVVQzmzRVn2fS3/QnNJYQQva7rprlx48ZFin11dTXPPPMMFRUX3rwbP2mZLu5eMpox+eeo9tfhtNgoPTmQV147TtmJswnLJYS4tPXKrpvmxo0bxze/+c3I61deeYWXX345gYmgcNpI7l6SR3ZO6Nm0mbZ0Nn5i49XXDnHq6JmEZhNCXHp6faEHSE1N5dFHH2XevNCMyFVVVTzzzDOsXbs2obmumjOem+5Ix287AUCGLZvNGxy8+tohThyJ3Vz7QgjRXK/so2/P+PHjeeCBByJTHO/cuZPf//73MZ8M7WJsNhu33XE5N97mikyWlmHLZuvnSbzy2lGK955IWDYhxKUhoePomwl2dhx9Z5WXl/Paa69FXk+dOpXZs2d3eT/RHhPrrq3jozXFBL15WMP3ANQFGpl8WR3jp4xIWK5oMjWbqbnA3Gym5gJzs8k4+jjKyclp0Xf/xRdf8Mwzz9DQ0JDAVOBKS+WWv7uchXdmkNm/lIagj1SrgwP7Mlm+vIyPV+9JaD4hRN/TZws9nO+7X7RoUWTZsmXLeP755xOYKsRms3H13PHcuSiHlPST1AY8pFmd1JzL5a0Vlbz1h93y0BMhRFT0qT769gwcOJBHHnkk8trtdvPMM8+wd68ZNzXdcFMhixcPoqCwhip/FVaLBasvj3ffrGH58v2Un6xMdEQhRC/WZ/vo21NbW8sLL7zQYtm1117LxIkT25w3JxH9gGdPn2P1h2dItw2ILKvx13PFRB/jJg5LWK7OMjWbqbnA3Gym5gJzs5nYR3/JFfomGzZsYOPGja2WL1iwgPz8/MjrRH4zNTY28tH7B3C7B5NkCX34qva7KRzVwNybZxr5TQ7yA9gdpmYzNReYm00KffviXuib7N69mw8//LDV8tTUVG644QauuOKKhH8z+f1+Nq47yOmygZGROt6gH6xlFOanctnUkQnNdyH5Aew6U7OZmgvMzSaFvn0JK/TNbdu2jXXr1rX53k033cTYsWPjnKgln8/H1s8Os7/EQaatX2R5tb8ep+0cE8e7KJg8PIEJQ+QHsOtMzWZqLjA3mxT69hlR6Js0NDSwZcsWtmzZ0ub7N954I+PGjYvbs2vb0lgf4KMP91Jek0qmLSuyvMpfS39XFddcN4pklxObzRb3bPID2HWmZjM1F5ibTQr9Bbrz4JF469+/P5988kmb3TsWi4Xrr7+e8ePHx72gNv9mOnrgFHv2VnKuNgeXNSmyjjvgJdN1huvn5eNMTmpvVzHNZhJTc4G52UzNBeZmk0LfPqNa9M01/08LBoPs37+f1atXt7numDFjmDNnDsnJyXHN1dzhvSfY/mUVft8gnJbQLx9fMEBdoJ7CUQ0Mzksnd8TAhGRLNFNzgbnZTM0F5mYzsdDbYx+l77BYLBQUFFBQUEAwGOTw4cOsWbMm8vzaQ4cOcejQIQCmT5/OlClTcLlccc04evwQRo8fgs/no6ykgrWfubFaUsiwuSg55qLkGFR9eox0Zy2zZg1k0JCcuOYTQsSftOg70Nnfzm63mzVr1nD8+PFW702cOJEZM2ZEJluLZ64mf/twDzW1QfyevBbLq/w1zJxkIW9kf1xpqQnJFi+m5gJzs5maC8zNJi36PszlcnH77bcDoQnVtm3bxu7du4HQLJo7d+4EICUlhfnz5zN48GAcDkfc8s2+4TIAPA1ePPUe3n3/JAEyybSls3cX7N3lpcp/huED6kl2WpleNCYhF3KFENEnLfoO9PS3c3V1NZ9//jl79rQ9Wdno0aO5/PLLGTVqVFxzNVnzzm7cDWD3X9jSr8VhrWX4IAtfua4gIdmizdRcYG42U3OBudmkRX+BZqNuEhkjpjIyMpg7dy5z584lGAyye/dudu/ezalTpwA4fPgwhw8fBsDpdFJQUEBhYSEDB8b2gmmTeQsLgdC0CyePVVJT7aXs9ADSrC5sljTKT8Mrr5VA0MdtCweQlhnfaw5CiJ6TFn0HYvnbub6+nhMnTrBmzRp8Pl+r910uF5MnT2b48OFkZ2e36EqJR6vhr2v2cLw8jRSrKzIFgzcYwBts5NqrrFjtVvoPzGjVBSUtra4zNZupucDcbNKiFy2kpKQwduxYxo4dSzAYpLa2lsOHD7N582bcbjdut5vPPvuMzz77DAhGYy6HAAASkElEQVSN6Z89ezaDBw+OS75r5oX69RvcHla9cxh/wIqV/qRZnWzeEFqnJnCWy0fXAZCR6WTU+CFxySaE6Dxp0XcgUa2GYDBIRUUFR48eZefOnVRVVbVaJz8/n8zMTHJzc8nLyyMpKSnmd+vWVrn528dH8AcseDznJ1trYks+wYD+qXg8XiZdMYSMrOiNNOopU1uAYG42U3OBudmkRS86zWKxkJ2dTXZ2NtOmTcPn83H48GG2b99OZWUlDQ0NHDhwoNV2I0eOJCsri7y8PAYOHEhGRkZUc6Vlulhw6+VAqF9/947QtYbSs5BMLv6GIZSGf2e/824NA7OOAZCcZOUr1+XLSB4hEkBa9B0wtdWQlpbGiRMnKC0tZd++fVRXV1NRUdHmumPGjMHpdDJx4kR8Ph+DBw+OesH1+/0c2nWC+nofrtRUduxKwnFBa98dKCM7I/TUrKF5Li6b2vln5EaDqf+XYG42U3OBudmkRS+iJjk5mczMTDIzMykoCA1/DAQCVFdXc/r0aY4fP86ZM2c4ffp05G7dpnH9AElJSYwePZrBgwdTWFiIxWLpUfG32WyMmxSaOTMnJ4cBecWcPhXqbjpd5qaifBAu6yAaakPr79jbSGnZ+TzpaQ6u+Gp+q/0KIXpOCn0fYrVaycrKIisri3HjxgHg8Xiw2Wzs2rULm83G+vXryczMpLS0lL1797J3714+/vhjALKyshg5ciSNjY3MmjWLhoYGXC5Xt+buycxOJzM7HYD8CVB6rJza2tCD2bfsqCXVOpjaqvNj92ur4OPVe0hPC43gsdksFE4dTpIzfjeVCdFXSaHv45xOJwCTJ08GYMKECQBUVlbi9XrZuHEjycnJ7Nmzh3PnzrFt2zYAdu3aFdnH+PHjCQQCXH311dTV1WG328nKyqIrBg8/P6fO8DGNHN5zkqZuw+Mn6vC486g5l0vNufPbHCnZz8wZ5x+nmJGdFrVpGoS4lEihv0T16xd6cMnNN98MwNVXX43P5yMpKYlNmzaRlJTEtm3bqKurizxEff/+/ZHtJ06ciNvtZtSoUYwbN46amhogNAS0I0lOB+OnnO+fHzfJz46NR/A0nL+X4FTZAOz+IZFhnAC1gRoWL5ZCL0RXyXz0HbiUL/gEg0GCwSCNjY1s3boVp9PJyZMnOXz4MA6HIzJrZ3NTp07F4XBQUVHBtGnTGDRoEF6vF5/P16WZPDf9bT9nzp7ff0VdKpm2fjQEW95Y5gk0sEgNxm7vuM1i6v8lmJvN1FxgbjYTL8bKqJsOyDdTaz6fD7vdTnFxMUeOHCEtLY2Kigr27dsHhC4UNzSE+uPtdnvkrt9rrrkGm82G2+0mJyeHMWPGdPqYR/ed5LOttQSD57+nA6SQaUujyl+JhebfxwGKpqcysqDl/D2m/l+CudlMzQXmZjOx0EvXjeiyptbzqFGjIpOxBYNBrrrqKpxOJ3l5eaxcuZLq6mrS0tKoqanh0KFD/PWvf22xn7Fjx+J2u6mpqcFut7No0SJ8Ph91dXV4PB4GDRoUOdaIgjxGXDC32q4txWzf34DNcr47x4IFlzWJz7aUc6S45URySclJjC/Mof/AzGifEiGMJoVeRIXFYiE9PT3y+qtf/Wrk38FgkJKSEux2Oy6Xi3379rFx40bOnj2Ly+XC7XYTDAb5zW9+02KfhYWFjBgxgrq6OtxuN1lZWRQWFkbev3z6KC6f3jKH3+/nj2+cI92WQ82FNxNXwZpTpSxaLIVeXFqk0IuYs1gsDBs2LPJ6xowZzJgxI/La7XazadMmHA5HZDjnmjVrIjN9NldVVUVdXV3kT2pqKgsXLoxM/WC1Wim6ys+58ppWOXbsSyXZMpA3VrT+WN0Y9LFwQZpRUzYIES1S6EXCuVwurr322hbLMjMz8Xq9uFwuUlNT2bNnD59++imbN28mJSWF1NRUvF4vZWVlrFy5Eo/HQ319PXV1dfj9fhYuXMjo0aMJBoOR946dOMjZ6jQslpbf9gGcZNoyWLP6OCnOQJsZxxdkMuayobE6BULElBR6YaTc3NwWr6dNm0ZhYSFOpxOrNTS1wtGjR1m7di319fW4XC6ys7NxOBzs2LGDd955B5fLRX19PYHA+eI9ffp0rrrqqkjxr6+v58Cuo5w6WY0reSgBT9t51m89x5jLYvblChFTUuhFr2CxWEhJSWmxbMSIEXz9619vsaypBe/1eklJSYm0/lNSUli7di1btmxh69attDXabOHMW3E4HHi8nvA+PHi8Hr7cV0e6cxivvtb+yLC8flVct0B+EwgzSaEXfYrFYuHGG29s8z2Hw8GRI0civwCa/pSVlbFhwwbeWfNWu/ttbDxNWvqVEPQRCHoJBrwEgo0EA14cWDl+Jp3y0nPtbg+Qme2K63OChWgihV5cMmbOnNnm2P0BAwZQX1+P3W4nOTmZlJSUFn+/9957lJeXUlX5TotuoObSUwpY/9dQH37oRrNG/EEvgYCHQMCDP+jFi417750Y069RiLZIoReXvNTUVK655pp2358xYwbHjh1rUfyb/1mx4nVq6vfR0Hgcf6CRQMAHtO4actiz2fZ521M4pDhPUlV7Dp+vkYLLhpM9qF+0vjwhpNAL0ZH8/Hzy89ufQnnKlMmUlZVFCr/T6Wz1y+Cttz6isfEsW7fvJRD0Egh4CQS9+AOeUFdQs6kdtu/N5aEH74zHlyYuEVEv9Eqp0cDjQKbW+o5o718I01x11VUdrlM4fgTbd1bi8ZVgszmwO+zYrA5sthTsNgdOZzLBoIWTZ0rwNVbw+d+2t/WhoBVXWjITphV0vKK4pHWq0CulXgAWAqe11hOaLZ8PPA3YgN9qrZ/SWh8G7lNKvRmLwEL0RtdcdzXXXHd1u+83zY+ydOlyGv21fP7FX9td90K1NfXMumZKNGKKPqqzLfqXgGeB3zUtUErZgF8Dc4ESYJNS6m2t9e429yCE6ND8G4rYvu0QFix0NFfVObebquqDlFdUxyec6LU6Vei11uuUUiMvWHwlcDDcgkcptQK4FZBCL0Q3jcofzqj84Z1ad9vGPazbcJDDx7fzzDM7On2M1JRhfPP+W7sbUfRCPemjHwIcb/a6BJiplMoGngSmKqUe01r/rK2NlVIPAA8AaK3Jyclpa7WEs9vtRmYzNReYm83UXNC9bNfO+wpf7jmK19f6uQDtqas/jbexttPH6mvnLB5MzNWTQt/W58qg1vos8GBHG2utlwHLmrYzcV5pkDmvu8PUbKbmgu5n+4d753dp/aVLX8fvr+I/fv5ip9a3WCwt7iLun57OberaLh0zVkz9/0zAfPQd6kmhLwGGNXs9FOjS00OaPWGqBzGEEJ2V7HRRU3uaWveRbmwdxF2XBFwb3VAi5npS6DcB+UqpUcAJYBGwpCs70FqvAlYB9/cghxCik77xvxZ2af3mrdPnf7sKd92RGKQSsdbZ4ZXLCf0az1FKlQD/rrV+Xin1MLCa0PDKF7TWu2KWVAiRUDarFQjy3HOv9Gg/FmsS995zM6mpKR2vLKKis6NuFrez/F3g3e4eXLpuhOg9CsYMYdvuGoKduZOrHcGAB5+3giP7Siic2v7dxiK6EjoFgnTdCNF7fOWaKXylhzdm/fH1tZSU7aShoZ2J/0VMyFw3Qoi4cThsAGzYtpON2/d0sHYXWSx8dcZEmRKiDQkt9NJ1I8SlZdy4IRwrPUow4MFPNFv1Qfz+GrbtTJNC3wbpuhFCxE3BhDEUTGj9TIDuaD4iqKqimpdfeQl/O88LuNRJ140QotdLdjkB8Pl9lBSfiumxrHYrecMGxfQY0SaFXgjR6zmdTizYcNcd4Y+rjsT8eDn9Cljyj20/stJE0kcvhOgT8kdO4ey52pgf5+y5A9R5eteoIemjF0L0CfNvKYrLcX71q8MEg73rWoA10QGEEKI3sWBrMdFbbyB99EII0RUWKx5PKb/+9UuJTsKTT/5rp9aTPnohhOiCfhlDqa6rafd9C5163G9cSR+9EEJ0wd33XPwZACbOky999EII0cdJoRdCiD5OCr0QQvRxcjFWCCH6OLkYK4QQfZx03QghRB8nhV4IIfo4KfRCCNHHWQyZs8GIEEII0QtZOlrBiBa9UmoLobDG/TE1m6m5TM5mai6Ts5may+RsCcjVISMKvRBCiNiRQi+EEH2cKYV+WaIDXISp2UzNBeZmMzUXmJvN1FxgbjbjcplyMVYIIUSMmNKiF0IIESMJf8KUUmo+8DRgA36rtX4qxscbBvwOGAwEgGVa66eVUv2B14GRwBFAaa0rlVKWcL6bgDrg61rrreF93Qv8W3jXT2itX45CPhuwGTihtV6olBoFrAD6A1uBf9Rae5VSzvDXMR04C9yltT4S3sdjwH2AH3hUa706CrmygN8CEwgNh/1fwD4SfM6UUt8FvhnOtBP4BpBLAs6ZUuoFYCFwWms9Ibwsat9XSqnpwEtACvAu8E9a6059JG8n2y+AmwEvcAj4htb63MXOR3s/r+19n3YnV7P3vg/8AhigtS434ZyFlz8CPAz4gD9rrX8Yz3PWHQlt0YeL2q+BBUAhsFgpVRjjw/qA/621vgyYBXwnfMwfAR9prfOBj8KvCWfLD/95AFgazt4f+HdgJnAl8O9KqX5RyPdPwJ5mr38O/Hc4VyWhbyTCf1dqrccC/x1ej/DXsgi4HJgPPBc+zz31NPC+1no8MDmcMaHnTCk1BHgUuCL8g2gj9LUn6py9FN6+uWieo6XhdZu2u/gTMDrO9gEwQWs9CdgPPBbO0Ob56ODntb1z3p1cTQ2yucCxZosTfs6UUtcBtwKTtNaXA78ML4/nOeuyRHfdXAkc1FofDv8mW0HoJMaM1vpUUytAa11DqGANCR+3qXX5MnBb+N+3Ar/TWge11huALKVULnAj8IHWukJrXUnoh6Yr30StKKWGAl8j1HIm3IK5HniznVxNed8E5oTXvxVYobX2aK2LgYOEznNPcmUAVwPPA2itveGWX8LPGaFPpSlKKTuQCpwiQedMa70OqLhgcVTOUfi9DK31+nCL9HfN9tWtbFrrNVprX/jlBmBos2xtnY82f147+D7tcq6w/wZ+SMubKRN+zoCHgKe01p7wOqebZYvLOeuORBf6IcDxZq9LwsviQik1EpgKfA4M0lqfgtAvA2BgBxljkf3/J/TNHQi/zgbONfthbH6MyPHD71eF149FrtHAGeBFpdQXSqnfKqVcJPicaa1PEGpRHSNU4KuALZhxzppE6xwNCf87Fhkh1BX3XjezXez7tMuUUrcQ6rrcfsFbJpyzccBspdTnSqm/KqVmdDNbVM9ZRxJd6Nu6qysuw4CUUmnAH4B/1lpXX2TV9jJGNbtSqqkvcEsnjh23XGF2YBqwVGs9FXBzvguiLfE6Z/0ItaRGAXmAi9BH5PaOEc9z1pGuZolZRqXU44S6NF9NdDalVCrwOPCTNt424ZzZgX6Eun1/AOhw69yEbO1KdKEvAYY1ez0UOBnrgyqlHISK/Kta6z+GF5eFP+oR/rvpI1l7GaOdvQi4RSl1hNDHu+sJtfCzwt0SFx4jcvzw+5mEPmbG4pyWACVa68/Dr98kVPgTfc5uAIq11me01o3AH4GrMOOcNYnWOSrhfNdK1DKGL2IuBO5udpGyq9nKaf+cd9UYQr+4t4d/FoYCW5VSg7uRKxbnrAT4Y7j7aCOhT9853cgWzXPWoUQX+k1AvlJqlFIqidDFjLdjecDwb9/ngT1a6/9q9tbbwL3hf98LvNVs+T1KKYtSahZQFf4IvhqYp5TqF25Zzgsv6xat9WNa66Fa65GEzsNftNZ3A2uBO9rJ1ZT3jvD6wfDyRUopZ/iqfj6wsbu5wtlKgeNKqYLwojnAbhJ8zgh12cxSSqWG/1+bciX8nDUTlXMUfq9GKTUr/LXe02xf3RIeDfIvwC1a67oLMrd1Ptr8eQ2fw/bOeZdorXdqrQdqrUeGfxZKgGnh78GEnzNgJaFGGEqpcUASoaKdsHPWGQkt9OH+qYcJ/UftCS3Su2J82CLgH4HrlVLbwn9uAp4C5iqlDhC62t80zPNd4DChiyv/A3w7nL0C+D+E/iM3AT8NL4u2fwG+p5Q6SKhf7/nw8ueB7PDy7xHuSgmfP02o4L0PfEdr7Y9CjkeAV5VSO4ApwP9Hgs9Z+BPGm4SGpu0k9P28jASdM6XUcmA9UKCUKlFK3Ud0z9FDhC7UHyQ0HLKpT7272Z4F0oEPwj8Hv7nY+ejg57W9c96dXO0x4Zy9AIxWSn1J6JP3veHWfdzOWXfInbFCCNHHJbrrRgghRIxJoRdCiD5OCr0QQvRxUuiFEKKPk0IvhBB9nBR6IYTo46TQCyFEHyeFXggh+rj/B7+We5xjkbXtAAAAAElFTkSuQmCC\n",
1398 "text/plain": [
1399 "<Figure size 432x288 with 1 Axes>"
1400 ]
1401 },
1402 "metadata": {
1403 "needs_background": "light"
1404 },
1405 "output_type": "display_data"
1406 }
1407 ],
1408 "source": [
1409 "df = s['count'].to_frame('freq')\n",
1410 "factors = [1e-2, 1e-3, 1e-4, 1e-5]\n",
1411 "for f in factors:\n",
1412 " sf = make_sampling_table(vocab_size, sampling_factor=f)\n",
1413 " df[f] = df.freq.mul(sf)\n",
1414 "df[factors].plot(logy=True);"
1415 ]
1416 },
1417 {
1418 "cell_type": "code",
1419 "execution_count": 27,
1420 "metadata": {
1421 "ExecuteTime": {
1422 "end_time": "2018-12-10T06:58:07.030148Z",
1423 "start_time": "2018-12-10T06:58:07.026839Z"
1424 },
1425 "slideshow": {
1426 "slide_type": "slide"
1427 }
1428 },
1429 "outputs": [],
1430 "source": [
1431 "sampling_table = make_sampling_table(vocab_size, sampling_factor=SAMPLING_FACTOR)"
1432 ]
1433 },
1434 {
1435 "cell_type": "code",
1436 "execution_count": 28,
1437 "metadata": {
1438 "ExecuteTime": {
1439 "end_time": "2018-12-10T06:58:07.362609Z",
1440 "start_time": "2018-12-10T06:58:07.192858Z"
1441 },
1442 "slideshow": {
1443 "slide_type": "fragment"
1444 }
1445 },
1446 "outputs": [
1447 {
1448 "data": {
1449 "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XucHFWd9/HPyQyXXAmhuU0SSJCgBLwgiLDeUNENyhJ3V4+AF1Tc7PqIwqKr8siDPOCyqOuy7Aq4EdmIi2Z/oiLPYxRR8YZooghouCUkQzIzCclMMrlDmOTsH6cGOp3u6Z5J91RV9/f9evVruqrPVP3qdHf96pw6XeVCCIiIiGTNmLQDEBERKUcJSkREMkkJSkREMkkJSkREMkkJSkREMkkJSkREMkkJKueccwuccz8e4vUZzrngnHv1aMYltXHOdTrnLi+a/plz7uY0Y6qHap/LYSxnj/qpUGaPOitdt3PuSufc8hrW1RR130yUoDLMOTfWOXe1c26Zc26Hc67PObfEOffRYSxmNXAk8Ns6xTTdOfcl59xy59zTzrn1zrnFzrlPOucK9VhHvdSp/kbbXwGXjsaKkh1ySB47nXNPOOf+yTk3bjTWX0fV6uyfgdMGJ5xzlzvnOkewHBll7WkHIEO6CXg9cDHwIDAJOAk4qtYFhBB2AWvrEYxz7mXAT4AngU8BS4kHOS8AzgU+AHy+wv/uH0LYWY84hmGf62+0hRA2jPIqvwF8DNgfeB0wn1hPHy5XOKX3cUjV6iyEsBXYuq/LkRSEEPTI6APoBy6qUmYB8OOi6aOBRwADDgBmAAF4dfL64PR7iMlmB7ASeFeV9TjgIeKOvr1SmaLnncBngRuBPmBJMv9i4AHiDmMtsBA4suj/zkjiewtwXxLf74ETksevgO3AYmB2Herv5cAPgHVJTEuAOSVlOoGriQlvU1L2oqR+/x3YCHSXrivZjouBbwPbgB7g0jLLvrxo+mfAzaXTwP9J6mtD8p6PLyozBrgGWJ9sw0LgEmCgyrbvsa5k3leANSXvxVuTen96cBuT9+f3wDNJfdxYEtMC4MfEFkl38p59GyiMoO7/MamDzUAv8DlgzBB1toA9vxNXAsuT5+9Ltqn4ceUQ9fER4NFk25cBn6bo8w/MBf6QbF8/8XN5Utr7jmZ5qIsv29YAc5xzU2op7Jx7KXGnfjdwbgjhmSGKfw64BXgZcBvwdefcKUOUfynwYuBzIYSBcgVC8o0t8lHizud04IKi+R9PlvWXxNbMwjKL+0fizuBkYCfwTWKC+EzRvP8cIl6orf4mJes/g7jDvAu40zl3XEm5jxB3UCcD/5Y8vktM7q8AvgT8m3Nudsn/fYa44zuJWOefd879VZW4S70dmJLEeD7wNuATRa9fQqzrS5P1LAauGOY6Bu0A9iuZ90Viy/h44A7n3EuAO4FfED8/FwBnA18u+b9Tk5jnEBPaS4ifuUHDqfseYj3/PfHg4JIRbt9/E9+HLmLX95HELsC9OOeuJH5WLyNu+8XA3xLfU5xzRwDfIn42TyB+zv8VKPv9kBFIO0PqUfkBvIrYnbaL2HqZTzxiK26pLCAeqb6eeAR3WckyZlC+BXV1SblfA/81RCw++b+TSuZ3EY9+twI/KJrfCfykhm08KVnu1GT6jGT6bUVl3pHM++uieX+ZzJuwL/VX4f8eBD5dsi13FE2PIR7N/7+SeRspakUl8X29ZNnfAH5VsuxqLaiHSpbxZeC+ounuMu/nQobRgiK2kE8nttAWlrwX7yn5v68Di0vmzQV2A0cXfS63AgcVlXlzsrxZw6z7X5aUuQboGqLOFlChBZVMXw50VqmPccRWUWmL7r1Af8lnd0at32k9hvdQCyrDQgj3Es/vvAb4GnA4sZvkTuecKyr6YuCHxB3dP9W4+PtKpu8FZgM4577snNta9DiKuAMr5zXEo+hFwNiS1xaXFnbOneGcu8s5t9o5t4XYdQSxa7LYg0XPB8+hPVRm3mEV4qqp/pxzhzrnbnTOPeqc63fObSUeDVeMJ4Swm9id9lDJvHVl4qlYz8PwQMl0d7ItOOcmAR3Ab6qst5ILkm1+Gvglsdv3opIype/jCcTWU7GfEz8jxdv2cAhhU9H0vcnf45PYa637cnU4Ndn2RjmB+Hn+dvF3AfgP4CDn3KHE9/8u4E/Oue865y52zk1vYEwtRwkq40IIAyGEX4cQvhhCmEvsQz8beG1RsVXEndi7nXMHjXBVxQnoCmLSGXz0AI8lr+2xcw0hrAwhLCe2KEpt22MFMdEtIh4VnwucApyTvLx/yf8+W7yaIeYN+Rmuof4WEBPYJ3g+2T5QJZ7B9ZebV+07VSnRD6V0UELxelzRvJH4LnGbZwEHhhDeEULoLSmzbe9/q7i+4cSxgNrqvtRI6nC4Buv3Hez5XXgxsa42hDgA6SzgDcTzZ38NPO6cO3sU4msJSlD580jyt/hIfRPwJmJX1o+dcwfXsJzTSqZPH1x2CGFdCGF50WOA2IL4E/Ap51zpOYpavYJ4VHpJCOHeEMJjJC2BUVRaf68Fbgwh3BlC+CPxvNUxdVxfxXquh6SF0pMsd6j1VrI5eY9XhQrnFstYShzxV+x1xOT0cNG840taOX+W/B3c/lrrvlwd9oQQyh0U1WIn0FalzFJiq/KYku/C4GMXxPOuIYTFIYRrQgivJbYk3z/CuKSEhplnmHPu58QTsL8jdikdS+x/7wfuKS4bQtjsnPtz4PvAT51zZ4YQ+oZY/IXOuUeTZb+b+KWveOI5hBCccxcQu4CWOOc+S/wSDxBPfr+a2PU0lGXEndjHnHO3EQdejPRkflU11t9jwLucc78i7rSuovrOazjOds5dROwKmgO8k9h6rKcvAv83eT8XE0fdDZ7vaYQvAPc75/6FeF5vBnE0420hhFVF5QJwa/JD2ynADcD3QwjLktdrrfuXJQMWvkFsdV9MPK80UiuBI5xzpxM/k9tDCNuLC4QQtjrnrgGuSXqD7ybuL19MPA/7SefcnwFvBH5ETK6ziN+Fr+5DbFJELahs+wHwLmK32GPEUWvLgFeV6YYhxN97nEUc1n2Pc67i+Rni75jmEfvR3wtcEEJYMlQwIYT7iUnl18SRUA8kj8uJo5nOq/L/DxFHZP0t8Uj744x8NFYtaqm/9xO/B4uBO4jn8oash2G6CjiT2AL938RBLLfXcfkQR459CbieOOT5NGLSerrO6wGeex/PIbaaHiQOmvg+8HclRRcTzzHeTUzQS9mzdVFr3f878bzU74jbeRNw3T5swh3Ez+v3iQcunyhXKIRwNXHU4AeJ2/mrZLozKbKJeGD3PeLn6hbiiNir9yE2KeJCaNRBlmSRc24G8QjyNSGEXw1dWvaFc25wBNx/pbDuW4CXhhBOHu11i9SLuvhEcs4510Ecdn8P8TzkXxBbxaWj8URyRQlKJP92EUebXQ0cCCwHPhRC+EqqUYnsI3XxiYhIJmmQhIiIZFKaXXxquomItK6qP7hO9RxUT09PmqsflkKhQG/vXiO7Mytv8UL+Yla8jaV4Gy+tmDs6Omoqpy4+ERHJJCUoERHJJCUoERHJJCUoERHJJCUoERHJJCUoERHJpKrDzL33txBv8LbOzE4s87ojXkX5LcRbJL/PzO6vd6AiItJaamlBLSDex6aSs4j3QZlFvH3DTfseloiItLqqLSgz+4X3fsYQReYCt5pZAH7jvZ/svT/SzNbUK8h9FXbtYvf8z8PGoe7fN7S+9nZ2DdR6w9H05S1eyF/MirexFG/jpRbzl75RU7F6XEliKrC6aLormbdXgvLezyO2sjAzCoVCHVZf3bOPL2XD/ffRPms2YyZMqv4PZTjnGJOjC+vmLV7IX8yKt7EUb+NlPeZ6JKhy11Mqu8VmNp94i2iAMFqX2Ni95N749+8+SZh08IiWkbfLmOQtXshfzIq3sRRv42U95nqM4usCphdNTwMydZG98PhSOGIqboTJSURERl89WlB3Ahd57xcCrwQ2Zer80+5dsOxh3CtenXYoIiIyDLUMM/8mcAZQ8N53AZ8B9gMwsy8Di4hDzJcTh5m/v1HBjkjXk7BjG8w6Ie1IRERkGGoZxXdeldcD8OG6RVRnYdlSANxxSlAiInnS9FeSCI8vhUMOw005NO1QRERkGJo6QYUQYNlSnLr3RERyp6kTFOvXwJZNcOzxaUciIiLD1NQJKjzxGADuBS9MORIRERmupk5QrHgUDhwLHUelHYmIiAxTUyeo8MSjMPM43Ji2tEMREZFhatoEFZ7eAV1P4l7worRDERGREWjaBEXnMgi7cccoQYmI5FHTJqjwxKPxyTEaICEikkfNm6BWPAZHTMONn5B2KCIiMgJNmaBCCLDiMQ0vFxHJsaZMUKxbA1s3g84/iYjkVlMmqMHzTxrBJyKSX02ZoFjxKIwdB0dOr15WREQyqSkTVHjiseQHuk25eSIiLaHp9uDh6e3Q/aR+/yQiknNNl6BYmfxAVyP4RERyrekSVFgRr2DOTCUoEZE8a74E1bkMjpiqH+iKiORc0yUoVi7DzZiVdhQiIrKPmipBhY19sGkDzDgu7VBERGQfNVWCYuXjALgZx6Ybh4iI7LOmSlChcxm0tcFRx6QdioiI7KPmS1BTZ+D22z/tUEREZB81TYIKu3dD53I0QEJEpDk0TYJi3RrYsQ1mKkGJiDSDpklQoXNwgIQSlIhIM2iaBEXncjjgQOjQFcxFRJpB0ySo0LkMjjoGN6Yt7VBERKQOmiJBhYEBWLUCde+JiDSPpkhQ9DwJz+6EmbqChIhIs2iKBBVWLgM0QEJEpJk0RYKicxlMmAiFw9OORERE6qS9lkLe+znA9UAbcLOZXVvy+lHA14DJSZlPmdmiOsdaUehcDkcfi3NutFYpIiINVrUF5b1vA24AzgJmA+d572eXFLscMDM7CTgXuLHegVYSnt0Ja1bhjtYFYkVEmkktXXynAsvNbIWZ7QQWAnNLygRgUvL8IKCnfiFW0f0k7NqFO+oFo7ZKERFpvFq6+KYCq4umu4BXlpS5EviR9/4jwHjgzLpEV4Ow6on4RFcwFxFpKrUkqHIndkLJ9HnAAjP7ovf+dODr3vsTzWx3cSHv/TxgHoCZUSgURhLzHjavW8PT4yZQeNEJDT0H1d7eXpd4R0ve4oX8xax4G0vxNl7WY64lQXUBxdcPmsbeXXgXAnMAzOw+7/2BQAFYV1zIzOYD85PJ0NvbO5KY97Dr8aUwfSZ9fX37vKyhFAoF6hHvaMlbvJC/mBVvYynexksr5o6OjprK1ZKglgCzvPczgW7iIIjzS8qsAt4ILPDeHw8cCKyvOdoRCgMDsHol7g1vbfSqRERklFUdJGFmA8BFwF3AI3GWLfXeX+W9Pycp9jHgb7z3DwLfBN5nZqXdgPW3tgsGnoXpOv8kItJsavodVPKbpkUl864oev4w8Kr6hlbd4AAJd7RG8ImINJt8X0li1QrY/wA4vLb+TBERyY9cJ6iw6gmYPlO32BARaUK5TVBh925YtRKn3z+JiDSl3CYo1q2BZ3aAriAhItKUcpugnhsgoQQlItKUcpugWLUC2tqhY3r1siIikju5TVBh1RMw9Whc+35phyIiIg2QywQVQoDVK3DTZ6YdioiINEguExSbNsLWLaAEJSLStPKZoLo6AXDTZqQahoiINE4uE1To7oxPph6dahwiItI4uUxQdHXC5ENwEyZVLSoiIvmUywQVujpB3XsiIk0tdwkqDDwLa7p0/klEpMnlLkGxtht2DagFJSLS5HKXoIJG8ImItITcJSi6OuMljg6fmnYkIiLSQLlLUKG7E46cjmuv6WbAIiKSU7lLUHR1qntPRKQF5CpBhS2boX+DBkiIiLSAXCUokitIqAUlItL8cpWgBkfwqQUlItL8cpWg6H4SJkyCSZPTjkRERBosVwkq9KyKNyl0Lu1QRESkwXKToEIIsGY17kjd4l1EpBXkJkHRvwF2bIcOJSgRkVaQnwTVswoA13FUyoGIiMhoyE2CCmtigkJdfCIiLSE3CYqe1TBhIkw8KO1IRERkFOQmQYWeVfEafBrBJyLSEnKRoJ4bwafzTyIiLSMXCYpNG2H7NjhSCUpEpFXkI0E9N4JPAyRERFpFLhJUWLM6PlEXn4hIy6jprn/e+znA9UAbcLOZXVumjAeuBALwoJmdX7coe1bDuAm6Bp+ISAup2oLy3rcBNwBnAbOB87z3s0vKzAIuA15lZicAl9QzyLBmFXRoBJ+ISCuppYvvVGC5ma0ws53AQmBuSZm/AW4ws40AZrauXgGGEKBHI/hERFpNLV18U4HVRdNdwCtLyhwH4L2/l9gNeKWZ/bB0Qd77ecA8ADOjUChUXfmu/g30btvChGNfxLgayjdKe3t7TfFmRd7ihfzFrHgbS/E2XtZjriVBletXC2WWMws4A5gG/NJ7f6KZ9RcXMrP5wPzBZfT29lZdeXjsTwBsm3gw22so3yiFQoFa4s2KvMUL+YtZ8TaW4m28tGLu6OioqVwtXXxdQPH47mlAT5ky3zOzZ81sJfAYMWHts/BUd3xyxNR6LE5ERHKilhbUEmCW934m0A2cC5SO0LsDOA9Y4L0vELv8VtQlwqe6oX0/mJLdZqiIiNRf1RaUmQ0AFwF3AY/EWbbUe3+V9/6cpNhdQJ/3/mHgHuAfzKyvHgGGp3rg8A7cmLZ6LE5ERHKipt9BmdkiYFHJvCuKngfg0uRRX2u7YerRdV+siIhkW6avJBEGBqB3Le7w2k6oiYhI88h0gqL3Kdi1SwMkRERaULYTVDKCzx2uBCUi0moynaA0xFxEpHVlOkGxthsmTMKNn5h2JCIiMsoynaAGh5iLiEjryXSC4qlunLr3RERaUmYTVNixPd7qXQMkRERaUmYTlEbwiYi0tswmqPBUcj1adfGJiLSkzCYonuoB5+DQI9KOREREUpDdBLV+LUw+BLff/mlHIiIiKchsggq9a9V6EhFpYZlNUKxfi1OCEhFpWZlMUOGZp+MQcyUoEZGWlckExfq18a8SlIhIy8pmguqNCcodemTKgYiISFoymaDCuqQFdZhaUCIirSqTCYr1a2HceF3FXESkhWUyQYX1a6Cg1pOISCvLZIJi/VMaYi4i0uIyl6DC7l3Qt07nn0REWlzmEhQbemHXAGgEn4hIS8tegkp+A+UKh6cciIiIpClzCSoM/kj3MLWgRERaWeYSFL1PQVsbHHxI2pGIiEiKspeg+tbH22yMaUs7EhERSVHmElTYsA4OOSztMEREJGWZS1D0rccdcmjaUYiISMoylaDCwAD0b1ALSkREspWg6O+DsBumqAUlItLqspWg+tYD4NSCEhFpeZlKUKFvXXyiBCUi0vIylaDYkCSoKYV04xARkdS111LIez8HuB5oA242s2srlHs78C3gFWb2u2FH07ceJk3G7bf/sP9VRESaS9UWlPe+DbgBOAuYDZznvZ9dptxE4KPAb0caTOjTb6BERCSqpYvvVGC5ma0ws53AQmBumXJXA58Hnh5xNH3rcRrBJyIi1NbFNxVYXTTdBbyyuID3/iRgupn9f+/9xystyHs/D5gHYGYUCs+fawohsG5jL2NPP4OJheydg2pvb98j3qzLW7yQv5gVb2Mp3sbLesy1JChXZl4YfOK9HwNcB7yv2oLMbD4wf3AZvb29zy9w80Z4dic7xk3gmaL5WVEoFOjNYFyV5C1eyF/MirexFG/jpRVzR0dHTeVq6eLrAqYXTU8DeoqmJwInAj/z3ncCpwF3eu9PqSmCQfoNlIiIFKmlBbUEmOW9nwl0A+cC5w++aGabgOfaiN77nwEfH/Yovo1JFj84u81NEREZPVVbUGY2AFwE3AU8EmfZUu/9Vd77c+oVSNi4IT7RfaBERIQafwdlZouARSXzrqhQ9owRRdLfB+3tMGHSiP5dRESaS3auJNHfBwdNwblyYzJERKTVZCZBhY196t4TEZHnZCZB0b8BN1kJSkREokwkqBBC7OJTghIRkUQmEhQ7tsMzT8PBU9KOREREMiIbCaq/L/5VC0pERBLZSFAbY4LSOSgRERmUiQQVBltQGsUnIiKJTCSowRYUk3UOSkREomwkqP4+mDBRd9IVEZHnZCJBhf4NGiAhIiJ7yESCon+DuvdERGQP2UhQW/pxEyenHYWIiGRI6gkqhACb+2GSEpSIiDwv9QTF9m0wMKAEJSIie0g/QW3uj3+VoEREpEhmEpRTghIRkSKpJ6igFpSIiJSReoJSF5+IiJSTjQTlxsCEiWlHIiIiGZJ+gtrSDxMn4ca0pR2JiIhkSOoJKug3UCIiUkbqCUo/0hURkXIykaA0xFxEREqlmqB0mSMREakk3RbUMzvg2Z1KUCIispd0E9SWzfHvhEmphiEiItmTboLatgUAN16/gRIRkT2lm6C2xgSlH+mKiEipdAdJJC0oxquLT0RE9qQWlIiIZFLK56CSQRLjJqQahoiIZE/6Lahx43Ftug6fiIjsqb2WQt77OcD1QBtws5ldW/L6pcAHgQFgPfABM3uy6oK3bQGN4BMRkTKqtqC8923ADcBZwGzgPO/97JJifwBOMbOXALcDn69l5WHrFv0GSkREyqqlBXUqsNzMVgB47xcCc4GHBwuY2T1F5X8DvLumtW/bAhMPqjlYERFpHbWcg5oKrC6a7krmVXIh8IOa1r51M04j+EREpIxaWlCuzLxQrqD3/t3AKcDrKrw+D5gHYGa47VsZWziMiYVCjeGmp729nUIO4hyUt3ghfzEr3sZSvI2X9ZhrSVBdwPSi6WlAT2kh7/2ZwKeB15nZM+UWZGbzgfnJZAg7trOjrZ1nenuHF3UKCoUCvTmIc1De4oX8xax4G0vxNl5aMXd0dNRUrpYEtQSY5b2fCXQD5wLnFxfw3p8E/Acwx8zWDStSjeITEZEyqp6DMrMB4CLgLuCROMuWeu+v8t6fkxT7AjAB+Jb3/gHv/Z01RzB2/PCjFhGRplfT76DMbBGwqGTeFUXPzxxpAE4JSkREykj/lu9jx6UdgYiIZFD6CWqcEpSIiOwt/QSlLj4RESkj/QR1oFpQIiKyt/QT1NixaUcgIiIZlG6COmAsboxutSEiIntLN0FpBJ+IiFSgBCUiIpmUboIapxF8IiJSnlpQIiKSSakmKKch5iIiUkG6LagDNcRcRETKSzdB7X9AqqsXEZHsSvl3UEpQIiJSnlpQIiKSSSknqANTXb2IiGSXWlAiIpJJKZ+DUgtKRETKS/d3UGpBiYhIBRrFJyIimaRzUCIikkk6ByUiIpmkFpSIiGRSuglqv/1TXb2IiGSXEpSIiGRSugmqvT3V1YuISHalm6DalKBERKQ8taBERCST1IISEZFMSvdSR86luXoREcmwdFtQIiIiFShBiYhIJilBiYhIJilBiYhIJtU0jM57Pwe4HmgDbjaza0tePwC4FTgZ6APeaWad9Q1VRERaSdUWlPe+DbgBOAuYDZznvZ9dUuxCYKOZHQtcB3yu3oGKiEhrqaWL71RguZmtMLOdwEJgbkmZucDXkue3A2/03msMuYiIjFgtXXxTgdVF013AKyuVMbMB7/0m4BCgt7iQ934eMC8pR6FQGGHYo6+9vV3xNljeYla8jaV4Gy/rMdeSoMq1hMIIymBm84H5g6/39vaWFsmsQqGA4m2svMWseBtL8TZeWjF3dHTUVK6WLr4uYHrR9DSgp1IZ7307cBCwoaYIREREyqilBbUEmOW9nwl0A+cC55eUuRO4ALgPeDvwUzPbqwUlIiJSq6otKDMbAC4C7gIeibNsqff+Ku/9OUmxrwKHeO+XA5cCn2pUwCIi0hpcCKk1dEJPT2lPYXblrX85b/FC/mJWvI2leBsv5XNQVUd660oSIiKSSUpQIiKSSUpQIiKSSameg0prxSIikrrsnoPy3v+eGGAuHopXMStexdtM8WYg5qrUxSciIpmkBCUiIpmUZoKaX71IpijexstbzIq3sRRv42U65jQHSYiIiFSkLj4REckkJSgREcmkWq5mXnfe+znA9UAbcLOZXZtCDNOBW4EjgN3AfDO73ns/BfhvYAbQCXgz25jcIfh64C3AduB9ZnZ/sqwLgMuTRX/WzL7WwLjbgN8B3WZ2dnKV+YXAFOB+4D1mttN7f0CyfScDfcA7zawzWcZlwIXALuCjZnZXA+OdDNwMnEj87dsHgMfIaB177/8e+GAS6x+B9wNHkpE69t7fApwNrDOzE5N5dfvMeu9PBhYAY4FFwMX7emeCCjF/AfgLYCfwBPB+M+tPXitbd5X2G5W+A/WMt+i1jwNfAA41s94s1HGleL33HyFe6HsA+L6ZfSKZn2r9Dseot6CSHewNwFnAbOA87/3s0Y6D+KZ9zMyOB04DPpzE8SngJ2Y2C/gJz1+Z/SxgVvKYB9wEz+0cPkO8y/CpwGe89wc3MO6LiVeVH/Q54Lok3o3EDx7J341mdixwXVKOZBvPBU4A5gA3Ju9Jo1wP/NDMXgS8NIk9k3XsvZ8KfBQ4JfmitxHrKkt1vCBZZrF61udNSdnB/ytdV71ivhs40cxeAjwOXJbEVrbuquw3Kr0/9Yx38KD2TcCqotlZqOO94vXevx6YC7zEzE4A/jmZn4X6rVkaXXynAsvNbEWShRcSK3JUmdmawSMdM9tC3HFOTWIZPDr/GvC25Plc4FYzC2b2G2Cy9/5I4M+Bu81sg5ltJH7x6vGl3ov3fhrwVmKLhOTo7Q3A7RXiHdyO24E3JuXnAgvN7BkzWwksJ74njYh3EvBa4u1YMLOdyVFyZuuY2KswNrnx5jhgDRmqYzP7BXvfDLQu9Zm8NsnM7kuO6G8tWlZdYzazHyW38gH4DfFGqIMxl6u7svuNKt+BusWbuA74BHteBSf1Oq4Q74eAa83smaTMuqJ4U63f4UgjQU0FVhdNdyXzUuO9nwGcBPwWONzM1kBMYsBhSbFKcY/m9vwr8QuyO5k+BOgv+qIXr/u5uJLXNyXlRzPeY4D1wH967//gvb/Zez+ejNaxmXUTjzRXERPTJuD3ZLuOoX71OTV5Xjq/0T4A/CB5PtyYh/oO1E1y77tuM3uw5KWs1vFxwGu897/13v/ce/+KEcY7KvVbSRoJqtwlLlJj5sZGAAAC/UlEQVQb6+69nwB8G7jEzDYPUbRS3KOyPd77wT7m39cQ01CvjWb9twMvB24ys5OAbQx9M8u06/hg4hHmTKADGE/s8qi07izU8VCGG9+ox+29/zSxu/22ZFbmYvbejwM+DVxR5uXMxZtoBw4mnr74B8CS1lBW4y0rjQTVBUwvmp4GpHLnQu/9fsTkdJuZfSeZ/VTSDCf5O9g0rhT3aG3Pq4BzvPedxOb3G4gtqslJd1Tpup+LK3n9IGI3wGjWfxfQZWa/TaZvJyasrNbxmcBKM1tvZs8C3wH+jGzXMdSvPrt4vquteH5DJIMIzgbeVTRIYLgx91L5/amXFxAPWh5Mvn/TgPu990eMIN7RquMu4DtJ1+NiYq9LYQTxjkb9VpRGgloCzPLez/Te7088YXfnaAeRHE18FXjEzP6l6KU7gQuS5xcA3yua/17vvfPenwZsSrpT7gLe7L0/ODkCf3Myr67M7DIzm2ZmM4h19lMzexdwD/D2CvEObsfbk/IhmX+u9/6AZHTOLGBxveNNYl4LrPbevzCZ9UbgYTJax8SuvdO89+OSz8dgvJmt4zJxjLg+k9e2eO9PS7b/vUXLqqtkxNgngXPMbHvJtpSru7L7jaS+K70/dWFmfzSzw8xsRvL96wJenny+s1rHdxAPYvHeHwfsT0w2mavfoYx6gkr6Mi8ivoGPxFm2dLTjILZI3gO8wXv/QPJ4C3At8Cbv/TLiiJ3BIfCLgBXEk4pfAf4XgJltAK4mvsFLgKuSeaPlk8Cl3vvlxP7irybzvwocksy/lKRrLalrI+54fwh82Mx2NTC+jwC3ee8fAl4GXENG6zhp6d1OHEr7R+L3Yz4ZqmPv/TeB+4AXeu+7vPcXUt/6/BBxEM5y4vDvwXND9Y75S8BE4O7ku/flJLaydVdlv1Hp/alnvJWkXscV4r0FOMZ7/ydij8sFSWsq9fodDl3qSEREMklXkhARkUxSghIRkUxSghIRkUxSghIRkUxSghIRkUxSghIRkUxSghIRkUz6H3ak05Y7FQgqAAAAAElFTkSuQmCC\n",
1450 "text/plain": [
1451 "<Figure size 432x288 with 1 Axes>"
1452 ]
1453 },
1454 "metadata": {
1455 "needs_background": "light"
1456 },
1457 "output_type": "display_data"
1458 }
1459 ],
1460 "source": [
1461 "pd.Series(sampling_table).plot(title='Skip-Gram Sampling Probabilities')\n",
1462 "plt.tight_layout();"
1463 ]
1464 },
1465 {
1466 "cell_type": "markdown",
1467 "metadata": {
1468 "slideshow": {
1469 "slide_type": "slide"
1470 }
1471 },
1472 "source": [
1473 "### Generate target-context word pairs"
1474 ]
1475 },
1476 {
1477 "cell_type": "code",
1478 "execution_count": 29,
1479 "metadata": {
1480 "ExecuteTime": {
1481 "end_time": "2018-12-10T07:00:33.846407Z",
1482 "start_time": "2018-12-10T06:58:10.973923Z"
1483 },
1484 "slideshow": {
1485 "slide_type": "fragment"
1486 }
1487 },
1488 "outputs": [
1489 {
1490 "name": "stdout",
1491 "output_type": "stream",
1492 "text": [
1493 "17,324,656 pairs created\n"
1494 ]
1495 }
1496 ],
1497 "source": [
1498 "pairs, labels = skipgrams(sequence=data,\n",
1499 " vocabulary_size=vocab_size,\n",
1500 " window_size=WINDOW_SIZE,\n",
1501 " sampling_table=sampling_table,\n",
1502 " negative_samples=1.0,\n",
1503 " shuffle=True)\n",
1504 "\n",
1505 "print('{:,d} pairs created'.format(len(pairs)))"
1506 ]
1507 },
1508 {
1509 "cell_type": "code",
1510 "execution_count": 30,
1511 "metadata": {
1512 "ExecuteTime": {
1513 "end_time": "2018-12-10T07:00:33.851520Z",
1514 "start_time": "2018-12-10T07:00:33.848200Z"
1515 },
1516 "slideshow": {
1517 "slide_type": "fragment"
1518 }
1519 },
1520 "outputs": [
1521 {
1522 "data": {
1523 "text/plain": [
1524 "[[1636, 654], [12035, 9246], [217, 6031], [1522, 7], [3917, 1024]]"
1525 ]
1526 },
1527 "execution_count": 30,
1528 "metadata": {},
1529 "output_type": "execute_result"
1530 }
1531 ],
1532 "source": [
1533 "pairs[:5]"
1534 ]
1535 },
1536 {
1537 "cell_type": "code",
1538 "execution_count": 31,
1539 "metadata": {
1540 "ExecuteTime": {
1541 "end_time": "2018-12-10T07:01:53.633534Z",
1542 "start_time": "2018-12-10T07:00:33.852713Z"
1543 },
1544 "slideshow": {
1545 "slide_type": "slide"
1546 }
1547 },
1548 "outputs": [],
1549 "source": [
1550 "target_word, context_word = np.array(pairs, dtype=np.int32).T\n",
1551 "labels = np.array(labels, dtype=np.int8)\n",
1552 "del pairs"
1553 ]
1554 },
1555 {
1556 "cell_type": "code",
1557 "execution_count": 32,
1558 "metadata": {
1559 "ExecuteTime": {
1560 "end_time": "2018-12-10T07:01:53.650930Z",
1561 "start_time": "2018-12-10T07:01:53.635164Z"
1562 },
1563 "slideshow": {
1564 "slide_type": "fragment"
1565 }
1566 },
1567 "outputs": [
1568 {
1569 "data": {
1570 "text/plain": [
1571 "array([ 1636, 12035, 217, 1522, 3917], dtype=int32)"
1572 ]
1573 },
1574 "execution_count": 32,
1575 "metadata": {},
1576 "output_type": "execute_result"
1577 }
1578 ],
1579 "source": [
1580 "target_word[:5]"
1581 ]
1582 },
1583 {
1584 "cell_type": "code",
1585 "execution_count": 33,
1586 "metadata": {
1587 "ExecuteTime": {
1588 "end_time": "2018-12-10T07:01:53.661703Z",
1589 "start_time": "2018-12-10T07:01:53.652325Z"
1590 },
1591 "slideshow": {
1592 "slide_type": "fragment"
1593 }
1594 },
1595 "outputs": [
1596 {
1597 "data": {
1598 "text/html": [
1599 "<div>\n",
1600 "<style scoped>\n",
1601 " .dataframe tbody tr th:only-of-type {\n",
1602 " vertical-align: middle;\n",
1603 " }\n",
1604 "\n",
1605 " .dataframe tbody tr th {\n",
1606 " vertical-align: top;\n",
1607 " }\n",
1608 "\n",
1609 " .dataframe thead th {\n",
1610 " text-align: right;\n",
1611 " }\n",
1612 "</style>\n",
1613 "<table border=\"1\" class=\"dataframe\">\n",
1614 " <thead>\n",
1615 " <tr style=\"text-align: right;\">\n",
1616 " <th></th>\n",
1617 " <th>target</th>\n",
1618 " <th>context</th>\n",
1619 " <th>label</th>\n",
1620 " </tr>\n",
1621 " </thead>\n",
1622 " <tbody>\n",
1623 " <tr>\n",
1624 " <th>0</th>\n",
1625 " <td>1636</td>\n",
1626 " <td>654</td>\n",
1627 " <td>0</td>\n",
1628 " </tr>\n",
1629 " <tr>\n",
1630 " <th>1</th>\n",
1631 " <td>12035</td>\n",
1632 " <td>9246</td>\n",
1633 " <td>0</td>\n",
1634 " </tr>\n",
1635 " <tr>\n",
1636 " <th>2</th>\n",
1637 " <td>217</td>\n",
1638 " <td>6031</td>\n",
1639 " <td>0</td>\n",
1640 " </tr>\n",
1641 " <tr>\n",
1642 " <th>3</th>\n",
1643 " <td>1522</td>\n",
1644 " <td>7</td>\n",
1645 " <td>1</td>\n",
1646 " </tr>\n",
1647 " <tr>\n",
1648 " <th>4</th>\n",
1649 " <td>3917</td>\n",
1650 " <td>1024</td>\n",
1651 " <td>1</td>\n",
1652 " </tr>\n",
1653 " </tbody>\n",
1654 "</table>\n",
1655 "</div>"
1656 ],
1657 "text/plain": [
1658 " target context label\n",
1659 "0 1636 654 0\n",
1660 "1 12035 9246 0\n",
1661 "2 217 6031 0\n",
1662 "3 1522 7 1\n",
1663 "4 3917 1024 1"
1664 ]
1665 },
1666 "execution_count": 33,
1667 "metadata": {},
1668 "output_type": "execute_result"
1669 }
1670 ],
1671 "source": [
1672 "df = pd.DataFrame({'target': target_word[:5], 'context': context_word[:5], 'label': labels[:5]})\n",
1673 "df"
1674 ]
1675 },
1676 {
1677 "cell_type": "code",
1678 "execution_count": 34,
1679 "metadata": {
1680 "ExecuteTime": {
1681 "end_time": "2018-12-10T07:01:54.026465Z",
1682 "start_time": "2018-12-10T07:01:53.662752Z"
1683 },
1684 "slideshow": {
1685 "slide_type": "slide"
1686 }
1687 },
1688 "outputs": [
1689 {
1690 "data": {
1691 "text/plain": [
1692 "1 8662328\n",
1693 "0 8662328\n",
1694 "dtype: int64"
1695 ]
1696 },
1697 "execution_count": 34,
1698 "metadata": {},
1699 "output_type": "execute_result"
1700 }
1701 ],
1702 "source": [
1703 "pd.Series(labels).value_counts()"
1704 ]
1705 },
1706 {
1707 "cell_type": "code",
1708 "execution_count": 35,
1709 "metadata": {
1710 "ExecuteTime": {
1711 "end_time": "2018-12-10T07:05:58.136170Z",
1712 "start_time": "2018-12-10T07:05:58.102377Z"
1713 },
1714 "hide_input": false,
1715 "slideshow": {
1716 "slide_type": "fragment"
1717 }
1718 },
1719 "outputs": [],
1720 "source": [
1721 "with pd.HDFStore(path / 'data.h5') as store:\n",
1722 " store.put('id_to_token', pd.Series(id_to_token))"
1723 ]
1724 },
1725 {
1726 "cell_type": "markdown",
1727 "metadata": {
1728 "slideshow": {
1729 "slide_type": "slide"
1730 }
1731 },
1732 "source": [
1733 "### Define Keras Model Components"
1734 ]
1735 },
1736 {
1737 "cell_type": "markdown",
1738 "metadata": {
1739 "slideshow": {
1740 "slide_type": "fragment"
1741 }
1742 },
1743 "source": [
1744 "#### Scalar Input Variables"
1745 ]
1746 },
1747 {
1748 "cell_type": "code",
1749 "execution_count": 36,
1750 "metadata": {
1751 "ExecuteTime": {
1752 "end_time": "2018-12-10T07:06:01.438878Z",
1753 "start_time": "2018-12-10T07:06:01.428723Z"
1754 },
1755 "slideshow": {
1756 "slide_type": "fragment"
1757 }
1758 },
1759 "outputs": [],
1760 "source": [
1761 "input_target = Input((1,), name='target_input')\n",
1762 "input_context = Input((1,), name='context_input')"
1763 ]
1764 },
1765 {
1766 "cell_type": "markdown",
1767 "metadata": {
1768 "slideshow": {
1769 "slide_type": "slide"
1770 }
1771 },
1772 "source": [
1773 "#### Shared Embedding Layer"
1774 ]
1775 },
1776 {
1777 "cell_type": "code",
1778 "execution_count": 37,
1779 "metadata": {
1780 "ExecuteTime": {
1781 "end_time": "2018-12-10T07:06:04.673449Z",
1782 "start_time": "2018-12-10T07:06:04.671203Z"
1783 },
1784 "slideshow": {
1785 "slide_type": "fragment"
1786 }
1787 },
1788 "outputs": [],
1789 "source": [
1790 "embedding = Embedding(input_dim=vocab_size,\n",
1791 " output_dim=EMBEDDING_SIZE,\n",
1792 " input_length=1,\n",
1793 " name='embedding_layer')"
1794 ]
1795 },
1796 {
1797 "cell_type": "code",
1798 "execution_count": 38,
1799 "metadata": {
1800 "ExecuteTime": {
1801 "end_time": "2018-12-10T07:06:05.431704Z",
1802 "start_time": "2018-12-10T07:06:05.387080Z"
1803 },
1804 "slideshow": {
1805 "slide_type": "fragment"
1806 }
1807 },
1808 "outputs": [
1809 {
1810 "name": "stdout",
1811 "output_type": "stream",
1812 "text": [
1813 "WARNING:tensorflow:From /home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
1814 "Instructions for updating:\n",
1815 "Colocations handled automatically by placer.\n"
1816 ]
1817 }
1818 ],
1819 "source": [
1820 "target = embedding(input_target)\n",
1821 "target = Reshape((EMBEDDING_SIZE, 1), name='target_embedding')(target)\n",
1822 "\n",
1823 "context = embedding(input_context)\n",
1824 "context = Reshape((EMBEDDING_SIZE, 1), name='context_embedding')(context)"
1825 ]
1826 },
1827 {
1828 "cell_type": "markdown",
1829 "metadata": {
1830 "slideshow": {
1831 "slide_type": "slide"
1832 }
1833 },
1834 "source": [
1835 "#### Create Similarity Measure"
1836 ]
1837 },
1838 {
1839 "cell_type": "code",
1840 "execution_count": 39,
1841 "metadata": {
1842 "ExecuteTime": {
1843 "end_time": "2018-12-10T07:06:06.339306Z",
1844 "start_time": "2018-12-10T07:06:06.314257Z"
1845 },
1846 "slideshow": {
1847 "slide_type": "fragment"
1848 }
1849 },
1850 "outputs": [],
1851 "source": [
1852 "dot_product = Dot(axes=1)([target, context])\n",
1853 "dot_product = Reshape((1,), name='similarity')(dot_product)"
1854 ]
1855 },
1856 {
1857 "cell_type": "markdown",
1858 "metadata": {
1859 "slideshow": {
1860 "slide_type": "slide"
1861 }
1862 },
1863 "source": [
1864 "#### Sigmoid Output Layer"
1865 ]
1866 },
1867 {
1868 "cell_type": "code",
1869 "execution_count": 40,
1870 "metadata": {
1871 "ExecuteTime": {
1872 "end_time": "2018-12-10T07:06:08.599771Z",
1873 "start_time": "2018-12-10T07:06:08.584301Z"
1874 },
1875 "slideshow": {
1876 "slide_type": "fragment"
1877 }
1878 },
1879 "outputs": [],
1880 "source": [
1881 "output = Dense(units=1, activation='sigmoid', name='output')(dot_product)"
1882 ]
1883 },
1884 {
1885 "cell_type": "markdown",
1886 "metadata": {
1887 "slideshow": {
1888 "slide_type": "slide"
1889 }
1890 },
1891 "source": [
1892 "#### Compile Training Model"
1893 ]
1894 },
1895 {
1896 "cell_type": "code",
1897 "execution_count": 41,
1898 "metadata": {
1899 "ExecuteTime": {
1900 "end_time": "2018-12-10T07:06:09.195879Z",
1901 "start_time": "2018-12-10T07:06:09.117732Z"
1902 },
1903 "slideshow": {
1904 "slide_type": "fragment"
1905 }
1906 },
1907 "outputs": [],
1908 "source": [
1909 "model = Model(inputs=[input_target, input_context], outputs=output)\n",
1910 "model.compile(loss='binary_crossentropy', optimizer='rmsprop')"
1911 ]
1912 },
1913 {
1914 "cell_type": "markdown",
1915 "metadata": {},
1916 "source": [
1917 "#### Display Architecture"
1918 ]
1919 },
1920 {
1921 "cell_type": "code",
1922 "execution_count": 42,
1923 "metadata": {
1924 "ExecuteTime": {
1925 "end_time": "2018-12-10T07:06:11.692010Z",
1926 "start_time": "2018-12-10T07:06:11.689022Z"
1927 },
1928 "slideshow": {
1929 "slide_type": "slide"
1930 }
1931 },
1932 "outputs": [
1933 {
1934 "name": "stdout",
1935 "output_type": "stream",
1936 "text": [
1937 "__________________________________________________________________________________________________\n",
1938 "Layer (type) Output Shape Param # Connected to \n",
1939 "==================================================================================================\n",
1940 "target_input (InputLayer) (None, 1) 0 \n",
1941 "__________________________________________________________________________________________________\n",
1942 "context_input (InputLayer) (None, 1) 0 \n",
1943 "__________________________________________________________________________________________________\n",
1944 "embedding_layer (Embedding) (None, 1, 200) 3476200 target_input[0][0] \n",
1945 " context_input[0][0] \n",
1946 "__________________________________________________________________________________________________\n",
1947 "target_embedding (Reshape) (None, 200, 1) 0 embedding_layer[0][0] \n",
1948 "__________________________________________________________________________________________________\n",
1949 "context_embedding (Reshape) (None, 200, 1) 0 embedding_layer[1][0] \n",
1950 "__________________________________________________________________________________________________\n",
1951 "dot (Dot) (None, 1, 1) 0 target_embedding[0][0] \n",
1952 " context_embedding[0][0] \n",
1953 "__________________________________________________________________________________________________\n",
1954 "similarity (Reshape) (None, 1) 0 dot[0][0] \n",
1955 "__________________________________________________________________________________________________\n",
1956 "output (Dense) (None, 1) 2 similarity[0][0] \n",
1957 "==================================================================================================\n",
1958 "Total params: 3,476,202\n",
1959 "Trainable params: 3,476,202\n",
1960 "Non-trainable params: 0\n",
1961 "__________________________________________________________________________________________________\n"
1962 ]
1963 }
1964 ],
1965 "source": [
1966 "model.summary()"
1967 ]
1968 },
1969 {
1970 "cell_type": "markdown",
1971 "metadata": {
1972 "slideshow": {
1973 "slide_type": "slide"
1974 }
1975 },
1976 "source": [
1977 "#### Validation Model"
1978 ]
1979 },
1980 {
1981 "cell_type": "code",
1982 "execution_count": 43,
1983 "metadata": {
1984 "ExecuteTime": {
1985 "end_time": "2018-12-10T07:13:01.364246Z",
1986 "start_time": "2018-12-10T07:13:01.330374Z"
1987 },
1988 "slideshow": {
1989 "slide_type": "fragment"
1990 }
1991 },
1992 "outputs": [],
1993 "source": [
1994 "similarity = Dot(normalize=True, \n",
1995 " axes=1, \n",
1996 " name='cosine_similarity')([target, context])"
1997 ]
1998 },
1999 {
2000 "cell_type": "code",
2001 "execution_count": 44,
2002 "metadata": {
2003 "ExecuteTime": {
2004 "end_time": "2018-12-10T07:13:01.715914Z",
2005 "start_time": "2018-12-10T07:13:01.711419Z"
2006 },
2007 "slideshow": {
2008 "slide_type": "fragment"
2009 }
2010 },
2011 "outputs": [],
2012 "source": [
2013 "# create a secondary validation model to run our similarity checks during training\n",
2014 "validation_model = Model(inputs=[input_target, input_context], outputs=similarity)"
2015 ]
2016 },
2017 {
2018 "cell_type": "code",
2019 "execution_count": 45,
2020 "metadata": {
2021 "ExecuteTime": {
2022 "end_time": "2018-12-10T07:13:02.480815Z",
2023 "start_time": "2018-12-10T07:13:02.470139Z"
2024 },
2025 "slideshow": {
2026 "slide_type": "slide"
2027 }
2028 },
2029 "outputs": [
2030 {
2031 "name": "stdout",
2032 "output_type": "stream",
2033 "text": [
2034 "__________________________________________________________________________________________________\n",
2035 "Layer (type) Output Shape Param # Connected to \n",
2036 "==================================================================================================\n",
2037 "target_input (InputLayer) (None, 1) 0 \n",
2038 "__________________________________________________________________________________________________\n",
2039 "context_input (InputLayer) (None, 1) 0 \n",
2040 "__________________________________________________________________________________________________\n",
2041 "embedding_layer (Embedding) (None, 1, 200) 3476200 target_input[0][0] \n",
2042 " context_input[0][0] \n",
2043 "__________________________________________________________________________________________________\n",
2044 "target_embedding (Reshape) (None, 200, 1) 0 embedding_layer[0][0] \n",
2045 "__________________________________________________________________________________________________\n",
2046 "context_embedding (Reshape) (None, 200, 1) 0 embedding_layer[1][0] \n",
2047 "__________________________________________________________________________________________________\n",
2048 "cosine_similarity (Dot) (None, 1, 1) 0 target_embedding[0][0] \n",
2049 " context_embedding[0][0] \n",
2050 "==================================================================================================\n",
2051 "Total params: 3,476,200\n",
2052 "Trainable params: 3,476,200\n",
2053 "Non-trainable params: 0\n",
2054 "__________________________________________________________________________________________________\n"
2055 ]
2056 }
2057 ],
2058 "source": [
2059 "validation_model.summary()"
2060 ]
2061 },
2062 {
2063 "cell_type": "markdown",
2064 "metadata": {
2065 "slideshow": {
2066 "slide_type": "slide"
2067 }
2068 },
2069 "source": [
2070 ""
2071 ]
2072 },
2073 {
2074 "cell_type": "markdown",
2075 "metadata": {
2076 "slideshow": {
2077 "slide_type": "slide"
2078 }
2079 },
2080 "source": [
2081 "### Create Keras Callbacks"
2082 ]
2083 },
2084 {
2085 "cell_type": "markdown",
2086 "metadata": {
2087 "slideshow": {
2088 "slide_type": "fragment"
2089 }
2090 },
2091 "source": [
2092 "#### Nearest Neighors & Analogies"
2093 ]
2094 },
2095 {
2096 "cell_type": "code",
2097 "execution_count": 46,
2098 "metadata": {
2099 "ExecuteTime": {
2100 "end_time": "2018-12-10T07:13:07.729779Z",
2101 "start_time": "2018-12-10T07:13:07.726679Z"
2102 },
2103 "slideshow": {
2104 "slide_type": "fragment"
2105 }
2106 },
2107 "outputs": [],
2108 "source": [
2109 "test_set = analogies_id.dropna().astype(int)\n",
2110 "a, b, c, actual = test_set.values.T\n",
2111 "actual = actual.reshape(-1, 1)\n",
2112 "n_analogies = len(actual)"
2113 ]
2114 },
2115 {
2116 "cell_type": "code",
2117 "execution_count": 47,
2118 "metadata": {
2119 "ExecuteTime": {
2120 "end_time": "2018-12-10T07:13:10.070985Z",
2121 "start_time": "2018-12-10T07:13:10.064607Z"
2122 },
2123 "slideshow": {
2124 "slide_type": "slide"
2125 }
2126 },
2127 "outputs": [],
2128 "source": [
2129 "class EvalCallback(Callback):\n",
2130 " \n",
2131 " def on_train_begin(self, logs={}):\n",
2132 " self.eval_nn()\n",
2133 " self.test_analogies()\n",
2134 "\n",
2135 " def on_train_end(self, logs={}):\n",
2136 " self.eval_nn()\n",
2137 "\n",
2138 " def on_epoch_end(self, batch, logs={}):\n",
2139 " self.test_analogies()\n",
2140 "\n",
2141 " @staticmethod\n",
2142 " def test_analogies():\n",
2143 " print('\\nAnalogy Accuracy:\\n\\t', end='')\n",
2144 " embeddings = embedding.get_weights()[0]\n",
2145 " target = embeddings[c] + embeddings[b] - embeddings[a]\n",
2146 " neighbors = np.argsort(cdist(target, embeddings, metric='cosine'))\n",
2147 " match_id = np.argwhere(neighbors == actual)[:, 1]\n",
2148 " print('\\n\\t'.join(['Top {}: {:.2%}'.format(i, (match_id < i).sum() / n_analogies) for i in [1, 5, 10]]))\n",
2149 "\n",
2150 " def eval_nn(self):\n",
2151 " print('\\n{} Nearest Neighbors:'.format(NN))\n",
2152 " for i in range(VALID_SET):\n",
2153 " valid_id = valid_examples[i]\n",
2154 " valid_word = id_to_token[valid_id]\n",
2155 " similarity = self._get_similiarity(valid_id).reshape(-1)\n",
2156 " nearest = (-similarity).argsort()[1:NN + 1]\n",
2157 " neighbors = [id_to_token[nearest[n]] for n in range(NN)]\n",
2158 " print('{}:\\t{}'.format(valid_word, ', '.join(neighbors))) \n",
2159 " \n",
2160 " @staticmethod\n",
2161 " def _get_similiarity(valid_word_idx):\n",
2162 " target = np.full(shape=vocab_size, fill_value=valid_word_idx)\n",
2163 " context = np.arange(vocab_size)\n",
2164 " return validation_model.predict([target, context])\n",
2165 "\n",
2166 "\n",
2167 "evaluation = EvalCallback()"
2168 ]
2169 },
2170 {
2171 "cell_type": "markdown",
2172 "metadata": {
2173 "slideshow": {
2174 "slide_type": "slide"
2175 }
2176 },
2177 "source": [
2178 "#### Tensorboard Callback"
2179 ]
2180 },
2181 {
2182 "cell_type": "code",
2183 "execution_count": 48,
2184 "metadata": {
2185 "ExecuteTime": {
2186 "end_time": "2018-12-10T07:13:13.272624Z",
2187 "start_time": "2018-12-10T07:13:13.270670Z"
2188 },
2189 "slideshow": {
2190 "slide_type": "fragment"
2191 }
2192 },
2193 "outputs": [],
2194 "source": [
2195 "tensorboard = TensorBoard(log_dir=str(tb_path),\n",
2196 " write_graph=True,\n",
2197 " embeddings_freq=1,\n",
2198 " embeddings_metadata=str(tb_path / 'meta.tsv'))"
2199 ]
2200 },
2201 {
2202 "cell_type": "markdown",
2203 "metadata": {
2204 "slideshow": {
2205 "slide_type": "slide"
2206 }
2207 },
2208 "source": [
2209 "### Train Model"
2210 ]
2211 },
2212 {
2213 "cell_type": "code",
2214 "execution_count": null,
2215 "metadata": {
2216 "ExecuteTime": {
2217 "end_time": "2018-12-10T07:29:39.905357Z",
2218 "start_time": "2018-12-10T07:13:13.616616Z"
2219 },
2220 "scrolled": false,
2221 "slideshow": {
2222 "slide_type": "fragment"
2223 }
2224 },
2225 "outputs": [],
2226 "source": [
2227 "loss = model.fit(x=[target_word, context_word],\n",
2228 " y=labels,\n",
2229 " shuffle=True,\n",
2230 " batch_size=BATCH_SIZE,\n",
2231 " epochs=EPOCHS,\n",
2232 " callbacks=[evaluation, tensorboard])\n",
2233 "\n",
2234 "model.save(str(path / 'skipgram_model.h5'))"
2235 ]
2236 },
2237 {
2238 "cell_type": "markdown",
2239 "metadata": {
2240 "slideshow": {
2241 "slide_type": "slide"
2242 }
2243 },
2244 "source": [
2245 "## Optimized TensorFlow Model\n",
2246 "\n",
2247 "Compile custom ops using `compile-ops.sh`.\n",
2248 "\n",
2249 "Run from command line."
2250 ]
2251 },
2252 {
2253 "cell_type": "code",
2254 "execution_count": 46,
2255 "metadata": {},
2256 "outputs": [
2257 {
2258 "name": "stdout",
2259 "output_type": "stream",
2260 "text": [
2261 "\u001b[31mcompile-ops.sh\u001b[m\u001b[m \u001b[31mword2vec.py\u001b[m\u001b[m word2vec_ops.so.zip\r\n",
2262 "\u001b[31mrun_tf.sh\u001b[m\u001b[m \u001b[31mword2vec_ops.so\u001b[m\u001b[m\r\n"
2263 ]
2264 }
2265 ],
2266 "source": [
2267 "!ls tensorflow/"
2268 ]
2269 },
2270 {
2271 "cell_type": "code",
2272 "execution_count": 49,
2273 "metadata": {
2274 "slideshow": {
2275 "slide_type": "fragment"
2276 }
2277 },
2278 "outputs": [],
2279 "source": [
2280 "# %%bash\n",
2281 "# python tensorflow/word2vec.py --language=en --source=Ted --file=ngrams_1 --embedding_size=300 --num_neg_samples=20 --starter_lr=.1 --target_lr=.05 --batch_size=10 --min_count=10 --window_size=10"
2282 ]
2283 },
2284 {
2285 "cell_type": "markdown",
2286 "metadata": {
2287 "slideshow": {
2288 "slide_type": "slide"
2289 }
2290 },
2291 "source": [
2292 "## word2vec using Gensim"
2293 ]
2294 },
2295 {
2296 "cell_type": "markdown",
2297 "metadata": {
2298 "slideshow": {
2299 "slide_type": "fragment"
2300 }
2301 },
2302 "source": [
2303 "### Evaluation"
2304 ]
2305 },
2306 {
2307 "cell_type": "code",
2308 "execution_count": 50,
2309 "metadata": {
2310 "ExecuteTime": {
2311 "end_time": "2018-12-07T00:33:47.261154Z",
2312 "start_time": "2018-12-07T00:33:47.256178Z"
2313 },
2314 "slideshow": {
2315 "slide_type": "fragment"
2316 }
2317 },
2318 "outputs": [],
2319 "source": [
2320 "def accuracy_by_category(acc, detail=True):\n",
2321 " results = [[c['section'], len(c['correct']), len(c['incorrect'])] for c in acc]\n",
2322 " results = pd.DataFrame(results, columns=['category', 'correct', 'incorrect'])\n",
2323 " results['average'] = results.correct.div(results[['correct', 'incorrect']].sum(1))\n",
2324 " if detail:\n",
2325 " print(results.sort_values('average', ascending=False))\n",
2326 " return results.loc[results.category=='total', ['correct', 'incorrect', 'average']].squeeze().tolist()"
2327 ]
2328 },
2329 {
2330 "cell_type": "markdown",
2331 "metadata": {
2332 "slideshow": {
2333 "slide_type": "slide"
2334 }
2335 },
2336 "source": [
2337 "### Settings"
2338 ]
2339 },
2340 {
2341 "cell_type": "code",
2342 "execution_count": 51,
2343 "metadata": {
2344 "ExecuteTime": {
2345 "end_time": "2018-12-07T00:33:48.336482Z",
2346 "start_time": "2018-12-07T00:33:48.331924Z"
2347 },
2348 "slideshow": {
2349 "slide_type": "fragment"
2350 }
2351 },
2352 "outputs": [],
2353 "source": [
2354 "ANALOGIES_PATH = PROJECT_DIR / 'data' / 'analogies' / 'analogies-{}.txt'.format(LANGUAGE)\n",
2355 "gensim_path = PROJECT_DIR / 'gensim' / SOURCE / LANGUAGE / FILE_NAME\n",
2356 "if not gensim_path.exists():\n",
2357 " gensim_path.mkdir(parents=True, exist_ok=True)"
2358 ]
2359 },
2360 {
2361 "cell_type": "markdown",
2362 "metadata": {
2363 "slideshow": {
2364 "slide_type": "slide"
2365 }
2366 },
2367 "source": [
2368 "### Sentence Generator"
2369 ]
2370 },
2371 {
2372 "cell_type": "code",
2373 "execution_count": 52,
2374 "metadata": {
2375 "ExecuteTime": {
2376 "end_time": "2018-12-10T07:30:26.875274Z",
2377 "start_time": "2018-12-10T07:30:26.871082Z"
2378 },
2379 "slideshow": {
2380 "slide_type": "fragment"
2381 }
2382 },
2383 "outputs": [],
2384 "source": [
2385 "sentence_path = PROJECT_DIR / 'vocab' / SOURCE / LANGUAGE / '{}.txt'.format(FILE_NAME)\n",
2386 "sentences = LineSentence(str(sentence_path))"
2387 ]
2388 },
2389 {
2390 "cell_type": "markdown",
2391 "metadata": {
2392 "slideshow": {
2393 "slide_type": "slide"
2394 }
2395 },
2396 "source": [
2397 "### Model"
2398 ]
2399 },
2400 {
2401 "cell_type": "code",
2402 "execution_count": 53,
2403 "metadata": {
2404 "ExecuteTime": {
2405 "end_time": "2018-12-10T07:36:42.963953Z",
2406 "start_time": "2018-12-10T07:30:46.174109Z"
2407 },
2408 "slideshow": {
2409 "slide_type": "fragment"
2410 }
2411 },
2412 "outputs": [
2413 {
2414 "name": "stdout",
2415 "output_type": "stream",
2416 "text": [
2417 "Duration: 367.1s\n",
2418 " category correct incorrect average\n",
2419 "0 capital-common-countries 0 132 0.00\n",
2420 "1 capital-world 0 12 0.00\n",
2421 "2 city-in-state 0 178 0.00\n",
2422 "3 currency 0 10 0.00\n",
2423 "4 family 0 210 0.00\n",
2424 "5 gram1-adjective-to-adverb 0 600 0.00\n",
2425 "6 gram2-opposite 0 182 0.00\n",
2426 "7 gram3-comparative 0 930 0.00\n",
2427 "8 gram4-superlative 0 420 0.00\n",
2428 "9 gram5-present-participle 0 702 0.00\n",
2429 "10 gram6-nationality-adjective 0 552 0.00\n",
2430 "11 gram7-past-tense 0 1122 0.00\n",
2431 "12 gram8-plural 0 870 0.00\n",
2432 "13 gram9-plural-verbs 0 506 0.00\n",
2433 "14 total 0 6426 0.00\n",
2434 "Base Accuracy: Correct 0 | Wrong 6,426 | Avg 0.00%\n",
2435 "\n"
2436 ]
2437 },
2438 {
2439 "name": "stderr",
2440 "output_type": "stream",
2441 "text": [
2442 "/home/stefan/.pyenv/versions/miniconda3-latest/envs/ml4t/lib/python3.6/site-packages/ipykernel_launcher.py:17: DeprecationWarning: Call to deprecated `accuracy` (Method will be removed in 4.0.0, use self.evaluate_word_analogies() instead).\n"
2443 ]
2444 }
2445 ],
2446 "source": [
2447 "start = time()\n",
2448 "\n",
2449 "model = Word2Vec(sentences,\n",
2450 " sg=1,\n",
2451 " size=300,\n",
2452 " window=5,\n",
2453 " min_count=10,\n",
2454 " negative=10,\n",
2455 " workers=8,\n",
2456 " iter=20,\n",
2457 " alpha=0.05)\n",
2458 "\n",
2459 "model.wv.save(str(gensim_path / 'word_vectors.bin'))\n",
2460 "print('Duration: {:,.1f}s'.format(time() - start))\n",
2461 "\n",
2462 "# gensim computes accuracy based on source text files\n",
2463 "detailed_accuracy = model.wv.accuracy(str(ANALOGIES_PATH), case_insensitive=True)\n",
2464 "\n",
2465 "# get accuracy per category\n",
2466 "summary = accuracy_by_category(detailed_accuracy)\n",
2467 "print('Base Accuracy: Correct {:,.0f} | Wrong {:,.0f} | Avg {:,.2%}\\n'.format(*summary))"
2468 ]
2469 },
2470 {
2471 "cell_type": "code",
2472 "execution_count": 54,
2473 "metadata": {
2474 "ExecuteTime": {
2475 "end_time": "2018-12-07T00:39:36.863411Z",
2476 "start_time": "2018-12-07T00:39:36.825185Z"
2477 },
2478 "slideshow": {
2479 "slide_type": "slide"
2480 }
2481 },
2482 "outputs": [
2483 {
2484 "data": {
2485 "text/html": [
2486 "<div>\n",
2487 "<style scoped>\n",
2488 " .dataframe tbody tr th:only-of-type {\n",
2489 " vertical-align: middle;\n",
2490 " }\n",
2491 "\n",
2492 " .dataframe tbody tr th {\n",
2493 " vertical-align: top;\n",
2494 " }\n",
2495 "\n",
2496 " .dataframe thead th {\n",
2497 " text-align: right;\n",
2498 " }\n",
2499 "</style>\n",
2500 "<table border=\"1\" class=\"dataframe\">\n",
2501 " <thead>\n",
2502 " <tr style=\"text-align: right;\">\n",
2503 " <th></th>\n",
2504 " <th>token</th>\n",
2505 " <th>similarity</th>\n",
2506 " </tr>\n",
2507 " </thead>\n",
2508 " <tbody>\n",
2509 " <tr>\n",
2510 " <th>0</th>\n",
2511 " <td>managing</td>\n",
2512 " <td>0.25</td>\n",
2513 " </tr>\n",
2514 " <tr>\n",
2515 " <th>1</th>\n",
2516 " <td>dancer</td>\n",
2517 " <td>0.25</td>\n",
2518 " </tr>\n",
2519 " <tr>\n",
2520 " <th>2</th>\n",
2521 " <td>she</td>\n",
2522 " <td>0.25</td>\n",
2523 " </tr>\n",
2524 " <tr>\n",
2525 " <th>3</th>\n",
2526 " <td>republic</td>\n",
2527 " <td>0.25</td>\n",
2528 " </tr>\n",
2529 " <tr>\n",
2530 " <th>4</th>\n",
2531 " <td>graduation</td>\n",
2532 " <td>0.25</td>\n",
2533 " </tr>\n",
2534 " <tr>\n",
2535 " <th>5</th>\n",
2536 " <td>england</td>\n",
2537 " <td>0.24</td>\n",
2538 " </tr>\n",
2539 " <tr>\n",
2540 " <th>6</th>\n",
2541 " <td>kentucky</td>\n",
2542 " <td>0.24</td>\n",
2543 " </tr>\n",
2544 " <tr>\n",
2545 " <th>7</th>\n",
2546 " <td>junior</td>\n",
2547 " <td>0.23</td>\n",
2548 " </tr>\n",
2549 " <tr>\n",
2550 " <th>8</th>\n",
2551 " <td>jordan</td>\n",
2552 " <td>0.22</td>\n",
2553 " </tr>\n",
2554 " <tr>\n",
2555 " <th>9</th>\n",
2556 " <td>zoe</td>\n",
2557 " <td>0.22</td>\n",
2558 " </tr>\n",
2559 " </tbody>\n",
2560 "</table>\n",
2561 "</div>"
2562 ],
2563 "text/plain": [
2564 " token similarity\n",
2565 "0 managing 0.25\n",
2566 "1 dancer 0.25\n",
2567 "2 she 0.25\n",
2568 "3 republic 0.25\n",
2569 "4 graduation 0.25\n",
2570 "5 england 0.24\n",
2571 "6 kentucky 0.24\n",
2572 "7 junior 0.23\n",
2573 "8 jordan 0.22\n",
2574 "9 zoe 0.22"
2575 ]
2576 },
2577 "execution_count": 54,
2578 "metadata": {},
2579 "output_type": "execute_result"
2580 }
2581 ],
2582 "source": [
2583 "most_sim = model.wv.most_similar(positive=['woman', 'king'], negative=['man'], topn=10)\n",
2584 "pd.DataFrame(most_sim, columns=['token', 'similarity'])"
2585 ]
2586 },
2587 {
2588 "cell_type": "code",
2589 "execution_count": 55,
2590 "metadata": {
2591 "ExecuteTime": {
2592 "end_time": "2018-12-07T00:39:36.909259Z",
2593 "start_time": "2018-12-07T00:39:36.867047Z"
2594 },
2595 "scrolled": false,
2596 "slideshow": {
2597 "slide_type": "slide"
2598 }
2599 },
2600 "outputs": [
2601 {
2602 "data": {
2603 "text/html": [
2604 "<div>\n",
2605 "<style scoped>\n",
2606 " .dataframe tbody tr th:only-of-type {\n",
2607 " vertical-align: middle;\n",
2608 " }\n",
2609 "\n",
2610 " .dataframe tbody tr th {\n",
2611 " vertical-align: top;\n",
2612 " }\n",
2613 "\n",
2614 " .dataframe thead th {\n",
2615 " text-align: right;\n",
2616 " }\n",
2617 "</style>\n",
2618 "<table border=\"1\" class=\"dataframe\">\n",
2619 " <thead>\n",
2620 " <tr style=\"text-align: right;\">\n",
2621 " <th></th>\n",
2622 " <th>0</th>\n",
2623 " <th>1</th>\n",
2624 " <th>2</th>\n",
2625 " <th>3</th>\n",
2626 " <th>4</th>\n",
2627 " <th>5</th>\n",
2628 " <th>6</th>\n",
2629 " <th>7</th>\n",
2630 " <th>8</th>\n",
2631 " <th>9</th>\n",
2632 " </tr>\n",
2633 " </thead>\n",
2634 " <tbody>\n",
2635 " <tr>\n",
2636 " <th>'ve</th>\n",
2637 " <td>have</td>\n",
2638 " <td>’ve</td>\n",
2639 " <td>had</td>\n",
2640 " <td>been</td>\n",
2641 " <td>'d</td>\n",
2642 " <td>got</td>\n",
2643 " <td>has</td>\n",
2644 " <td>seen</td>\n",
2645 " <td>'re</td>\n",
2646 " <td>anthropology</td>\n",
2647 " </tr>\n",
2648 " <tr>\n",
2649 " <th>for</th>\n",
2650 " <td>to</td>\n",
2651 " <td>in</td>\n",
2652 " <td>and</td>\n",
2653 " <td>of</td>\n",
2654 " <td>at</td>\n",
2655 " <td>is</td>\n",
2656 " <td>the</td>\n",
2657 " <td>with</td>\n",
2658 " <td>so</td>\n",
2659 " <td>that</td>\n",
2660 " </tr>\n",
2661 " <tr>\n",
2662 " <th>work</th>\n",
2663 " <td>working</td>\n",
2664 " <td>works</td>\n",
2665 " <td>worked</td>\n",
2666 " <td>collaborate</td>\n",
2667 " <td>interfere</td>\n",
2668 " <td>partnerships</td>\n",
2669 " <td>do</td>\n",
2670 " <td>research</td>\n",
2671 " <td>happen</td>\n",
2672 " <td>resonate</td>\n",
2673 " </tr>\n",
2674 " <tr>\n",
2675 " <th>time</th>\n",
2676 " <td>moment</td>\n",
2677 " <td>night</td>\n",
2678 " <td>periods</td>\n",
2679 " <td>year</td>\n",
2680 " <td>lapse</td>\n",
2681 " <td>seconds</td>\n",
2682 " <td>slept</td>\n",
2683 " <td>tense</td>\n",
2684 " <td>sabbatical</td>\n",
2685 " <td>lifetimes</td>\n",
2686 " </tr>\n",
2687 " <tr>\n",
2688 " <th>'m</th>\n",
2689 " <td>am</td>\n",
2690 " <td>'re</td>\n",
2691 " <td>’m</td>\n",
2692 " <td>i</td>\n",
2693 " <td>optimist</td>\n",
2694 " <td>woo</td>\n",
2695 " <td>want</td>\n",
2696 " <td>was</td>\n",
2697 " <td>technologist</td>\n",
2698 " <td>bruno_giussani</td>\n",
2699 " </tr>\n",
2700 " <tr>\n",
2701 " <th>at</th>\n",
2702 " <td>the</td>\n",
2703 " <td>in</td>\n",
2704 " <td>and</td>\n",
2705 " <td>archives</td>\n",
2706 " <td>for</td>\n",
2707 " <td>on</td>\n",
2708 " <td>from</td>\n",
2709 " <td>della</td>\n",
2710 " <td>fireman</td>\n",
2711 " <td>sachs</td>\n",
2712 " </tr>\n",
2713 " <tr>\n",
2714 " <th>had</th>\n",
2715 " <td>have</td>\n",
2716 " <td>was</td>\n",
2717 " <td>got</td>\n",
2718 " <td>has</td>\n",
2719 " <td>could</td>\n",
2720 " <td>'d</td>\n",
2721 " <td>'ve</td>\n",
2722 " <td>were</td>\n",
2723 " <td>would</td>\n",
2724 " <td>wanted</td>\n",
2725 " </tr>\n",
2726 " <tr>\n",
2727 " <th>put</th>\n",
2728 " <td>putting</td>\n",
2729 " <td>squeeze</td>\n",
2730 " <td>bring</td>\n",
2731 " <td>puts</td>\n",
2732 " <td>take</td>\n",
2733 " <td>strap</td>\n",
2734 " <td>sew</td>\n",
2735 " <td>taped</td>\n",
2736 " <td>hold</td>\n",
2737 " <td>throw</td>\n",
2738 " </tr>\n",
2739 " <tr>\n",
2740 " <th>your</th>\n",
2741 " <td>my</td>\n",
2742 " <td>you</td>\n",
2743 " <td>their</td>\n",
2744 " <td>our</td>\n",
2745 " <td>her</td>\n",
2746 " <td>his</td>\n",
2747 " <td>purse</td>\n",
2748 " <td>yourself</td>\n",
2749 " <td>wallet</td>\n",
2750 " <td>the</td>\n",
2751 " </tr>\n",
2752 " <tr>\n",
2753 " <th>things</th>\n",
2754 " <td>ways</td>\n",
2755 " <td>thing</td>\n",
2756 " <td>modes</td>\n",
2757 " <td>distractions</td>\n",
2758 " <td>attributes</td>\n",
2759 " <td>something</td>\n",
2760 " <td>technologies</td>\n",
2761 " <td>projects</td>\n",
2762 " <td>places</td>\n",
2763 " <td>sorts</td>\n",
2764 " </tr>\n",
2765 " </tbody>\n",
2766 "</table>\n",
2767 "</div>"
2768 ],
2769 "text/plain": [
2770 " 0 1 2 3 4 5 \\\n",
2771 "'ve have ’ve had been 'd got \n",
2772 "for to in and of at is \n",
2773 "work working works worked collaborate interfere partnerships \n",
2774 "time moment night periods year lapse seconds \n",
2775 "'m am 're ’m i optimist woo \n",
2776 "at the in and archives for on \n",
2777 "had have was got has could 'd \n",
2778 "put putting squeeze bring puts take strap \n",
2779 "your my you their our her his \n",
2780 "things ways thing modes distractions attributes something \n",
2781 "\n",
2782 " 6 7 8 9 \n",
2783 "'ve has seen 're anthropology \n",
2784 "for the with so that \n",
2785 "work do research happen resonate \n",
2786 "time slept tense sabbatical lifetimes \n",
2787 "'m want was technologist bruno_giussani \n",
2788 "at from della fireman sachs \n",
2789 "had 've were would wanted \n",
2790 "put sew taped hold throw \n",
2791 "your purse yourself wallet the \n",
2792 "things technologies projects places sorts "
2793 ]
2794 },
2795 "execution_count": 55,
2796 "metadata": {},
2797 "output_type": "execute_result"
2798 }
2799 ],
2800 "source": [
2801 "similars = pd.DataFrame()\n",
2802 "for id in valid_examples:\n",
2803 " word = id_to_token[id]\n",
2804 " similars[word] = [s[0] for s in model.wv.most_similar(id_to_token[id])]\n",
2805 " \n",
2806 "similars.T"
2807 ]
2808 },
2809 {
2810 "cell_type": "markdown",
2811 "metadata": {
2812 "slideshow": {
2813 "slide_type": "slide"
2814 }
2815 },
2816 "source": [
2817 "#### Continue Training"
2818 ]
2819 },
2820 {
2821 "cell_type": "code",
2822 "execution_count": 62,
2823 "metadata": {
2824 "ExecuteTime": {
2825 "end_time": "2018-12-07T00:45:39.510380Z",
2826 "start_time": "2018-12-07T00:39:36.914176Z"
2827 },
2828 "slideshow": {
2829 "slide_type": "fragment"
2830 }
2831 },
2832 "outputs": [
2833 {
2834 "name": "stderr",
2835 "output_type": "stream",
2836 "text": [
2837 "/home/stefan/.pyenv/versions/at-3.6/lib/python3.6/site-packages/ipykernel_launcher.py:5: DeprecationWarning: Call to deprecated `accuracy` (Method will be removed in 4.0.0, use self.evaluate_word_analogies() instead).\n",
2838 " \"\"\"\n",
2839 "/home/stefan/.pyenv/versions/at-3.6/lib/python3.6/site-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
2840 " if np.issubdtype(vec.dtype, np.int):\n"
2841 ]
2842 },
2843 {
2844 "name": "stdout",
2845 "output_type": "stream",
2846 "text": [
2847 "1 | Duration: 36.0 | Accuracy: 8.03% \n",
2848 "2 | Duration: 37.1 | Accuracy: 8.69% \n",
2849 "3 | Duration: 36.1 | Accuracy: 8.91% \n",
2850 "4 | Duration: 36.1 | Accuracy: 8.43% \n",
2851 "5 | Duration: 36.5 | Accuracy: 8.52% \n",
2852 "6 | Duration: 36.3 | Accuracy: 8.77% \n",
2853 "7 | Duration: 36.0 | Accuracy: 8.96% \n",
2854 "8 | Duration: 36.1 | Accuracy: 8.64% \n",
2855 "9 | Duration: 36.4 | Accuracy: 8.88% \n",
2856 "10 | Duration: 35.9 | Accuracy: 8.57% \n"
2857 ]
2858 }
2859 ],
2860 "source": [
2861 "accuracies = [summary]\n",
2862 "for i in range(1, 11):\n",
2863 " start = time()\n",
2864 " model.train(sentences, epochs=1, total_examples=model.corpus_count)\n",
2865 " detailed_accuracy = model.wv.accuracy(str(ANALOGIES_PATH))\n",
2866 " accuracies.append(accuracy_by_category(detailed_accuracy, detail=False))\n",
2867 " print('{} | Duration: {:,.1f} | Accuracy: {:.2%} '.format(i, time() - start, accuracies[-1][-1]))\n",
2868 "\n",
2869 "pd.DataFrame(accuracies, columns=['correct', 'wrong', 'average']).to_csv(gensim_path / 'accuracies.csv', index=False)\n",
2870 "model.wv.save(str(gensim_path / 'word_vectors_final.bin'))"
2871 ]
2872 },
2873 {
2874 "cell_type": "markdown",
2875 "metadata": {
2876 "slideshow": {
2877 "slide_type": "slide"
2878 }
2879 },
2880 "source": [
2881 "## The `google` command-line Tool"
2882 ]
2883 },
2884 {
2885 "cell_type": "markdown",
2886 "metadata": {
2887 "slideshow": {
2888 "slide_type": "fragment"
2889 }
2890 },
2891 "source": [
2892 "### Run from Command Line"
2893 ]
2894 },
2895 {
2896 "cell_type": "code",
2897 "execution_count": null,
2898 "metadata": {
2899 "slideshow": {
2900 "slide_type": "fragment"
2901 }
2902 },
2903 "outputs": [],
2904 "source": [
2905 "%%bash\n",
2906 "file_name=../data/wiki/en/wiki.txt\n",
2907 "time ./word2vec -train \"$file_name\" -output vectors_en.bin - cbow 1 -size 300 -min-count 10 -window 10 -negative 10 -hs 0 -sample 1e-4 -threads 8 -binary 1 -iter 1"
2908 ]
2909 },
2910 {
2911 "cell_type": "markdown",
2912 "metadata": {
2913 "slideshow": {
2914 "slide_type": "slide"
2915 }
2916 },
2917 "source": [
2918 "### Load Trained Model & Word Vectors via `gensim`"
2919 ]
2920 },
2921 {
2922 "cell_type": "code",
2923 "execution_count": 32,
2924 "metadata": {
2925 "slideshow": {
2926 "slide_type": "fragment"
2927 }
2928 },
2929 "outputs": [],
2930 "source": [
2931 "file_name = 'word2vec/word_vectors/vectors_en.bin'\n",
2932 "model = KeyedVectors.load_word2vec_format(file_name, binary=True, unicode_errors='ignore')"
2933 ]
2934 },
2935 {
2936 "cell_type": "code",
2937 "execution_count": 33,
2938 "metadata": {
2939 "slideshow": {
2940 "slide_type": "fragment"
2941 }
2942 },
2943 "outputs": [
2944 {
2945 "data": {
2946 "text/plain": [
2947 "(100000, 300)"
2948 ]
2949 },
2950 "execution_count": 33,
2951 "metadata": {},
2952 "output_type": "execute_result"
2953 }
2954 ],
2955 "source": [
2956 "vectors = model.vectors[:100000]\n",
2957 "vectors /= norm(vectors, axis=1).reshape(-1, 1)\n",
2958 "vectors.shape"
2959 ]
2960 },
2961 {
2962 "cell_type": "code",
2963 "execution_count": 34,
2964 "metadata": {
2965 "slideshow": {
2966 "slide_type": "fragment"
2967 }
2968 },
2969 "outputs": [],
2970 "source": [
2971 "words = model.index2word[:100000]\n",
2972 "word2id = {w:i for i, w in enumerate(words)}"
2973 ]
2974 },
2975 {
2976 "cell_type": "markdown",
2977 "metadata": {
2978 "slideshow": {
2979 "slide_type": "slide"
2980 }
2981 },
2982 "source": [
2983 "### Compute Accuracy"
2984 ]
2985 },
2986 {
2987 "cell_type": "code",
2988 "execution_count": 37,
2989 "metadata": {
2990 "slideshow": {
2991 "slide_type": "fragment"
2992 }
2993 },
2994 "outputs": [],
2995 "source": [
2996 "analogy_path = PROJECT_DIR / 'data/analogies/analogies-en.txt'\n",
2997 "accuracy = model.accuracy(questions=str(analogy_path), restrict_vocab=100000)"
2998 ]
2999 },
3000 {
3001 "cell_type": "code",
3002 "execution_count": 38,
3003 "metadata": {
3004 "slideshow": {
3005 "slide_type": "fragment"
3006 }
3007 },
3008 "outputs": [
3009 {
3010 "name": "stdout",
3011 "output_type": "stream",
3012 "text": [
3013 " category correct incorrect average\n",
3014 "0 capital-common-countries 459 47 0.91\n",
3015 "7 gram3-comparative 1207 125 0.91\n",
3016 "10 gram6-nationality-adjective 1472 168 0.90\n",
3017 "4 family 438 68 0.87\n",
3018 "1 capital-world 7058 1132 0.86\n",
3019 "12 gram8-plural 1082 250 0.81\n",
3020 "14 total 17974 6458 0.74\n",
3021 "11 gram7-past-tense 1118 442 0.72\n",
3022 "9 gram5-present-participle 726 330 0.69\n",
3023 "13 gram9-plural-verbs 580 290 0.67\n",
3024 "2 city-in-state 2601 1641 0.61\n",
3025 "8 gram4-superlative 605 387 0.61\n",
3026 "6 gram2-opposite 254 502 0.34\n",
3027 "5 gram1-adjective-to-adverb 312 680 0.31\n",
3028 "3 currency 62 396 0.14\n",
3029 "\n",
3030 "Overall Accuracy: Correct 17,974 | Wrong 6,458 | Avg 73.57%\n",
3031 "\n"
3032 ]
3033 }
3034 ],
3035 "source": [
3036 "summary = accuracy_by_category(accuracy, detail=True)\n",
3037 "print('\\nOverall Accuracy: Correct {:,.0f} | Wrong {:,.0f} | Avg {:,.2%}\\n'.format(*summary))"
3038 ]
3039 },
3040 {
3041 "cell_type": "markdown",
3042 "metadata": {
3043 "slideshow": {
3044 "slide_type": "slide"
3045 }
3046 },
3047 "source": [
3048 "### Project Data using `tensorboard` Projector"
3049 ]
3050 },
3051 {
3052 "cell_type": "code",
3053 "execution_count": 66,
3054 "metadata": {
3055 "slideshow": {
3056 "slide_type": "fragment"
3057 }
3058 },
3059 "outputs": [],
3060 "source": [
3061 "PROJECTION_LIMIT = 10000\n",
3062 "proj_path = Path('word2vec', 'projector')\n",
3063 "pd.Series(words).iloc[:PROJECTION_LIMIT].to_csv(proj_path / 'meta_data.tsv', index=False, header=None, sep='\\t')\n",
3064 "pd.DataFrame(vectors).iloc[:PROJECTION_LIMIT].to_csv(proj_path / 'embeddings.tsv', index=False, header=None, sep='\\t')"
3065 ]
3066 },
3067 {
3068 "cell_type": "markdown",
3069 "metadata": {
3070 "slideshow": {
3071 "slide_type": "slide"
3072 }
3073 },
3074 "source": [
3075 "### Project Analogies"
3076 ]
3077 },
3078 {
3079 "cell_type": "markdown",
3080 "metadata": {
3081 "slideshow": {
3082 "slide_type": "fragment"
3083 }
3084 },
3085 "source": [
3086 "#### Incremental PCA"
3087 ]
3088 },
3089 {
3090 "cell_type": "code",
3091 "execution_count": 35,
3092 "metadata": {
3093 "slideshow": {
3094 "slide_type": "fragment"
3095 }
3096 },
3097 "outputs": [
3098 {
3099 "data": {
3100 "text/plain": [
3101 "0 3.69\n",
3102 "1 3.19\n",
3103 "dtype: float64"
3104 ]
3105 },
3106 "execution_count": 35,
3107 "metadata": {},
3108 "output_type": "execute_result"
3109 }
3110 ],
3111 "source": [
3112 "pca = IncrementalPCA(n_components=2)\n",
3113 "\n",
3114 "vectors2D = pca.fit_transform(vectors)\n",
3115 "pd.Series(pca.explained_variance_ratio_).mul(100)"
3116 ]
3117 },
3118 {
3119 "cell_type": "markdown",
3120 "metadata": {
3121 "slideshow": {
3122 "slide_type": "slide"
3123 }
3124 },
3125 "source": [
3126 "#### Group Analogies by Category"
3127 ]
3128 },
3129 {
3130 "cell_type": "code",
3131 "execution_count": 39,
3132 "metadata": {
3133 "slideshow": {
3134 "slide_type": "fragment"
3135 }
3136 },
3137 "outputs": [],
3138 "source": [
3139 "results = pd.DataFrame()\n",
3140 "correct = incorrect = 0\n",
3141 "for section in accuracy:\n",
3142 " correct += len(section['correct'])\n",
3143 " incorrect += len(section['incorrect'])\n",
3144 " df = pd.DataFrame(section['correct']).apply(lambda x: x.str.lower()).assign(section=section['section'])\n",
3145 " results = pd.concat([results, df])"
3146 ]
3147 },
3148 {
3149 "cell_type": "markdown",
3150 "metadata": {
3151 "slideshow": {
3152 "slide_type": "slide"
3153 }
3154 },
3155 "source": [
3156 "#### Identify Analogy most similar in 2D"
3157 ]
3158 },
3159 {
3160 "cell_type": "code",
3161 "execution_count": 41,
3162 "metadata": {
3163 "slideshow": {
3164 "slide_type": "fragment"
3165 }
3166 },
3167 "outputs": [],
3168 "source": [
3169 "def find_most_similar_analogy(v):\n",
3170 " \"\"\"Find analogy that most similar in 2D\"\"\"\n",
3171 " v1 = vectors2D[v[1]] - vectors2D[v[0]]\n",
3172 " v2 = vectors2D[v[3]] - vectors2D[v[2]]\n",
3173 " idx, most_similar = None, np.inf\n",
3174 " \n",
3175 " for i in range(len(v1)):\n",
3176 " similarity = cosine(v1[i], v2[i])\n",
3177 " if similarity < most_similar:\n",
3178 " idx = i\n",
3179 " most_similar = similarity\n",
3180 " return idx"
3181 ]
3182 },
3183 {
3184 "cell_type": "code",
3185 "execution_count": 42,
3186 "metadata": {
3187 "slideshow": {
3188 "slide_type": "fragment"
3189 }
3190 },
3191 "outputs": [],
3192 "source": [
3193 "def get_plot_lims(coordinates):\n",
3194 " xlim, ylim = coordinates.agg(['min', 'max']).T.values\n",
3195 " xrange, yrange = (xlim[1] - xlim[0]) * .1, (ylim[1] - ylim[0]) * .1\n",
3196 " xlim[0], xlim[1] = xlim[0] - xrange, xlim[1] + xrange\n",
3197 " ylim[0], ylim[1] = ylim[0] - yrange, ylim[1] + yrange\n",
3198 " return xlim, ylim"
3199 ]
3200 },
3201 {
3202 "cell_type": "code",
3203 "execution_count": 43,
3204 "metadata": {
3205 "slideshow": {
3206 "slide_type": "slide"
3207 }
3208 },
3209 "outputs": [
3210 {
3211 "data": {
3212 "image/png": "iVBORw0KGgoAAAANSUhEUgAABJYAAAKACAYAAADOyPLPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3XecHXX1//HXIfQOShOBgCK9G6QTqoB08FBECCKKFAFZEZASFBB1KV8LEooElXYApUhHCR1CF34UkSoI0jtIyfn98Zk1N+tms0l2d+7ufT8fj31k79y5c8/c7M7OnDmf87HMREREREREREREZHJNU3cAIiIiIiIiIiIyMCmxJCIiIiIiIiIiU0SJJRERERERERERmSJKLImIiIiIiIiIyBRRYklERERERERERKaIEksiIiIiIiIiIjJFlFjqATNbzswuM7M3zew1M7vKzFbs5fcYbWZXd1o23WRuY6SZjZnEOn2+L82umT6DnvwfV/+vj/ZHPCJTazAcL83s/8zs3k7LZjOzj8xsq07LjzazmJz37uL90syGd/P8CDN7emreQ6TVTOpY1BvHkYbXDa1+j1ebininNbODzewRM3vPzP5pZr82s7mmNkYrpp3S2ESkbzTTNYnI1FJiaRLMbCXgNuB5YAlgceBB4AYz+3xvvU9mjsjMTRre93rgh721/Wqb/bIvzayZPgMzOxy4dlLrZebIzFyyH0ISmSqD6Hh5I7CCmc3esGxtYFpg407rrg7c0ovvLSJTqSfHov4475pMPwH2BfYGPgVsTon7WjMbAmBmawMfTsG2Hwd26aU4RaQXNNM1iUhvUGJp0k4BHgX2zswXM/PVzDwYuJe+PQHpiztLde1LM2mmz2BawPr5PUX60mA5Xt5E+d1cs2HZesCrNCSWzGwa4EsosSTSbKbkWFR3Rc+uwPGZeUNmvp+ZDwBbAB8AS1XrDJnCbde9byLyv5rpmkRkqimx1A0z+xywGnBGZmanp4+kZJkxs8+Y2blm9qKZvW9mj5nZbg3b6SiRXtrM7jCzD8xsrJmt0LDO6OprSTNLYF3gqOp1G1brbGFmt1flkm+Y2XVm9oXe3Jdq3T3N7HEz+4+Z/T8z26bTvjxnZqua2f1m9o6ZnW1m05nZblXp9utmNrLT+2e1b7eY2btm9mczm8PMNq3e620z+01jqbaZzWlmv61KQ98xs0vMbKGG50ea2flmtouZPWNmb5nZ6WY2Yz98BmlmQye2zMrQlTvMbOOqrP1dM/tjR0m7mb0IHAWsW73umGr5aDNrszK85i0z29A6DdmpPuvjzOz56nO70cxWaXh+mJndVH1mr1Sf0bxdfSYivWUwHS8z8xXgYWCdhsXrAT8FPt/wu78MJQH1QPWe05jZYWb2bBX3XVYqDDriHm5mD5nZGmb2dzP7/UQ+y6+a2T+qz+cSSvWCiPTAZByLuj2OmNmjZrZXp21H4/GqmxiGV8e4lc3sHitD2/5qZot087IhwKpm9t8bTpn5n8xcOzMfMrPzgRuq7aeZ3VJ9P5uZnWrl3OwDM3vKzA6untuk2rdFgLOq132+em7+6lj8ppm9VH0/36T2TUSmXk+OU53P/6vXdb4mGGNmO1THgPfN7PPVOr8ys72q64BvVuuuWl0zvG9mT5rZMWY2fcO2njazbczszOoa4ikz27HT+69mZndW2/i7me1SLT+1OkY1rru3mZ019Z+WDBRKLHVvuerf/9f5icy8NTNPrx5eDIwDVgbmAA4ARpnZop1edhKwD/BZykXLn61TEiQzH81MowzFODozLTOvN7OlgQuBXwALAIsB/wBG9ea+mNl3gOOrOOcBTgAusAn7BswCHAxsCQwDNgB+T7m7vwKwA3CYmW3U6a1+AnwLWLL6DH4HjKAMMRlevW73Ko5pgKuABYGVgC9QPuMrrCoJr6wKbAOsUe3jl4CD+uEz6ImFge9TPqehwGzAz6r3mx84Grix+j8+vOF1WwOLAktm5vVdbPc0yme2ITAvcAZwvZktUH02fwauqZ5bAZgBOL+L7Yj0psF0vKTa5tpQktzAssCpwAvAl6t11gDuyMxPqsc/Ab4JfBWYH7gEuKbTxeQslOPL7pTj3wTMbDngXODnwHyU3/fvT0bcIq2up8eijmVdHkeAC4D/XlRZGRq7IfDHHsYxM+U49k3Kceh5yt/riRkF7AY8ZGY/NbPNzWzWhjh3pCS4qWJcq3rqNOBzlET4bMBOwA/MbN3MvLrat2eA3avX/aM6lv4VeINyfrUE8G/KcVaV1CJ9b7KOU5OwH+V3eaHM/Ee1bGVgK+BLmXmGmS0FXA/8gXJ9sAHluu0nnbZ1PHA75Zh1IHB2xzmMmc1Dub64tnp+F+BEM1ufcrzcwsxmadjW1yjnM9IilFjqXscf9Jcmsd6ewHcy81+Z+WFmXkU5UKzSab0TM/Oe6m74fsCclAuQnngWGJaZ52Xme5n5GqWEclgPX9/TfTkc+FlmXpuZb2Xmb4EAjmhYZ07gyMx8NjMfqZ7fANg3M1/LzGuB+ykJoUanZ+bDmflP4LeUEu99qvLPeygHvI7XbEBJGo3IzGcy81+UC7HFgG0btvlpysnS85n5DHAmJbnU159BTywA7JmZj2fmy8Cvuomt0ULAHtU+T6C6+N4N2CszH6nK5X8P3AV8m3JSOS9wX/Vz8jywM+WPi0hfGkzHSygXmcOqC7B1gHsz823gOsYPh/tvfyUzmw3YHzg0M+/MzDcy81hKSfvBDdsdChxbnTh+wv/6HnB9Zo6qjj9XApNzginS6np6LJqU84G1zewz1eNtgb9Ux4GemA04KjPvy8w3gXa6Pwc4HNgLeJ1yHLgceLGqKuhuCNyRwE6Z+WRmfpSZdwBj6P54tyOlEnKfzPx3Zr5OSWAvQ6ncEpG+1VvHKYBpMvOQ6nypw/LA1zLzierxD4BrMvP0zHw7M5+iDLfbt7FqCbg9M8+o1rmEcmOv4/xsb+BFyjXgG5k5lpKYWp5yzvQm5dqu43plUUoCW1qEEkvde636d1LDEF6nZGyfqMqQk5IpnqXTeo91fFOdmPwdWLongWTmO8BCVoZzvGxmHwN/6+I9JmaS+2JluNRnKBdCje6iVL40apyl7C3g75nZ2FDyPWCmSbzmhU4HwcbXrAS8mJkvdDxZnZg93imWRzLzrYbHbzL+YN1Zb38Gk/JSZj7dw9gajc3Mjyby3BcpQ28eqkras/p52wj4Qma+QUlgXWZlGM5oyh2DcZMZu8jkGkzHSyh9lqanXAiux/iTo2uB9asLvTUY319pKUp1YE+OHd31ZFoOuKfTsie6WlFEutTTY1G3qhtnD1GqqWHK7r7f2fB9t+cAWYyqKpHmBr5CSWT/kHLjaGJeBn5oZejee9UxdVu6P94No9yEGtdwHvER5RysR0OGRWSq9MpxqnJrF8seqW6qdRgGbN/p2uFWynnO0Ib1Go9ZMOFxa3ng7szxQ/cy86TMPDkzx1EqxXeqntoZuHAiN9BkkFJiqXv3Akm5gzMBK+PvD7PSE+h6yh/irwPzVmXHdzDpxszT0MMLfjNbF7iUUp74JUqJ9eTMFDbJfQHeb4hrglWAzkNQOsfdVSKk8/5Pzmve7yKOrmJ5v4t1JvZz3aufQSddTf87ObE1ereb5z6gfI5DqpL2xq+dATJzP8rF6RmUZN1I4P6qokKkrwym4yWZ+SIlubUOJbF0Q/XUdZQhfJtRTsbuqJZPzvGzu9/xaSeyDRHpmZ78re+pC4AdzWwByg2vKycnkMzsfB7Q5e+ylf5sn2143duZeWVmbkOpBNikq9dVLqLcXNoXWLA6pp4/sfeqfMD4ofidv07rwa6JyNSZ0uNUV9cbXZ1TdF72AeOH+nb++nvDet1duxjdH1cuADap2gdoGFwLUmKpG9WFxTXAXl2MOT+EMmRpWcoFy96ZeVtmvlX9Qv3PgYLxs3pQjUFdDHhkIm/fOcO7DXB/Zh5ZlTt/CKzVxeumeF+qqoCn+N8hbMOAu3v6Xr3kAWC+hhL0jj4ni09pLL34Gbxe/Ttnw/M9/r9oMCVZ/AcpB/UJKiDMbDszW8nM5jOzzaphcqMyc29K7J+fwhhFemQwHS8b3EjpebYEVcPfzHyJcnw6snqP96p1/0FJ5E7t8fNhYMVOyya3WlKkZfXwWNSVrv4mn0/5HT4MuDQz/9NrgU5oQ+BRM5u/i+fepFR5Q6cYq+PnBsAPMvP6zHzdzKajtBJo1Hnf/gYsU63bsS0zs30mEoOI9KIeHqdeZ8JrDZjyc/m/0en8pOrNutckhto2eohSYd64jcPM7HvVw9spfSh/DEyfmZ2rn2SQU2Jp0g6kXJSfYmbzVF/tlHLA44DnKNnd7c1sFjNbEjiHcjDo/It6rJUZu+YGfkk5UbhwIu/7JLCGmc1a9fj4O7C4lY7+s5jZ1pSGkEzGAWFS+wJlrGybmW1kZaaREcD2wLE9fI9ekZk3US7kzjSzhau7hWdQPpeeNs7sylR/BtWQvAeAg8xsdjNbg9L3aHI9CSxlZguZWedhg12qhtadDZxhZquY2cxmtitwFiXhNCtwkZntb2Zzm9nMlF4K/6FcsIr0pcF0vISSWFqZMjy18S7etZRhqf8d0lY9f3IV96pWZr08hJIUOmEy3vNEYGMz27U6/uxESW6JSM/15G99Z52PI2Tmk5TE8D704d33LL0pLweuM7P1q+PWp8xsH0pPt19Wqz4NpJUZZ6cH3qY03d6qOh9ZhHKO8AkTHlOfpMxCO2O1b+dRhuKcVZ1jzUU59rQxPoklIn1rUsepm4BlzWyr6phwOGUI65T4CeXc4vtWZt1ejHL+tf5kDFf7DaXNwJHVOc7alN5sd0EZzkvpSbsP5RgjLUaJpUnIzEcpMwMNpfxhfoRSNbNOZv6z6hG0I2UM/quUkuRTgPsovXoaHVU99wLlDv3m3dz9Oo6SJHiF0sH/NMosan+mnETsxvgZhTq/zxTtS7XO6ZSZy35b7c9+wJaZeXNP3qOXbU8p5XyY0p/pI2DDTr2cJksvfgZfr173ImX2pm9NQTgXUvq2PEyZsaWn9qZc0F5J6a3wTWCzzLy3atL3ZUrFxtOUn7Vtq+cn5z1EJttgOl5Wbqz+vaHT8murfzv3SjqKkvjueN/NgPUy8/GevmFm3l3F+yNKU88dKE03RaSHevK3vgudjyMdzqf8re98HOhtu1AmIDmJcvx4mNJnae2OO/9V7IdQLgjfpFQ3bktJgP+bck5xHeX8oPFYdwilAe+blMkDPqScK8xCmTzhiWr9dRqqMEWkD/XgnOk+ysy5vwb+xfiZqqfkvf4ObApsRzlW3ERJmn99MrbxAuWYtE21jTMoEwA0Xh91DMPVMLgWZA39t6SPmNlQyvCqRTs1cxYRkQY6XopIM6kqD+fLzAPrjkVEpJmZ2WrAbzKzc1sAaQHT1h2AiIiIiEgzqXq7fYbSFHvjmsMREWlaVeuNGSj9lUbVHI7UREPhREREREQmNAYYCxyfmepRKCIyccdQ+mg+D5xecyxSEw2FExERERERERGRKaKKJRERERERERERmSJKLImIiIiIiIiIyBRRYklERERERERERKaIEksiIiIiIiIiIjJFlFgSEREREREREZEposSSiIiIiIiIiIhMESWWRERERERERERkiiixJCIiIiIiIiIiU0SJJRERERERaTlmlmZ2eadl05rZS2Y2ZjK3NcLMRlbfn2xmq/RepCIizW3augMQERERERGpwb+AL5rZfJn572rZpsALU7PRzDxgqiMTERlAVLEkIiIiIiKt6CPgQuBrDcu+Dpzd8cDMvmBm15vZDWZ2v5kd1PDcd8zsXjO7DFi5YfloMxtuZnOZ2RVmNtbM7jOzb1XPT2NmPzWzm6vndquWDzWzP5vZqWZ2i5ldbWaz9/FnICIy1QZVYqndbGS72TF1xyEiA1u7WbabTdvweHj7ZJbET2L7T/fWtkRERGSqnA10JHbmBBYF7m14fnngkMxcDxgGfNPMZjCzpYEDgXWArYAFutj2bsCLmbkqsDawsJlNA+wBfDYz1wY2AA4zs2Wq16wP/Coz1wIeAHbq1b2VHqnOBW9sN7uh3ezOdrNv1x2TSDMbVIklERERERGRnsrMewAzs5UABy7otMpfgd3N7FbgOuAzwLyUhNBVmflOZiZwdRebvw34ipmdSamEGpWZ44BNgIuq938buBLYuHrNQ5n5UPX9I9V7ST02aCsJxQ2Ag9tLMlFEujAoeyy1m+0JjAA+AWYA9mzL/Ft7aaj3EeXOwnzA79syT2g3mwUYBSwEzAg8Bezcljmu3ex+4LfANsBswO5tmQ/28y6JSJNoL3cafwKsQTm+/Lot8+x2s6HAr4DngGWBdwBvy3yr3WwJynHkXeD6hm2NAIa2ZY6sHo8GRrdljqmqmn4DbEk5Zp0ErAQMAd4Ctq+2dwLl7uYnwB3Ad9syP+mr/RcRERmEzqZcOywP7Aws0fDcKEqCZ3hmfmRm91D+Fne+QW+dN5qZY81sKWAt4IvA3Wa2zkTW7/jb/V6n5UMmb1ekt7VlvtNe/t+Xbzf7B/BzYDnKteEZbZmj2s1WBk4BkvKzcUBb5u3tZp+mnM/NS7n2PqAt865adkSkDw3GiiUDFgTWa8tcBzgUaGt4flVgM8pF4Z7t5e7E54Er2jLXbcv8EvAhMLxafzbg/SpbfQRwWL/shYjU7S/tZmOqIXAnNyzfA/hsW0P5enun8vW2/y1fPxv4cVvmxpST056YF/ioLXNNyjHtsbbMtdsy1wDGUu6qrgBs2Za5YlvmKsDTlDupIiIi0nPnALsA72Vm58bdnwOur5JKawMdVStjgI3MbEYrw+e37rxRKze7v5yZl2fmUcD9wBeAa4Ftq3VmoVQrXdP7uyW9od1sAWA14C7gYOCZtsz1gTWBEVUl05GUm42rU84Vl69e/n/AdW2Z6wLfAc5vN5u+v/dBpK8NxoqlBG4ALmo3mxuYjlI50OGS6m7+e+1mV1IuBE8ENm03u6l6/VBKqSuUuwTnVN8/AswbEdNR/gDsAXzL3V/p210SkRps0Jb5MZQeS8DIavkmwB8A2jLfro4jGwN/Ah5qm7B8faF2s9mApdoyO0rkrwR+2YP3n4lS5URb5rPtZh+2m/2Vctyer4rhIiDbS9PQW4Dr2zL/OeW7LCIi0noy80Uzux34fRdPHwL8ysxeA+6hXGfMlZn3mdlpwK3Am5Sq4c4uBc4ws/0ooyLGAldVzy1rZrcB44CRmfmYlepnaR5/aTfL6vt92zKfaDfbEqDjX2AWSrLxeuDYdrMVKT8n51bPbwLsC1CNoHkLWAa4r5/2QaRfDLbEklGGugXwlbbMu9vNlmPCi7jOZafjgL2BdSl3/t9oNzuBhrLTtsz3ImKGeQ4++CtvXHDBCsDfKI39/g283Xe7IyJNanLK17srlR/X6fGMndZ9B6Dd7CuU49SmbZkvtpcT1E9Vw+yWoVRgrgSc1252XFvm2Yh0YuXkuOMGCsBrmbmtlaq8DShl/Xtk5n5WhmkOycwzawlWRKQfZObQhu83b/h+DNXohcy8lvHVJ51ffzITVjV3LB/R8HDLzs9X9uvidU8zftQEmTl6Iq+V/vHfm4wNhgDfacsc23nldrPLKedkw4GRDT2ZJnbeKDJoDLbE0jKUpM9bwL3tZgZ8q9M627ebnU2pZNoY2JVS+np7lVSaj1LK2rmP0nem++xnj55mlllmBz5VLZsGOCcinqA05/sb8Ky762AhMnh1lK//sX18+fq2E1u5LfPNdrMn2s3Wbcu8kTKEbVz19LPAFgDtZosB6wGndrGZzwH3VUmlWSn9H65pNxsGbNuWeSgwpt1sRiZy8itS2SA7nSRn5vDq2/upLnR0MSMiItKlqynXjmPbzaajVKXtA/wAOLEt87x2sz8BL1Cqma4FtgNOr24GzkjP2yKIDBgDPrFUXdj9CZgZeAw4hpL4uRt4ndLbZJ2GlzxFGcP8aeCstsx7281epdzp35BShXQhMFfj+7j7yWftv/8r+eGHo6r3glIdtQxl+tDvURrpvhIRrwKvAi9RSiHvAB5z97d6efdFpP+dCSzb3lC+3pb5WHv35eu7AaPazT6mlNC/Vi2/Edij2ta/gD9O5PVnA1u0lxlp3qAc8xakTIe8c7vZWOBjSlL9G1Oxb9KCzOzpzBxaDcEYnZnDq4qloVk1lhcREREAjgV+0V6GTo6jNO9+qt3sLOCs6lxvNuCw6ubigcCZ7Wa7UM7VvC3zo9qiF+kjVmbHbA3VrHBPt03FndiImINykbcB8GdK5cCngaUoDdyWBeahJLc6vqanJJo6vl6hJLhupVQ5PaMqJxER6UtdDIXbLzMfVGJJRERERKbGgK9Y6m/u/iawdUTsDbzh7gm8XH3d1LhuREwPLAasQplJ4LOURNPSlCqnAyhVTq9GxCuMr3K6D7idUuX0Zn/sl4iItIT/GQonIiIiIjI1WqpiqZlExKcpCabVKT1RPk1JOnX821WV09OUKqcHUJWTiIhMhqpiabrOiSVVLImIiIjI1FBiqck0VDmtRJlVoKPKqSPhNDdVlRMl2fQqpVqqo8rpUVU5iYhIZ0osiYiINI+I+CplZvL/NHx9AFzi7o/VGZvI5NJQuCbj7h8Cj1Zf5zU+V1U5LUWpclqO0stpnmrZNsAMwGsNw+peZcIqp6dV5SQiIiIiIlK7fwNfBeZtWPYqpY+vyICiiqVBICKmo1Q5rUipclqI8RVOHV/vMb7CqaOX0/2Mr3J6o/8jF2ldEbEl8Ii7P153LCIAZrYC8OPM3LLuWERERFpBRPyc0ne3o+DjQWAtzSYuA40SS4NcRHyK8VVOK/C/vZxmZMI+Tq8CzwC3URJPT7u7Gr2K9LKIuAwYCqzn7q/WHI60ODM7DRgGHJKZ19Qdj4iISCuIiCHAGGAtSgXTjJQCgIuBH7n7+/VFJ9JzSiy1qKrKaVHGVzktzIRVTnMD7/O/vZweoCSdHnX31/s/cpHBISLuAr5IqRpc390/qDkkEREREelnEbEQcDNwC3AKcBxlVvFngT8A7e7+UX0RikyaEkvyPyJibmBJSsJpecq438ZhdTMBrzHh0LpnKBfI9wFPqcpJZOIiYg5KqfNC1aIrgC3dfVx9UYmIiIhIHSJic2Csu79UPd4UOJJyLfY4MAoYpXNFaVZKLEmPRcS0lCqnlShD6xZh/LC6uavvP2DChNMrjO/l9Ii7v9b/kYs0l4j4MqUxY8d4+o+AM939O/VFJSIiIiLNIiIM2An4HqW1ySNAO3CBu+siXpqKEkvSKxqqnDp6Oc3DhEPrOqqcXmXiVU4q8ZSWEBG/AnYG3gXGAT8DLnL3f9camIiIiIg0lYiYBtgb2BP4HPA34Gh3V09EaRpKLEmfaqhyWoEytG4R/nfGug+YMOH0MuWAeZa7v1xD2CJ9KiJWAN4BdqecKKzi7k/VG5WIiIiINKuqR+7BwNco7RTuBg519ztqDUwEJZakRhExF6XKaTXK8Lp5GD+0bh5gd3e/qL4IRfpWRCwN3Akc4+4/rTseERGRZlFVaRwBrAdsqP6dIkVEzASMBLalXDfdARzk7g/XGZe0NiWWpOlUVU5DgWc0PE4Gs2rs/IPA0+6+ed3xSGuLiFmAxdz9wbpjEZHWFhGLA2dRKt7vAjZ39/fqjUqkuVSTwRwPbEZpO3ILcKC7P1NrYNKSlFgSEalRRFxOGS66nBoxSp0iYkvgVGBtd3+i7nhEpPVUVUpHAnsAnwC/pky1rr+PIhMREfMCJwAbVIv+Qqlgeqm+qKTVTFN3ACK9zcwuNbMxZvaimT1WfX/MRNYdamZj+jlEkUY3USr0lqk5DpHNgQWAP1ZDlUVE+k1EfAG4GTgI+Dslyf1zJZVEuufuL7n71ymTKN0ObAKMjYjfVFVNIn1OiSUZdDJzq8wcDlwN/Dwzh2fm4TWHJTIxAXxIacQoUqfFqn+XBy6NiOnrDEZEWkNETBMRPwL+CnwGOIrSU+mf9UYmMrC4+zPuvh2wLvAIsBNwd0T8tOrLJNJnlFiSlmBm05jZT83sZjMba2a7dbHOvGZ2q5l9zsweN7NPVcsXMrMLq++/V61zp5kd2t/7IYNPNQ7+aWCVmkORFlb1+1qwYdFalKSniEifiYglKVVKBwKPAWu5+4mqUhKZcu7+sLtvCmwKPEeZgfieiPhhNbOcSK9TYklaxR7AZzNzbcr448PM7L9Dj8xsFuACYP/MfAK4GPhq9fQuwLlmthGwFeUuwNrAZma2ST/ugwxeTwGLVI3rReqwOKVS4G3gA0p/hlurhJOISK+KiCERcQzlWPMZSl+lDd39+XojExk83P12d18P2B54AzgUuCsi9qn6mYn0Gv1ASavYBLgIIDPfBq4ENq6emxb4HTALcG+17FzGD036CnBFtY1LMvPjzPyw2t6m/RK9DHZXAQsDq9YdiLS0k4BvV99fpN4mItIXImIpSpXSAZThOmu5+0k63oj0DXe/BlgT+CbwEfAzSg+mnXUDSXqLEkvSSjofOD+p/l0N+D1lbP/XATLzb8AcZrY98FCVSOpuGyJT41LgTWDnugOR1uTuf3f3kcCFwLOMT7yLiPSKqkrpOOB6YH7gMGAjVSmJ9D13T3c/H/gS8H1gOuB04LaI0I1ymWpKLEmruBbYFv477G1j4Jrqudsy8xJK9v4gM5uhWn4ucBol6dSxjS3NbFozm7ba3uX9FL8MYu7+MvAMsGzdsUhrc/ePKT+Li9Ydi4gMHhGxLHAL8F3gYWBNd/+FqpRE+pe7j3P3U4AvAscAcwAXRMSYiFi93uhkIFNiSVrFmcDrZnYbJaE0MjMfq54bB5CZr1GGt+1bLQ/gjcy8tXr+Gqq+I8AY4MLMvKHf9kAGuycofZY0a4fU7X5g4Yj4bN2BiMjAVlUp/YRyc25eSo+Xjd39hXojE2lt7v6Ru/+EMnnMKZReZ1dFxNVVIlhkslimbhSIdMXMtgZWzMyRdccig19EbA+cA2zl7lfXHY+0rohYiVJZcKi7/6LueERkYKouTs+gVOPeBuymhJJIc4qI2YHjgc0ofWdvAQ5096frjEsGDs1AJNIFM7sMmB3Ypu5YpGVcA7yRrNAGAAAgAElEQVQEOKDEktTpAUqfpQ0AJZZEZLJExBDgOErfyveBHwCnaNibSPNy97eAvSNiHuAEYENK/6W/At9z95dqDVCaniqWRESaRETcDJi7r1V3LNLaIuIqYAF3X7HuWERk4IiI5SkNgZdmfJXSi/VGJSKTKyIWoswWuy7wHmUG40Pc/Y1aA5OmpR5LIiLN4zFgaETMUXcg0vLuofT8UhNvEZmkiJg2In5Oqb79FHAwsImSSiIDk7v/0923pySW/h+wA3B3RPw8ImauNzppRkosiYg0jwDmATTtq9TtPMpw+e3rDkREmltErECpTtqL0vx/TXf/jYa+iQx87v6wu28GfJkyTH4vSoLp8IiYvt7opJkosSQi0jxuBJ4Dtq47EGl5D1NOINepOxARaU5VldIJlL6AcwNtwGbu/u96IxOR3ubuY919fWA74A3gEEqCab+qr5q0OPVYEhFpIlWTxFndfdW6Y5HWFhFXAAsDy6vyQEQaRcSKwGnAUsDNwAg19xVpDRFhwFeB71P6qT1K6cd0js4XWpcqlkREmstDlN4289UdiLS8OyiJpcXrDkREmkNETBcRJzK+Sukgd99MSSWR1uHu6e4BfAk4CBgCjAJuj4jNag1OaqPEkohIczkHmB3Ypu5ApOVdUP27Q61RiEhTiIiVKb2U9qQ0+F/d3U+rNyoRqYu7j3P3U4FhwI+BWYELIuLGiFiz3uikv01bdwAiIjKBu4FnKE0ST605Fmltj1N+FlevOxARqU9ETAf8DNgJeBs4wN3PrDcqEWkW7v4RcHxEnAwcSZn444qIuBNoc/cHaw1Q+oV6LImINJmIuBqYz91XqjsWaW0RcRnwOWBZ9U0QaT0RsQrlJseSlAkmRrj7K/VGJSLNLCJmB44DNqdUMd0CHOjuT9UamPQpDYUTEWk+dwNDI2KxugORlnczpc/S0nUHIiL9p+ql9H/AlcAcwHfdfXMllURkUtz9LXffF/gicAVlqNxtEXGOeogOXkosiYg0n3MpQ5W97kCk5V0IfEwZAiMiLSAiVgVuB74BjAXWcPez6o1KRAYad3/F3XejNPm+BdgIGBsRoyJiznqjk96moXAiIk2mmsb1b8Az7r553fFIa4uI+4EX3X2TumMRkb4TEdMDJ1BuarwBHOfuZ9cblYgMFhGxBHAisAbwKnApcIS7v1drYNIrlFgSEWlCEfFnYFHU20ZqFhGXAEsAy7j7uLrjEZHeFxFfAn4NfAEYA+zu7q/WGpSIDEoRMQw4njJE7nngPOB4d/+w1sBkqmgonIhIcxoDLAIsW3McIn+l9Flaoe5ARKR3RcT0EfEr4PJ//etfq4wePfr5HXbYYc4ddtjhBjP7kZlN8lrBzA6c3Pc1sxXN7JfV98ub2QZTEL6IDEDufpe7bwBsC7wG/AC4OyL2j4gh9UYnU0qJJRGR5nQh8B/ga3UHIi3vj8D7qM+SyKASEatTeintCtx2+OGHc9VVVy2XmesAKwOLAUf0YFP7T+57Z+b9mblf9XBlYO3J3YaIDGzufj2wFrAb5Zz3OEoPpl2rthAygCixJCLShNz9GeAZygm3SG3c/TnKz+KKdcciIlMvImaIiFOAy4BZgL3dfet33333v+tk5sfAd4H9zGxmADM7wszGmNmdZna0mc1kZmOA+avlS5jZSDM73MxuNLPdzGxmMzujenynmW1SbWto9ZptgEOAEWamBuEiLcbd090vojT4PgAYAvwGuD0i1Gd0AFFiSUSkeT0NDI2IaesORFre08Ai+lkUGdgiYg1KldIulFmaVnf3P3S1bma+BvwTWMLMdgHmyczhwGrAcsC61eMXM3N4Zj5WvXQnYNvMPBv4IfBSZq4LbAOcZmbzNLzHnyi9VkZn5u69vsMiMiC4+zh3Px34InA0MBtwXkTcFBGqaBwAdIIoItK8rgA2o9zFubXmWKS1XUP5WRxGuSgVkQEkImYATga2p8zGtJe7n9uDl04LjAO2BJasKpQAZgeWBK7u4jXXZWZH4+9NgG8CZOa/zOweytCX+6ZwV0RkEHP3j4GfRcQvgMOBrwKXR8RY4Pvu/kCtAcpEqWJJRKR5XUaZ8nnnugORlncp8BawQ92BiMjkiYg1gTsoPftuplQpTTKpZGZzAwsAj1OGpxxTVSYNz8yVM/Pkibz0nc6b6vT4k8naARFpOe7+gbsfTqlg+h2wOPCXiLg0IharNzrpihJLIiJNyt1fpvS2WabuWKS1ufu/gWcpw19EZACoeimNoiSGZwK+5e7buvvrk3qtmU1LqXAalZnvUSqTdjKzIdXzo81srWr16TqWd+FaYLvqNfMBy1OG4DV6D5h78vZORFqBu7/t7t+lJJguB1YBbo2IcyNi/nqjk0ZKLImINLcnKL1tZqo7EGl5T1F+FqevOxAR6V7Vk+QOYEdgDLCau5/fg5f+xcxuBh6lTAN+VLX8TOAx4DYzuxV4LDM7EkRnAfea2V5dbO9YYNFqmxcC36h6NzW6FhhmZrea2aI93kkRaRnu/qq7705pD3ELsAFlBrnTI2KueqMTAMvMumMQEZGJiIjtgXOArd39qrrjkdYVEbsCpwEbu/tNdccjIv8rImYEfgFsC7wMHOnuF9YblYhI74qILwAnAmtQkuCXAUe4+7vdvlD6jCqWRESa2zXAS4DXHYi0vCuA1ykVECLSZCJiXUqV0g6Mr1JSUklEBh13/7u7bw5sSGkb8U3gnogYqcrqeqhiSUSkyUXEzcA07r5m3bFIa4uIO4AP3H143bGISFFVKf2SUqX0b0qV0kX1RiUi0n8iYn3gR8CKlKH7ZwK/dHdNFtBPVLEkItL8HgOGRsScdQciLe9JYOHqQlZEahYRwylVSg78hTLjm5JKItJS3P2vwNrAbsD7lP5ud0XEiIjoPDOl9AEllkREmt/5wKeBzeoORFrepcCClJM3EalJRMwUEWcCFwPTAd/w4s2aQxMRqYW7p7tfTGnwvT9gwK+BOyJiq1qDawFKLImINL+bgeeALesORFre1cArwFfrDkSkVUXEBpQqpe2A6ylVShfXG5WISHOoEkxnAMMoM1vOAvwhIm6KiHXqjW7wUo8lEZEBICL+Cszm7sPqjkVaW0TcBnzs7jo5E+lHETEz5e77lsCLwA/d/ZJ6o5JWYmYJ3ESpBJkTuAQYmZnjzGwTYJXMPLbOGEU6i4gZgCMoN8XmB8YC33f3+2sNbJBRYklEZACIiJOBnYHl3f3FuuOR1hURvwPWBZZy9/fqjkekFUTEhpSptRcGrgK+5e5v1xuVtJoqsTRdZn5sZtMCo4HHM/PoeiMTmbSImJXSe2kLYA7gVuBAd3+i1sAGCQ2FExEZGM4FZqPM+iNSpz9S7vitX3cgIoNdRMwcEaOBoJy37+ruOympJHXLzI+B7wL7mdnMZjbczEYDmNloMzvAzK4zs7+Z2U7V8vnM7DIzu8HM7jGzkzq2Z2ZPm9kPzOzWWnZIBj13f8fd96cMkbsUWBm4NSLOi4gF6o1u4FNiSURkYLgHeAbYqO5ApOVdT5nSfPu6AxEZzCJiY0ovpa0p/c1Wd/fL6o1KZLzMfA34J7BEF0+vBGxMqXA9rlq2NPDrzFwP+CKwkpktVj03L/BRZq7Zt1FLq3P3V939G8CqwI2UG2V3RsQZETF3vdENXBoKJyIyQETE1ZRKkZXcXQdvqU1E3AyYu69Vdywig01EzAL8Btgc+BdwiLv/ud6oRCYcCtew7EFgF2AuYERmjqgqlyIzr6zWeTIzFzOzGSnNlNcGPgGWBLbPzJurbc+VmW/0715Jq4uIxSlDjdcEXgcuA45w93dqDWyAUcWSiMjAcTewCLBo3YFIy3scWCQiZqs7EJHBJCI2AW6n9AC5Clitu6RSRCwcEddFxOf7K0aRDmY2N7AA5W9CZ409+DquOX8MzAdslJnrArcAQxrW04W89Dt3f9zdtwA2AJ4C9gDujoijq8bf0gPT1h2AiIj02DmUfgY7Mr6sXKQOFwJfowxz0DTnIlOpqlI6lVKl9Bywi7tf0c36BhwC7EU5n18Z+Ec/hCoCQNW8+2RgVGa+Z2Y9ednngEsy830zWwIYDvyy76IU6Tl3vw/YMCLWA34EHARsGxFnAf/n7p/UGmCTU8WSiMjA8Silz5L6D0jdxgAvUHq/iMhUiIivUHopbQ78mVKl1F1SaTHK7+APgaeBddw9+j5SEQD+YmY3U85JXqMMbeupHwP7m9mNwA+ASyhD6ESahrvfAKxDGeL5PnAMpYJp9yqpL11QjyURkQEkIi6j3PFbzt3H1R2PtK6IuBGY3t1XrzsWkYGomvr6VOArlAbIP3D3q7pZfxrgcGBPYFz12uPVc09EpG9UiaTdgf2ALwAPAzu4+5O1BtaEVLEkIjKwjKH0WVqu5jhEHgUWjgjdbRaZTBGxOaVKaTPgcsqMb90llZYAbgIOpgx5W8fdf6KkkohI33H3dPffAsOAIyn5EzWY74J6LImIDCwXUYY/7Aw8UHMs0toCGAFsCpxbbygiA0NVpXQa5ffmn8BO7n5NN+sPofT6GAF8SBl2dKISStIKImJmygX9XA1fcwAvuPupdcYmrcXdPwZOqL6kC0osiYgMIO7+bEQ8Q2nUKlKnWyhNhrdEiSWRSYqILYCfAJ+hTGf9HXd/r5v1lwVOp1So3gHs5u7P90esIk1iRsrEJQt2Wn42ZSioiDQJDYUTERl4nqRM9a6bA1Ibd/8PpZn8onXHItLMImK2iDgP+D2lN9IO7r7bxJJKETFtRPwMuBaYFzgU2EhJJWk17v4asA2luq/DO8DDVTWfiDQJJZZERAaeq4CFgdXqDkRa3sOUJOc8dQci0owiYitKtdGXgT9RZny7rpv1VwRuA/YG/gas6e6/1NA3aVXufhdwNPB6teh1YCRwT0SMjIhZ6opNRMZTYklEZOC5lHJitXPdgUjLOw+YE9ii7kBEmklEzB4R51OG7HwMfNXdd++mSmm6iDgJuJrSR+YgYFN3f7HfghZpUu5+JmVI3BvA9pQZut4D2oD7IuL0iFigxhBFWp6GUYiIDDDu/krVZ2npumORljeWMkRhM+C3Ncci0hQiYhvgWGB+4GJgX3d/v5v1VwVOAZYAbgR2d/eX+yNWkQFkf+Bxdx9L+dtzZkSsBxxCGS73lYgYC4x09/trjFOkJVmmKmtFRAaaiDgXWB1YursLFpG+FhHXA3O6+xfrjkWkThExO6XZ9ibA08CB7v7XbtafHjgRcEoV6k/cfXTfRyoyuETE4sCPKS0C5qDMmnsScJmGkYr0Dw2FExEZmC6mzCy0ft2BSMt7EFhYwxCklUXE9pReShsCF1J6KXWXVFqzWn9XSk+l1ZVUEpky7v64u+9ImTH3bErT+3OBsRFxQJXEFZE+pKFwIiID0zXAv4GvAlfUHIu0tnOBb1OGIpxScywi/Soi5gDOADYGngK2dfcbu1l/RuD/gG2BV4G93P3c/ohVZLCrZpE7ICJmAPYFdgKOA/aMiKuBY9z99e62IdJfzOxYYC9gvsz8uA/fZ2Hgi5n5x+rxycDvM/Oe3nwfVSyJiAxA7v4OZajF4jWHInIv8CywUd2BiPSniHBK1dH6QFCqlLpLKg2v1t8RGFOtr6SSSC9z9/+4+wnAMOBrwMvAHpRG3+dGxGK1BihSdAybXm9SK5rZimZ2s5lNSfXdYsCWHQ8y84DeTiqBeiyJiAxYEXE6sCmwrLu/UXc80roi4hpgHndfue5YRPpaRMxJqVLaCHgS+K6739zN+jMBvwa2olSaHuHuF/dHrCJSRMQqwJGUZNN0wD3Ace5+U62BSUsys2UofcHuBBbNzL3MbAzwB0oV+DeABYEfAR8BiwDLZebrZrZG9drpgLeAb2fm89Xrz6P07fsU8D3gfeA3lMkk7svMTc1sNDA6M8eY2f2UkQ9rAUOAHTPzOTPbEvg+MA6YDfh+Zv6lu31SxZKIyMB1ATAPZUYukTrdCywSEQvXHYhIX4qIHYHbgeHA+ZTeSN0llTamXDhsC1xLqVJSUkmkn7n7Pe6+FSWxdDnwOeCKiLgtInaLiCH1RigtZjvgsuprGzPryMusB6yWmQ8Co4GvZeaalITTlmY2J/BLYLvMXIcyYcQJDdtdMDM3oAwDPTYzbwcOAK7OzE27iGMZ4KbMXBc4B9inWr40sEW1fHvgqEntkHosiYgMXDcDz1HKWzWcQup0PvBdyonSSTXHItLrImIu4ExgA0qV0tbufks3688CnApsTjlOf93dL++PWEVk4tz9eeAbETEr8APKOdRvgAMj4hLg5+7+bp0xSkvYGtg4M18xs7eAtavlF2bmJ9X3L1NmOaTh3zWAhYBLzAxKoVBjsdD5AJn5iJnN24M4Xs/Ma6rvHwE6Zvi9EjjNzBYEPqZUT3VLFUsiIgOUu/8HeIZy102kTg9SfhaH1xyHSK+LiJ0oVUprU5L4q00iqbQ5pUppc+DPlKomJZVEmoi7v+PuR1BmktsPeA9oo/RhOl0znUpfMbPPAwsDfzCzqynFPttXT7/TsOrFwKXVELc3gd9ThqvdnZnDq691MnOthte81/B9T6rw3uv0eIiZDQGuBn5bVUtt3pNtqWJJRGRgewDYNSLmdfeX6g5GWpO7j4uIZ4ChEWHurgaOMuBVVUpnUZpz/wP4hrvf1s36swGnUXrfPQPs6O7XTGx9Eamfu39CqUY8MyLWAw6hzHL6lYgYC4x09/vrjFEGne2AtswcDWBmqwJ/BJ7oWKEaGrcrsFBmjmtYfjtwipkNzcynzWxTYLPM3K+b93sPmHsy4pu1+uroP/btnrxIiSURkYHtp8CNwCt1ByItbyywP2X2kScmsa5IU4uInSk9Jeam9LY4sKoSndj621Gaqc5HuUDY19073wkWkSbm7jcAN0TE4pTf59Wqxw9Qhnlfphsn0gu2oFQBAZCZY83sDWD1hmXjzOw54E4ze5fShPuvQDuwO3C+mX0AvAF8axLvdx8wvZndQem31K3MfNPMjgHuMLOXgd8BM0zqdZoVTkRERKZaRCxFSS4d6+7H1x2PyJSIiLmB31KqlB6nJIhu72b9zjPEHejuY/ohVBHpY9Xx4EhgY8qsXA9TGhyf4u4f1hmbDG5mthHwjczcqXo8K3ADsGdmNmUFnRJLIiIiMtUiwii9lp5y9y3qjkdkckXE14HDgbmAAL7X3cVj1XvpSMrsnBcD+7v7B/0Rq4j0n4iYAdiXMtPW0sBTlB40x7j763XGJoOTmX2KcpPj05Tm2TMCdwEHZObHdcY2MUosiYiISK+IiD9T7uour+ECMlBExKcovZSGU6qU9nb3O7tZv7Gq6R/Afu5+az+EKiI1qm6gbE0ZTrQCZRjSbcDh7v5knbGJ1E2JJREREekVEXEYpfHpqu7+aN3xiExKROwGHEapUjofaJtElVLj+pOsahKRwSkiVqFULA4DpgPuAY5z95u6faHIIKXEkoiIiPSKiPg8cDdwsruPrDkckYmKiE9TqpTWBf5OqVIa28368wCjgXWAxya1voi0hohYEPgR5dgwP2VI+CjgD9WMcyItQYklERER6RXVMIH7gefdfbO64xHpSkR8g1JZNwfjq5Q+msi6Rplxpw2YDTgPOHhi64tIa4qIWYGDga2AxSkJ60uAn7v7u3XGJtIflFgSERGRXhMRlwGfB5Z193F1xyPSoapSGk2pUnoU+I67393N+vNX669JmQ3qW+7+QN9HKiIDVUQMAUYAewDLA/8CxgBHufsL9UUm0reUWBIREZFeExHfowwLWN3dH6w7HhGAiNgD+AEwO5OoOqqqlPYD9gdmAv4AHObuTTkTj4g0p4gYDhwKrAJ8CIwFRrp7U04XLzI1lFgSERGRXhMRiwD3AaPc/dC645HW1qk30qPAt9393m7WXxA4G1iN0itlT3d/qB9CFZFBKiIWB35MOa7MATwAnARcphlUZbBQYklERER6VUTcB7zk7l+uOxZpXRGxJ6XnySR7I1VVSt8D9gWmpzT2PkrNd0Wkt0TE3JSZ5DYGFqEMsT0HOEWzS8pAp8SSiIiI9KqI+BOwFLCMLsylv0XEvJSqo7WARyi9kSY69KSqshtNmTb8fmAPd3+sH0IVkRYUETNQktg7AUsDTwFXA8e4++t1xiYypZRYEhEZRMzsUkqZ9ZLAm8ALwC2ZeXgfvucmwCqZeewUvHY0MDozx/R2XFKfiNgH+DmwtrvfU3c80hqqqqNvAwdRqpT+ABwysd5I1fqHVq+ZBjgD+LGazotIf6iOQVsDBwArAG8AtwGHu/uTdcYmMrmUWBIRGYSqhM0tmXnGZLzmMGD6zBzZSzF8lnJht2tmPttNnEosDTIR8RlKD4nfuftBdccjg19EzEepUlqTUqW0Z3czuEXE5yjD3VYG7gV2d/cn+iNWEZHOImIVyjC5YcB0wD3Ace5+U62BifSQEksiIoNQ58SSmR0BbECZ4ejqzDzKzAw4gTJcZAjwaGZ+zcxGAEM7EkyNyR8z2ww4ijK7yV2Z+T0zGw6MyMwRZvZF4BfAJ8BCwNaZeX/1/l8GrHpu58x8rtO276DcqRtWrfP1zPynmW0JfB8YR6lC+H5m/qXPPjzpFRFxN/C6u29UdywyeDVUKbUBs1CS2Yd2U6U0DXAE8E3KMeVU4Hg10BWRZlBNIPAjyoQD81MmERgF/EFDy6WZTVt3ACIi0rfMbBdgnswcXiWTLq6GrwGsCnwJmAG418xm6mY7n6Kc3AzLzBfN7HQz2xx4p2G1XwFHZOZ1ZrYXsJWZPUFJFK2dmWlm3wC+Rbkz12hl4KAqWfVN4JeUEvGlgS0y8w0zW4zSC0WJpeb3NLB8REyradqlL0TE/JQqpTUoTXC3m0SV0pLAmZQhJ2OBEe7eZTWliEgd3P15YI+ImJUy+cBWwG+AAyPiEuDn7v5unTGKdEWJJRGRwW9LYEkzG1M9np3Sg+kWSkJpemDm6t/urAHcm5kvVo8PAP5DqXjq8DKlxxPAnACZ+baZPQJcZWazVO9/dxfbfyEzb62+D+DE6vsrgdPMbEHgY2DBScQpzeEqYHNK8vK2mmORQaSqUtqbMovbTMCvKT1JJlalNIRSATCCcsw6EjhJVUoi0qzc/R3gyIg4mnLs2oNSmblzRIyhzFr5Qn0RikxIiSURkcFvCHBMZkbjwirJMwPlzv1LwD6Z+b6ZjaMMWeswY/XvNJ2WW7Ws0f9n777j5KyqP45/DimEEiBBOpggHaRXqUGQANIieEHFH6EJigLqUKQGEEEcRBHpJXS4iCI1KEhogtQQmgJC6EgntEBIzu+Pc5cMw+4mbHZ3tnzfr9e8svPUOwPzlPOce+6VwGlmti+RQfAzM1sJOAnY1N2fMrOtge2baafVvXcz60OMlLKbu48xs4FEWrh0fdcAxxKj3iiwJO0i57wQ07KUHiFqKbV4TMg5fxU4C1gRuAvYJaX0Ume0VURkZpXub+cA5+SchxEDDnwL2CrnfA8wqrVRL0U6iwJLIiI93xjgO2Z2pbtPKXWNziZqIN3g7gfVLf8csDVA6Xq2MVGH5C7gVDObz91fK9MuAibVrHsgsLi7f9o9zsyGAk+XoFI/YFdgYjPtXNTM1itZS9sBY4E5y6upeOVebfoGpNOllF7NOT8HrNDotkj3V7KUfkxkSs5GdJU9rKWaIznnvsBxwM7AB8DBwB+VpSQi3VVKaSwwNue8FHAMsA5wS855HPA74God46RRFFgSEen5zgGWAP5pZp8A17r7HWa2NHCyma0LTAbeJDKLbgV2N7N/Ai8BfwZw91dLJtJ1ZjYZuL1kEQ2r2dd9ZT9vEQGne4FfA98zs3uIoXT/DGzRTDufB5KZHU/UZNrF3d8xs18Cd5vZa8AFRJaVdA/PAGvknPunlD5udGOkeypZShcAXyMyFvdIKT3ayvKrAGcCywF3ErWUXmlpeRGR7iSl9CSwU855XmIwgs2AS4DHcs4XA6fqnCudTaPCiYj0UmZ2MXCBu99YinqvCZzp7qu0cXtLA+cyrUj3AOBS4Hx3v2oG1p/g7kPbsm/pmnLO3yP+nxhenrSKzLCSpbQvsB/RJfc84IhWspT6Ab8BdgLeBU5IKZ3VSc0VEWmInPOsREbnd4gBT54BbgSOSSm91ci2Se+hwJKISC9lZlsARxKZRX2JGkcnuvuf27i9AUSWwFJEgdwBwFPAj9y9ua5v9esrsNTD5JwHAY8S6fl7N7o90n3knBcmspTWAcYDu6eUHm9l+bWAU4FliG60u6WUXuuEpoqIdAklGL8d0WV4ZSJL/J9Et+GnG9k26fkUWBIREZEOk3O+C/gopTSs0W2Rrq/cGO1PZCr1J7KUjmwlS6k/0YX328BbwK9SSud3UnNFRLqknPPqRDe5tYB+wP3E8fG2VlcUaSMFlkRERKTDlHoP6wLLp5Q+bHR7pOvKOS9CZCmtDTxEZCn9u5Xl1yOKeC8J/IPIUnqzM9oqItIdlOPq0cCGwIJEnbozgItaCtiLtIUCSyIiItJhcs47ABcD26SUbmx0e6TrKVlKPyNqhPQj6nKNSilNbWH5AURAaQTwGnB0SunSTmquiEi3k3Oekxi5dxtgaeAJ4CrgNyml9xvZNukZFFgSERGRDpNzngt4DLgxpbR7o9sjXUvOeTHgfGLwgPFE1tF/Wln+68BvgcWBvwF7ppTe7oy2ioh0dznnPsAuwB7ASsTov2OJLscvN7Bp0s0psCQiIiIdKud8JzAlpbRho9siXUPJUvo5kaXUFzgHOKqVLKXZgT8ST9tfIUaHu7KTmisi0qOUY/Aw4GBgdeBj4B4iW3RcA5sm3ZQCSyIiItKhcs7nAxsDyynlXnLOX2ZaltKDRJbSk60svzlwArAYcAOwV0rp3c5oq4hIT5dzXgo4hhiFc25gHPA7YkRXBQtkhiiwJCIiIh0q57w18Cfg2ymlqxvdHmmM8oT8AGAfYBbgLAnWDDoAACAASURBVOCXrWQpzQmcDnwTeAH4RUrp2k5qrohIr5JznpcYSW4zYAjRjf1i4NSU0seNbJt0fQos9SBmNgQ4BVgIcODP7n5cB+5vFDDB3Ud31D5ERKT7yznPQVygjk0p7dLo9kjnyzkPIbKU1iCylHZNKT3VyvJbA8cBiwDXAD9UtpuISMfLOc9KdFP+DrA88AxwI3BMSumtRrZNui4FlnoIM5sFeAA42d3PNbP+RLr42e7eISOlKLAkIiIzKud8G9AnpbReo9sinadkKR0E/BAwIkvp2FaylOYCzgQ2B54Dfp5S+nsnNVdERIpy/N4O2B9YGXgb+CdwWErp6Ua2TboeBZZ6CDMbDvzB3ZeumbYasBywBVGXYAARcf6uu081s3HEkL4jgIHAru7+sJl9DTgemAoMAk5w90vKNo9h2sXeu8BYdx9tZocDw4mLxillHy90wkcXEZFuIOd8NnH+WCGl9E6j2yMdL+c8FBhNZCk9QGQp/beV5XcAjgYWIIbB/klK6YOOb6mIiLQm57w60U1uLaAfcB9wXErptoY2TLqMvo1ugLSb5YBHaie4+wNmNgWY2pS1ZGYXECMA/IMIJn3o7hub2RbAIUTK40rASHd/xszmAu4GLjGzzcu6XyNqI9wKjDWzgUQwaQN3dzPbDfgBcEQHf2YREek+MvB9onbDFQ1ui7SznPPApoLa5Sn3wcDexAOn44gbkJaylOYhRoXbFHga2D6lNLYz2i0iItOXUrof2C7nvAjxAGBD4Lqc88PAGcBFKaUpjWyjNJYCSz3HJ0SGUb3xwBZmdhtRd2ko0JRS3ocoyAbwODB/+XsMcJSZLU0MPTm0TP8GUbfpEwAzuwnA3d81s8eBG/r16zf3PPPMM3SppZbynPORGklARESK24CXiLR6BZZ6kJzzbMBDOeddiCLb5wOrAfcTWUotdpnIOX+XeBA1L3ApsH9KaVLHt1pERL6olNKLwO5lcIUDgW2A04Cf5pyvAn6jeni9kwJLPcd44Ce1E8xsbWLYyI2Abdz9bTM7kQgoAeDutSnmTdP/RNQ32K10mXujTJ+lbp8GsMoqqwybffbZzx01atR/hgwZssj999+/4K233vqmgkoiItIkpTQp5/wssESj2yLt7nhgcaJ+0uzEg6xjgeNbuhbIOQ8GzgM2Bp4Cdksp/bNzmisiIjMjpfQecETO+Wjg/4A9gZ8D3805jwWOTCm93MAmSidTYKmHcPfbzGyime3u7ueY2azAr4gLvdElqLQA8aT44elsbgngxhJU+g7RZQ7gFmA/M/s9Ua9pS+DklVZaad+PP/54ziFDhqz9ySefcMstt9CvX7/+OefrgdfK/v4F/Bt4XQEnEZFe63Fg25zz4JTSm41ujMy80i1iu/J2GeJcv0VKaUIr6+wK/AKYG7gA+JmGshYR6X5SSp8A5+aczyNKphxM1O/9Zs75XmBUSmlcA5sonUSBpZ5le+AMM/shUfPoEqL45aVmtinwP6L7waDpbGc/4Doze43oFvekmQ1y96vNbH2i5tLbwB0AF1544Y7zzTffzQcccMAa88wzz2xrrbUW99xzD0TXuuWIC87ZgbeAN3LObwJvAK8SxTzvAZ5UMVcRkR7vcmA34JvAhQ1ui7SP04Av17wfDCwKTKhfMOc8H9FNbgPgP8D3Ukr3dkIbRUSkA5XEgVuAW3LOSxF1mL5W3o8DfgdcrQSDnkujwkm7yTnPTVwwbgKcmlI6KOfcDxgCrEh0y/sKcdE5b/l3MDGyQFOwqenfl4mA0wPAUxoVRkSk+8s59weOBE5LKWnk0G4u57wOcA3wTnm9Xf79U0rpkprljBjUowLMSTz4OjilNLnTGy0iIp0i5zwvMZLcZsT94GNEfd9TlaXa8yiwJO0u57wH8HBK6V/TWW4A0e1uFWLoysWIgNMgpgWenM8HnZ4jsqbGA0/rwCQiItL5cs5zEYGiV1oZ8W1B4qHTusRNxQ9SSg91XitFRKSRcs6zAj8mRh9fjshovRE4JqX0VgObJu1IgSXpknLOA4EliVFl1gQW4vOZTh/x2aDTm8CTRNDpUeBZDXspIiLS+UqW0k+I7vWzEV0fDy31OEREpJcp54XtgP2BlYks138Ch7U2eqh0DwosSbdTRpJZmshyWoWo5VQbdJoHeJ/PZzo9RgSdHgdeUh9fERGR9lcKep9PdIF/GNgjpfRoY1slIiJdRc55daKb3FpEWZT7gONSSrc1tGHSZgosSY9RouALECmWawFfBb7EZ7OcBgIT+WzA6XU0cp2IiMhMKefhCvAj4kZhNDHktLKHRUTkc8qDiKOBDYEFiXuyM4CLdO7oXhRYkl4h5zwLMUrNCsDaxJDI8/LZoFPTyHW1mU7/QyPXiYiItCrnPITIUloDGAfsllJ6orGtEhGR7iDnPCdwILAN0TPlCWJ089+klN5vZNtkxiiwJL1eGbluKJHh9DVgcT5fRLxp5LraoNNLRJbTg2jkOhER6YXKg5uDgb0AA84Cjm2pmLeIiEhLcs59gf8D9iRGFf9TSmlkQxslM0SBJZFWtDJyXVOW07zAVD4fdHoWjVwnIiI9WM55SeBcYqCN+4kspf82tlUiItLdla7VGxE9Rl5sdHtk+hRYEmmjmpHr1gBWZ8ZGrnsDeIoIOj0CPKf+wyIi0tXlnPs0na9KltIRwO7Ew5VTgRNUn1BERKR3UmBJpAPknOcl+gevybSR62oDThq5TkREuoWc86LALURm0iJEltJKRP3BkSml5xrYPBEREWkwBZZEOlHdyHVrM23kusE1r7mAd/hs0Ok1olvd3UQxO41cJyLSw1TNLgJuqbifU96fDEysuB9W3v+I0gW74n5sZ7Ur53w1sDUw7pM331zwrbPOGjjfQQcdAZykc5GIiIgosCTSRZSuBYsByxNFxJfm80XEWxq57n7gXjRynYhIt1U12x3YsOK+S3k/Hvi44r5GeX8xcHLF/V+d1aac8yZAJs5BEOea7VJKL3VWG0RERKRrU2BJpBtoZeS62iLifYG3geNTSic1pqUiItJWVbMlgJsr7kOrZgsB5wErAKtV3F+rmj0K7A98r+I+smp2APB9op7ff4HdK+7vV822BI4EPgburbj/rGo2C3AcsC4wK/DHivv5VbOhwCnAC8Q55j0gAe9idmLfBRfcywYMmH3WpZZi0K67YrPM8vBzO+00d2Xq1CFVs5GlfUsDiwJjK+4/74zvSkRERLoOBZZEurmaketWJbrXnZZSeqyxrRIRkbaomj0HbACsDywMLEvUN7oFOB/4JTCSCDC9CAyquH9cNduDqHn0IjAOWLPi/krV7Czgr8QAE8Mq7t+rmg0E7gO+RdT7ewxYq+L+SNXs18DTwL9mmWOOmxY999x+wMS3r7ii/xwbbPB8vwUXfOe5nXdevvLxxwuVwNIBwDpEQOouYJeK+386+nsSERGRrqNvoxsgIjMnpTQJeLS8Lmpwc0REZOaMJYZY3hj4AzAB2BaYDNxes9xEovbejVWzm4DbK+7jq2ZbAw9U3F8py+1PZDRdTjlHVNzfrZpdD2wG/AV4pOL+SFn+caJb9tNTJ0167/nddnuaKVNu9kmTbvjBFVc8AFDdcccJNe24ruL+LkDV7N/EYBUKLImIiPQiszS6ASIiIiLyqVuIwNIqRObR34GvE1lMtzUtVHGfSmQ1HULU3juxanYEcW1nNdszpl3v1U4HmFL+/aBuep+K+0SmTFna33//aJ806QPg0qrZLs2093PrTv8jioiISE+iwJKIfIaZLWpm15jZPWY2zsz2LdM3NLPVapb7aeNaKSI9iYVflePO7WY2xsyGNLpdDTIW2B74b8V9asX9beAZou7R3U0LVeP7OR24p+J+KvBbYGWiO9rqVbP5yqKnE4GpvxFd36iazUFkK93YUiOqZmsCR1Xcx1bcTwJGAyu138cUERHpfGZ2rJm9YWZ9y/uxFvUGZ3a7C5rZFTO7ne5KgSURqXclcLa7r0UUCh9hZiOIG5Pam4r9GtE4EemRdiTqxK3j7hsAvwEubmyTGqPi/gwxEMPNNZNvAJ6quH9Ys9yzRC2ku6tmdwB7A0dU3F8F9gWuq5rdCbxQcR8DnAO8VTX7JxFQGjWdWkgPAAOqZveUdTYCTmy3DyoiItIYmxPdzDdubSEzO8TMRs3oRt39FXf/9sw1rftS8W4R+ZSZDQOOKTd2TdPWBa4D3gEmEfU4lgeGE0/PjwReBk4lukAMAi509xPN7AIgu/u1ZnYoMIu7H2NmSwM/BSrAGUQ9jwHEU/nvenTxEJFewsz2BzYFtnX3KWXabETAaai7jyrTRhOZM9sCL7j7iWX6WcSxZGGimPRUYCBwgLvXBmhERESklzKzFYBjgH8Bi7v73mY2lhgU41niAcr6xD3Nv939e2W9w4FNgNmAMe5+ZMlyGg08BCzi7juY2QR3H1rudT53b9RZn7MRlLEkIrVWBB6sm/YwUeh/NHC8u//C3bcFXnH3Ye5+K5HJdLC7bwysCexhZrMC1wJblO2sRwSjALYinsAvCVzn7hu5+9rE0NjDOurDiUiXdSExXP3zZnaRNV/Lp9YlRNcwSir7Su5+HxH03trdNwJ2IALf3YqZuZldUzetr5m9Wi5+MbPNS7C+te1MaGH6wWb2zfZqr4iISDeyPXB1eY0ws9p4yHBgLWKU7fWAVc1sNjPbGZjP3YcRo6CuaGabl3VWB+5w9x3q9tPSvVGPpcCSiNRqruhqH2I0otb8A9jVotvF34msgfmJ7habmtmcxKhEH5vZ/MA3yjrjgSFmdpuZ3Up0tVikXT6JiHQb7v4G0RXuW8BjRDbj3UCzF2Hufi8wj5ktTqSy/6PMuh44sxyLzqN7Hk9eAtYwswVqpm1BZIYC4O5j3P3Ytmzc3Y939+vqp5vZ8PqAloiISA+zHXCtuz9OjK66Qc2814nrjv7A7OVfgG2AYeXhzi3AUGDZMm+KuzdXV6mle6Meq2+jGyAiXcojRCS/1mp8Poup3hnEENXD3H2ymd0P9HH3d8zsReCHREHavsB3iW6475nZPkQwaRt3f9vMTkQjCon0OhaFoh9297uBu83s10SAaQE+O5LZgJq/LyOylpYE/mhmfYAxwG7uPsbMBhIZl93NZOJJ6veIgtwA3wfOJy5um7otj3T3kSVYfzYwD9FdeU+P+ktYjBK3GTH623fd/cVSL2KCu482s/8j6jFNIboOrmhR2LvZLspmNg44FxhRlt/V3bvjdywiIr2MmS0JfBm4yMwg7ktqM40eJwJL9wCvAvu4+4fl+uKX7p7rtjcUeK+F3TV7b9R+n6brUcaSiNS6GZjbzLaDT2ucHAUcTwwpPbhm2Q/MrOn9EsBN5cC5AdEdpcm1wKHEiETXAkcAN9Wsd1cJKi1APEUQkd5nW+DYmpT0eYmgxmOU44mZfYXPFtq8hAhUL+/u44A5y+u2Mn+vTmh3Rzkf2AXAzOYBFieKaTfnd8AN7r4hcbxuquGwCPCgu68PXEoEkD5VMklPBjYtXZHHE3UlWuuiPBD4sKT2Hw4cMvMfVUREpFNsD1TcfXN335yo4ziCaQ+wtiHOpyu7+zfcvWnk1DHAd0qACTMbbWbrT2dfrd0b9UjKWBKRT7m7lxHgTjezw4jI+unu/ncz+y/w5xJ02oy4IbnDzDJwMHCKmb0J3E+kiQ4iRly4Fti/pJxiZq8TB2iA3wOXmtmmwP+AK8p6ItK7HAGMAu40s0nEsWcvojvtthajkr0E/LlpBXf/j5l9QukGVzIkf0lkPL0GXEALXem6One/38KqRG2Gy1tZfDiwT1nvTjN7pEx/x92burb9h6gZUesj4H1gTjN7lwgaQQSYtjCz2wAnUv7/Xub1YdpofY/Tw9P6RUSkR9maqPMKgLvfY2ZvE6NgQ9zDnFwGLpoMvAmcRIyqugTwz3Ldca2731EyllrS2r1Rj6RR4URERES6iJoRZX5OpOyvRGRmLQOMcvdhdV3h3gKWcPc3y/qD3f3Npu2UabXLj2JaV7jRwIbAc8DV7v7b0kV5K+A7NV2UHy7L125zKDC6FDMVERHp1szsYuACd7/Roq/cmsCZ7r5Kg5vWLShjSURERKTruRh4FLjb3V82s2VaWO5mouj52SX7cw9gp+lt3MwWApZz96/UzWqui7LqKImISE93EXCUmf2CiJMYcHRjm9R9KLAkIiIi0sW4+ytmdhdw4XQW3Q8418y+TxTv3m0Gd/EaMKh0M/yIqKN3JeqiLCIivZC73wDc0Oh2dFfqCiciIiLSy5jZnsBX3P0X5f28wDhgRXd/u6GNExERkW5FGUsiIiIivc/NwI5mdjtRpHtW4BwFlUREROSLUsaSiIiIiIiIiIi0ySyNboCIiIiIiIiIiHRPCiyJiIiIiIiIiEibKLAkIiIiIiIiIiJtosCSiIiIiIiIiIi0iQJLIiIiIiIiIiLSJgosiYiIiIiIiIhImyiwJCIiIiIiIiIibaLAkoiIiIiIiIiItIkCSyIiIiIiIiIi0iYKLImIiIiIiIiISJsosCQiIiIiIiIiIm2iwJKIiIiIiIiIiLSJAksiIiIiIiIiItImCiyJiIiIiIiIiEibKLAkIiIiIiIiIiJtosCSiIiIiIiIiIi0iQJLIiIiIiIiIiLSJgosiYiIiIiIiIhImyiwJCIiIiIiIiIibaLAkoiIiIiIiIiItIkCSyIiIiIiIiIi0iYKLImIiIiIiIiISJsosCQiIiIiIiIiIm2iwJKIiIiIiIiIiLSJAksiIiIiIiIiItImCiyJiIiIiIiIiEibKLAkIiIiIiIiIiJtosCSiIiIiIiIiIi0iQJLIiIiIiIiIiLSJgosiYiIiIiIiIhImyiwJCIiIiIiIiIibaLAkoiIiIiIiIiItIkCSyIiIiIiIiIi0iYKLImIiIiIiIiISJsosCQiIiIiIiIiIm2iwJJ8YWY22sy8mdfrHbS/YWX7C5b3o8xsbEfsS6S3MbOLyu9rjRlYdpiZec370e35W7TQt+b90NK2ddprHzXbntDCcczNbPRMbvv4sp3tZmDZJcuyi87MPqezj1+a2U0dtX2R7s7Mljeza8xsYnldbWbLNrpdtcxsrJmdXvP+M8dLEWksM1uxHDveMbM3zewGM1ul0e3qKGbWr+79KDP7d6PaI42nwJK01Y3ubnWvL3XEjtx9bNn+Kx2xfZHeqlwUfBN4G5huEKSeu49092Ht2KQngZ1rtj+h/Pbvbsd9NG17aNOxC/gPcFTNsWzkTG5+O9r4nYpI5zKz5YA7gYeBxYGvAOOBO81sqUa2rZa7D3P3vWsmfeZ4KSKNY2arAv8EXgSWAZYijim3mNmSjWxbRzCzDYCPa6e5+yh371IBeelcCiyJiPRew4A3gPOAEY1tCgDd/ul7uUldGPg1sLUyCkS6vN8Dt7n7Ie7+hru/7u6HAbeWeV2Vji0iXcepwL+BH7n7K+VYciDwAHBoY5vWIfo0ugHS9SiwJO3KzJYraaCvmdkHZvaQmW1ZM3+Ymd1hZlua2RMlXfSEMu+gst4rZrZ33Trewv7GmNnxddNOMLMjO+ozivQgI4Brymv5+qfz5bc33swmlS5vi9fN/0y3VDMbaGanmdnrZvaWmV1rZkvUrXOwmb1gZu+a2d/MbCkz27z8xocA55WuYUvWdIUbamabmNkUM1uobntPN3WVM7Olyz7fL/s41cwGtvXLMbMhZvYXM3uvfJ7RZjb3dFYbAdwIXAkMBjas2+bKZnZ3+U7vA1aumTfczD4xs/lrppmZPWulq6KZLWtm15fP+LyZnWJmc9Ysf4eZ7WBmZ5V9DK2Zt5eZvVyOu2eY2Wxt/W5EegIzWwT4BnBKM7P/CGxuZguUbmg/KMeUieV3tF/NdpqOVcvX/L7vMbOVazdoZnua2ZNm9pGZPWpmI2rmzW9ml5nZG+X4ONbM1qqZP7Ycc5s9XpZlFjSzS8pv/NXy9wLt+62JSK1ynbMOcLa719+vHEFkMk3v9z+0XLesZWbjynXH+WbWz8x2Kef7t8xsVN2+3czWNbO/m9mHZvaYmX29Zv5AMzu9bHuSmT1jZgc2s40VyjFmQpm2rpndbNGl710zu8vM1i7zLgNuqVn3jvL3p9eEZnahmd1Qt5+NzOzxmvc7m9njpd3jzey7X/zbl65EgSVpNxZP5v9GdCtZjripOgnIZjZ7zaJfBr4DrAdsDvzQzC4FZgOWAA4CTjWzpWdgt5cDO9a0wcq2L5npDyTSg5XfyjZEUOl24B1qspZKcONa4C/AgsAhwMHT2exfgQWA1YHFiCf+tzQFMMxsL+AA4PtEVs+twPXAzaVL2rPArqU72lN1274FeAXYoaaNawLm7neX9t4G/AtYFFiTOAaNnuEvpYZFQGos8BYRUFsbWBq4cDqrbgdc4+5PAk9Q0x2ufA/XAw8BiwC789nv9CbgdWD7mmnrAJPd/T6LOnO3AXeU9dcmvu9z6tqwP/AqsIi7TyjTVgbWB1YCNgKGA8cj0rutWv4d18y8xwFjWvD3AOAR4viyL3CCmX2zbp2TgH3KMo8B15rZAAAz+yHxm9sHmA84EbjcptWQOxWYlbh+Whi4GbjR6oLZ7j6mueNl2c8/iG64SxPdcf5X2mBf6FsRkS9ixfLvo/Uz3P1Odz9rBn7/AHMABxLXZmsCmxDXHOsRx6EdgUPM7Bt1uzkJ+BWwEHAVcHVNQPlM4t5qQ2AgcY90kJlt1Mw2ziYeMg4m7ueuBYaW7V4HXFY+007AxuVvc/f1m/lOLgY2Kdtqkoj7NsxsF+CE8n0MBvYijqlbN7Mt6S7cXS+9vtCLuFHzZl5zEBdpfeuWfwNYo/w9DJgMzF0z/8/AQ3XrvAqkmnW8Zt4oYGz5ex7gI2CdmmXvbfR3pJdeXf0FrEUETfqW95cBd9XMPxJ4rG6d3Vv5LQ4DpgAD69Z5AhhZ/n4GOKRu/o3AsuXvCU3LlvdDy7FlaHn/W+COmvknAL+uacu9ddteuHb9Vr6LfwOj6qb9EHgNGFAzbdmyvVVb2M4i5fg2b3lfBZ6r+/5eA2atmTa8bHPR8v7kpu+05jMfW/7+Ze1/ozLty3Xr3wHcXrfML8t/69r97krUR5it0f8v6qVXo15EjaIpRIC6ft6s5be1PRFkvrRu/mgiKF57rBpeM38g8C7w/fL+ReCgum1cBFxX/h4HnFA3f/6av8fWHqeaOV6OJAJJVjOtL/ABMKzR37VeevXUVzmOOLB0K8tM7/ffdAxZvmb+b8s1Q/+aafcAB9a8d2CvmvezAE8Dh5f3SwFfqtvvlUClbhvH1byflbrrnHI886ZtUXdvVqaNYto1Yd9yPNqjpl0vA8uV9xOAferWP4Sa6x+9ut9LGUvSVs0V734fmARcVFI2Pyrp2oOJoFOTl939nZr3E4mngLU+IDKYWuXubwNjiAg8wPdQtpLIjBgBjHH3T8r7a4G1zWzh8n5FojZArf+2sr01iQuHiVYzwhpxUbN0yQAaCtxbu5K7D3f3GR1F5GJgXZs2gtq3KU/Qyv7XqNv3i2XejGQ/1luVCKxNqmnrv4kbxZVbWGc74G53f6O8vxZYzKaNuLci8Ki7f1SzTv13ejGwgZktVLIMduCzn3Gdus/4bJlX+xnvaKZt/6nb771AP+q6N4r0MhOJ49bgZuYtWP5t+j3/p27+/cDyddM+Xcbd3yUC68uXjMqF+fwx9V6mHU9OAPazKBOQS3eVAV/gs6wJzA9MrTk+TCaupdpyDBSRGfNm+Xfe5mbO4O+/Se310ETgCXevLZLd3P1R7XFnKvAg045NrwGHmtm/LUqUOPAtPntfBjXXDeVaYVaL0iYvm9nk0haaWa9Z5doyM61XyYbAq+7+uJnNR3TlPaXueuZYdKzq1hRYknZjZvMS3VX6EWmcgzzStV8h0smbTG1m9cnNbXIGd3058O3SzWQE027CRKRl2wHblz73k4gUaAO2LfP78vlzRGu/yUnAs80EnM3dD6lZt81dMtz9fuJGLZVucB+7+4M1+z+/hf3/rQ27+5Dmz5Gz0PLN3nZE4KfpOx1Tpjd1MZzud+ru/yIyu75NdIN7z90fLrMnAee08Bn/UbOZ92fg8zUFFJutXyfSSzQdP5obEnxV4trkwWbmQfyWm7ueaW6ZD2ve1zLK8cTdLyFutkYRx4CdgMfNbPXp7KPJJODWFo4PZ87gNkTki3uAOJeuUD/DzDYlMqChld9/kxIYqtWW+6PaY9OfiDpyPya6xxtxn1S/jU+vG8xsGaJr/kvA14lspaZ2fpFruEuAjUsg6duUbnBETxOAjZs5Vi3c7JakW1BgSdrTBsTTsl3c/UF3/8CioOSC01lvZl0NzE087XvI3V/u4P2JdGvlomFBYA53H9D0IupzNNUEeozP32y1lKkDMTz3omb2pbp9jTSzJd19IvA8UX+pad4sZnZbaQ9El5TpuYTop/9pX/2a/X+mvWY2u5n9xNpWwPshYLmm+ihle8sRT+vuq1/YzOYhahgtVPedXsBnv9Plzax/zarNfae1n7E2UD6eaTVhmvY7R/mM03uK+OW6OiurExeST09nPZEey92fJ2q9/biZ2T8ErqzJsF6ubv5KRB2mWp8uU36TXwEeL9lLz1D3+yWyjO4zswFm9i3gLXe/xN0PIn6jzxHHgebUHy/HAyuYWb+aNpiZ7VPqs4lIB3D3V4hu/Xs3U8/sYOJ6q8Xffzs0ofa4Y8BXiaD0PESdpoPc/SZ3f6scH9ZqYTtNtgA+cPe93f3xkrldX0dputdr7n4XkVWdiCypy8v0iURXuPrrmXVUY6l7U2BJ2lNTsd1kZrOVp2ynEymiHTYspbu/RxSV2wd1gxOZESOIfv31T8IuJp4uzUMUkl3cYrTGuc1sOLBnSxt091uJIuCXWIxcNqeZHUDUCGh6Wn8i8DMz26AEe44lgtFNx46ngY3KTVZLWUGXEBdFcYliqgAAIABJREFUI/lsYOkUIrB1opnNV26kziG6x7433W/k8y4mAi+nlO0tSWR1Xe/uzV0IbgX8y91fb2Y7y1sMRnAh8RTx12Y2yMzWJQoCN/cZ1yXqNtR+xpOJ/yYnlDYtBJxHZDZ8MJ3PMxmolv2uStRdOqmue5xIb7Qv8A0zO9bMBpvZvGb2ayJQfVDNcpuZ2Y8tRlkaQfw+T6rb1rFmtmYpWPsHovvIFWXecUDFzL5RtjGS6Op6LPH7/BVwusVolP2J4riL0nLGVP3x8lLieus8M/uymQ0ijr8VpnVjEZGO8VNgSWLwofnKq0oEoH9F67//mXWAmW1qUej/aKLY9lnE7/5/wLZmNpeZDQHOJ4JCrd2XPQEMshiBcnYzG0YcC71mvQmAm9lmZtbfzFra3qVEzc4X3L226/8ooovetmUfXyOOlYt90Q8vXYcCS9Ju3P0R4AfAYcTFzWnAoUSWQkenNl5GFKK9soP3I9ITbEeMHFLvT8QFxzfd/QWiW9xI4sLkF8RoY63ZgSjOeBeRQr0xsKG7N9U6Opk4LlxRllsV2Mrdm558HUw8pX+HFi62PEaLuxd4xd0frZn+BvFkbmXiKf94oh7SFu7+hbt7lYDLcKIu1HPA3URx3ZayB1r6Tm8mPuuIkrXwTaLWwCvE97Ff/QqlltODxIVYbe2E18pnXKO0aRxxrN1yBj7j/xE3r/8lMtOuBo6azjoiPZ67jyd+k2sRv6uniRHV1nf352oWPZ24SXwZ+CNRQPe6us0dSQTlXya6xWzVFLx197OIzOpzibpNPwG2cffbyzHwG8DsRAbDO0Rg6iB3b6l7/2eOl6UOy3Aiq/JR4re+MHEMnl7gWURmQjlvb0BcMzxNZDMuRfz+nm/t998Ouz+SuEb7H5EZtI27/690q/sWsFqZ9w/g78TotC3el7n79cT1wTnESLVHEFmdE5vWK9meBxMPz96h+e7ElPnzUVemxN3PBw4nHji+VfY1yt1P/WIfXboSa8P1tkiXY2Y7ATu6+4jpLiwi7cLMjgHWdvfNGt0WEZGOYmZjidGKRrUwfyjR1WVxd5/QWe0Skd6tFL3e2N3HNrotIn0b3QCRmVFqGMxFDFF5YIObI9IrlG5sBmxEZM2IiIiIiEgvpa5w0t2dT/QFvs7dx0xvYRFpFycRXbmmELUDRERERESkl1JXOBERERERERERaRNlLImIiIiIiIiISJsosCQiIiIiIiIiIm2iwJKIiIiIiIiIiLSJAksiIiIiIiIiItImCiyJiIiIiIiIiEibKLAkIiIiIiIiIiJtosCSiIiIiIiIiIi0iQJLIiIiIiIiIiLSJgosiYiIiIiIiIhImyiwJCIiIiIiIiIibaLAkoiIiIiIiIiItEmXDCxZ+JWZ3WNmt5vZGDMbYmZfNrNvdcD+RprZ7tNZ5qftvV8REelZqmaLVs2uqZrdUzUbVzXbtxP2ObI6nXNYG7c7ofw7rGo2ur23LyIyo6pmQ6tmL9RNG1U1G9mgJvV4ZuZmdquZ3WZm483saDPrkveOItJ4XfXgsCOwKrCOu28A/Aa4GPgKsE1778zdR7v7OdNZbL/23q+IiPQ4VwJnV9zXAr4GjKiajejIHVbcR1emfw4TERH5ojZx9w2B1Yj7sMMb3B4R6aL6NroBLVgQmAIYgLvfbGYfAKcBC5rZDe6+hZktDJwCDAL6Awe5+x1mNgqYDGwILABc6O4nmtlQYDQwAOgHHO3uf7V42jEUOAo4Efh62f/dwIHAdWW/Y4G9gGeIYNeKwEDgbHc/w8yGAXsAfYDX3L3Dn1SLiEjXUI1zwMcV978CVNw/rJodCpxYNbsTOBuYB5gE7Flxf7ZqtiVwJPAxcG/F/WdVs6WBU4lzySDgwor7idVmzmEV97+WJ/ZDK+6jqmaLA38E5gA+AfaquD9VjfPiJ8D6xDn2gor7b6tmcwBnAIuV7T4DfLfiPrWZz7cJ8KOK+/bl/aHA+Ir7Ne32JYqIfEFVs3HAucAI4rp814r7w1WzrwHHA1OJY+kJFfdLqma3EceyR6pmZwG3VdwvrJptRjzYHg2cVbY1V5n/07KvCcT9yDYV9/WqZusCxxDH5InEMffFzvrsncXdP7HIwH3CzH7j7h+Y2eHAJsBswBh3PxLAzLYBfk6cw54H9nL3iWZ2N3AVsHlZZ093H29m/dF9lUi311Uzli4EFgWeN7OLzGwXYBywP3Hg2qIsdz5wrLtvDCTgHDPrU+atBWwJrAvsaWarlvVvdvd1gK2B5er2uzKwjbuv4u6rAxOAwe4+DHjF3Ye5+3+IYNOz7v51YD1gpJktX7axFfB7HfxERHqdFYEH66Y9DHwV+B1wQyWe/B5FBJvmJYI621YiO3dg1WwrYCXg4Eqc29YE9qiazUo5h1VaPocBXAT8ruK+EVAt75tsXNZbC9i3ajY7sCRwXcV9o4r72kSAa1gLn+8WYNWq2dzl/RbAmBn4XkREOtJA4MNyzDwcOKRMXwkYWaZvCBxWpl9LHL8gjodblb+3Am4Algf+WNZbgzjufaUsMz8wuQSV5gH+AGxfju1nEQ+oeyR3f5MIFC1jZjsD85V7pHWAFc1sczNbgrhP2szd1yce0jf99/gScf80DDi0b9++l+ac9xk+fPhD66677mK6rxLp3rpkxpK7v1ECQWsT2UMV4GfEgQoAi6esGwInmtmnk4GFyt9XufsU4AMzu75sZyxwmpktCNwH1HcdeBpwM7sauAO4yd2fb6aJ25Q2NHXLm4M4Cb0OTHD3u9v40UVEpPvq08K0ycBwYB+AivudVbNHiHPYAxX3V8qy+wMfEU/Ij6ma/b6suzBxMzMWOK3awjmsajYQ+GrF/W9lPzdUzS6pmg0qi1xZcZ9clp0AzAuMB7YoT/CdyN79e3MfruI+tWp2FbBDaf+4pu2JiDRQH6JkBsDjxPESIvB9VMkC/Zg4vkEElv5YnXa9v27VrC+wasV9fNVsAHBkNTJypgDLAosQ9wmzEdlREA+vFwOuqsa9yCx03Yf27aUvkQG2DbBs6c0Bcd5alugu92XgxnJ/1h94rmndiy66aFzO+drLL798kV133XW5d99995SnnnqK1157rV/NtnRfJdINdcnAkpmtCTxcDiR3m9mvgceIjKImfYBPStS7fn0o3ehqTHX3q8zsLqIrwDrAsWa2YtMCJU1zBeJEsSpwqZn9yt3Pr9tWH+CH7n5P3X6HAe990c8rIiI9wiPA9nXTViOymFbjs+elPsQNSO00K9POIG6OhlXcJ1fN7gf6VNyvqtadw6o157BWTCn/flA3vQ/wI2AjolvH21WzE2k+QNbkYqLLwmN8NhtKRKQjvUt0N6s1O/AOQMW99vjWdAz7E3AmsFsJjL9Rln2sarYQcbz+GxEo2Qt4oqx3DBF4/0bp0nwlnz0uNl3r9wHuq7hv2Q6fr8szs8HEA/wnic/+S3fPdcv8BLjW3X/U3DZeeumlZ4cMGTIVWK4M1sSUKVN83333fWvFFVd8CPgz8K+U0iTdV4l0L101qr4tEfRpat+8RO2HscBgiCAQMK5plDgzW9jMxtass4OZ9bF46rAZcKuZHQks6+5XAgcRJ6OFm3ZaAlpHuftYdz+J6GO9Upndr6ab3Rhg57JOPzO73qKuhYiI9F43A3NXzbYDqJrNRnR7O77M+1aZvilRQ+kuYPWq2Xxl/dOJ7NolgJtKUGkD4skt1XIOq7RwDqu4vws8VjX7Rll+M6IG0sRW2rwEcFcJKi0AbNfaB6y430/UaFq7oqfIItJJKu5vAB9Uo0dD0/F1Y+CBVlZbArixBJW+Q3SZa/I34KfAP4gMpmOIbnBN640tQaVlaLl78F3AiqX+HVWzLapmf/jin67rs8jo+h1whkcQbwzwnaZ7IzMbbWbrAzcBw61kyprZXmZ2QNN2DjzwwO2AbW+99da/DB48eMqcc87JCius8M7tt98+BNjpk08++dsxxxzz5m9/+9t799tvv8MGDRo0f855wc7+vCLyxXXJjCXgCGAUcKeZTSKi4nsRT337l+Jv+xPBndPNbD/iacNB7j61ZCw9A9xI9Oc9z90fMLN3y/L9gDmB7O4Pm9nqZb8PAN81s3uIIqcTgd3KvPOAB8zsNOBY4OSS/TSVKDL3jJkN6cDvREREurCKu5cR4E6vmh1GnLtOr7j/vWr2GHBu1ez7RPHu3Srur1ajGOp1VbPJwO0V9zFVs6nAKVWzN4H7idpGg4BLyrY/PYeVArWr1zRjF+CMUlj7PeD702n274FLS7Drf8AVZV+t+RPTsqBERDrL/wGnluPlLMBvyyAIzS48yxxzHOiTJ99c7d//f8DVwJNVs0EV97eIYNKqFfd3q9EFy4igCESQ6eyq2e7Af4mC0587LlbcX6+a7QpcVo37lbeBH7Tj5+0Kbi4P7RcivrMjy/RziADcP83sEyJL6Q4Ai/Pf38rAS88Be9dsb/Edd9xxLDBg7733PgLYa4cddnh41113feWOO+5Yec4553x7+PDhry+66KJT3njjjeXnm2++BYHxOecXifpODxBZTY+mlHQeEulCzN0b3YZ2V0aFm+DuoxvcFBERkU6Xcz4YWIbIgronpdRuJ/uq2Z+BSsX96fbapojIzMo5b0cEnwaX1zxEEH7zlNI9ra0rHc/MJrj70NppOedVgIVSSjfUL59znpUonv4tInN30fLqD7wAvEh0y7sauD2l1Fp2roh0MAWWREREepic8zZE9tHHRAbvY0TB2ZtSSlPbss2q2XpEfaWbKu5HtFdbRUTaQ855NeAaaroIA9eklLZpYRXpRM0Flr6InLMBiwPfJGoDLkYUVZ+fyLh9kQg43QJcD0xoz4cqItK6HhlYEhER6c1yzn2Bhyj1mYpngdVSSm82plUiIh0n5zw/EVRoOu69AGyYUnqmca2SjpRznosyAAXRNa8pq+kj4r//88CjRPe5+1NKHzeoqSI9ngJL7STnvAgxvOaCRPR8IWAB4LiU0n8b2TYREel9cs6XATuWt68A30wptVboVkSkW8o570zUaJ0LeAtYFhidUtq1oQ2TTpVz7gN8FRhBjMa6GBFoGgi8BuyZUhrTuBaK9FxdtXh3d1QhCozPVjPtGeDnjWmOiIj0cqcCWwEvA3MDZ+SchytjSUR6ipzzQKKb73Ci3s5ORGBpLLoG73VKQe+HyguAnPNCxP8fm9dOF5H2pYyldlL6/f6FSMVsGp7i38DIlNK/GtYwERHplcp56UGimO2KwIlEDYrNU0qvNbJtIiIzK+e8KXAS0VPgT8BPUkoflXkDUkqTGtk+EZHeRIGldlRGL/gHsC7wJjARmBd4BLgMOKPphCciItKZcs47ACcDrxLBpVca3CQRkS8s59yfOJZ9m+jm+7OU0o2NbZV0d2bmwG1EgsA8wFXAKHdv04AXIr2NAkvtLOc8L5F++w7RBaFCjF6wDFE49U7g6JTSc41qo4iI9E45562B04iuIluklF5ocJNERGZYznll4BxgaeDvwK4aZl7aQwks9XP3T8ysLzAaeNLdj2psy0S6BwWWOkDOeRlgaNPTk5zzLMDWwI+BlQEHxgN/IIZB1X8EERHpFDnnzYCzgXeJgt4TGtsiEZHWlWvpo4HdgA+AX6aURje0UdKj1AaWyvvBwBPAl939AzM7HNiEqKc7xt2PLMttQ9Tz6kOMQreXu080s7uJrKfNyzp7uvv4zv5cIp1FgaVOlnMeQoxasS4whKjDdB1QTSm908i2iYhI75BzHgacT9ygbZNSerKxLRIRaV7O+cvARcQoX3cD30sp/a+xrZKepj6wVKY9SAQzVwDWcvd9zcyAK4EziYLx5wObuPtHZrYfsJC7H2xmTwGHu/ulZrYpUHX3VTr7c4l0FgWWGiTnPAD4ATF6xYrA68B9wLEppXGNbJuIiPR8Oed1gYuBj4ERKaXHGtwkEZFPlQEIfkSUlegLnAKcoEx/6QgtBJYeBnYGDgWWJWroAswFXECcPw8Gni7T+wPPuftOZjYBWNzLzbaZvQks5e5vdMLHEel0Cix1ATnntYFfEE9i5iKKfZ8PjE4pTW5k20REpOfKOa8BXA5MBXZIKWkoZhFpuJzzYOLGfSPiuniXlNITjW2V9GStdYUDLgQud/dct85PgOXc/UfNbG8C8JWm4t8lsLSEu7/VoR9EpEEUWOpCSuHvg4HNgKWI6PdtRD/ylxrZNhER6ZlKMdw/AbMAO6WU7m1wk0SkF8s5bw8cCwwmbugPSil90vpaIjOnmeLd5wLPu/uhZrYnsCWwg7tPMbPRRK3CN4BrgTXc/S0z2wuYy91/UwJLh7j7JWa2IfAHd1+5EZ9NpDMosNQF5Zz7ADsAewErAZOBh4ATgZuUAiwiIu0p57wcUWS0P7BzSunOBjdJRHqZnPPsRN2arYAJwA9TSnc1tFHSa5TA0m3EQ5aFiIBRpQSaZgF+BWwMfAJc6+7HlfW+A/yMqFn4HLC3u79fAktnEQkDA4AfuLuygqXHUmCpi8s5LwUcDnwNWBh4DPgr8LuU0nuNbJuIiPQcOecliAvp2YGRKaVbGtwkEeklcs7rAacSA9v8Fdg7pfRhY1sl0nZmNsHdhza6HSKdRYGlbqI8xfkxsD0xMsErwD1ENzkVXBURkZmWcx5KBJfmBvZMKY1pbItEpCfLOfclMvK/Swxkc3BK6a+NbZXIzFNgSXobBZa6mTJCxobAAcCqwGzAw0Q/30vVB11ERGZGznkR4AZgXqIrytUNbpKI9EA552WJwWqWB24hCnSrsLGISDekwFI3lnNeADgE+DqwBPAU8A/gVymlVxvZNhER6b7K+eVGYAFg35TSFQ1ukoj0EOUh6aHA3sRw7SeklE5vbKtERGRmKLDUA5Q04u8BuxHFvt8HHgR+A9yuYt8iIvJF5Zy/RASXFgV+nlK6qMFNEpFuLue8EHAxsBZwHzFYwAuNbZWIiMwsBZZ6mJzzCsBhxAl7fuBRYhjpP6oIooiIfBE550FEcGkocEhK6ezGtkhEuquc825EptJswBnAMSmlqY1tlYiItAcFlnqonPNAYH9gW2A54EXgbuDolNJTjWybiIh0HznnuYExwFLAESmlUxvcJBHpRsoxZDSwKfA4sFtK6ZGGNkpERNqVAks9XOnHPhzYjyj23Rd4CDgduFJPikREZHpyznMSBb2XJ0YjPanBTRKRbiDnvCVQJbLoLwN+mlKa3NhWiYhIe1NgqRcpI/0cRowqtzjwBNHF4dcppTcb2TYREenacs6zA9cDKwPHp5R+3eAmiUgXlXOeFTgVGAG8AOyXUrqlsa0SEZGOosBSL5Rz7g+MBP4PWBF4G3gAOC6ldE8DmyYiIl1YznkAcC2wBvC7lNKoxrZIRLqanPMawJnEiMXXA3uklN5vbKtERKQjKbDUy+WcVwd+QdwkDAYeJlKVz0wpfdTItomISNdTMhH+CnyNGBjikAY3SUS6gJxzH+A44sHlRKIm22WNbZWIiHQGBZYEgJzzPEAF2BJYFngWuIMYseO5RrZNRES6lpxzP+AvRNfqs1JKP29wk0SkgXLOXwEuJLrK3g58P6X0emNbJSIinUWBJfmMnPMswDbAPsAqwFSi2Pcf4P/Zu+8wu6qqj+PflYTee4cgvQSpEkAgNGnS4wZfEIKodEEZQaVF6ZiA9I6hw6ZXIzUgXUIgIMVGFFCR3gIhwHr/WHvI5TIzmZnMzJny+zzPfTL33HPPWfdmzp5z1tl7bW5LKekXRkREyDkPAK4FNgEuBQ7U3wiRvqVMEvMTYpIYgFOB09QWiIj0LUosSbNyzksARwHrAYsTU8TeDoxMKb1bZWwiIlK9MvTlKmDL8u/euqAU6RtyzvMBlxPnieOA3VNKL1UblYiIVEGJJZmqUqx1b2Bnotj368ATwHEppaerjE1ERKpVerpeRvR2vQEYpuSSSO+Wc/4/YDgwO/A74PCU0ueVBiUiIpVRYknaJOe8DnAYsDpxMvEMcAlwSUppcpWxiYhINcpwmIuBocCtwG66yBTpfXLOswIXAVsAfwN+mFJ6stqoRESkakosSbvknOclEkzfApYB/gHcDxybUvpPlbGJiEjXK8mlc4BdgdHALimlz6qNSkQ6Ss55Y+A0YBHgeuAAzSAsIiKgxJJMo1JfYyiwD7AKMIko9n0KcLeGQ4iI9B0luXQaMAy4FxiaUvq00qBEpEPknG8HlgAaUkqjq45HRES6DyWWpMPknJchin0PBhYGngNuImYH+aDK2EREpOvknEcAPyKmHd9eQ6VFer6c8yxAv5TS+1XHIiIi3YsSS9Lhcs4zAwcCOwErAv8FHgeOSSk9X2VsIiLSNXLOxwEHAI8C22rIjIiIiEjvpMSSdJoyJGJD4GfAqsBMRLHvC4CrVHtDRKR3yzkfCRwCjAW+nVL6qOKQRERERKSDKbEkXSLnvABwOLARsBQxk8g9wPEppderjE1ERDpPzvlQYF9gw5TSv6qOR0REREQ6lhJL0qVyzgOA3YDvA4OAD4FxwEnAQyr2LSLS++Scp08pfVJ1HCIiIiLS8ZRYksrknFcGjgDWAuYH/gxk4BwNlxARERERERHp/pRYksrlnGcDfgJsC6wAvEIUe/1VSukfVcYmIiIiUgUzOw7YB1jA3T+tOh4REZHmKLEk3UYp9r05cDBR7Ls/MB44G7gxpfR5heGJiIiIdBkzG1t+/Lm731VpMCIiIi1QYkm6pZzzosQwufWBJYG/AKOBk1NKb1UZm4iIiEhnMrOVgGOAx4jzoAzs5+5DzWwR4H53X7qsOxrYBhhWHp8BMwA/dPfxZjYc+BTYDLgYuAr4DVHrcjbgQnc/r8s+nIiI9DpKLEm3lnOenij0vRtxAvQO8CQxm9yfqoxNREREpDOY2VHAv4jE0hhgceIm21LA94DfAoOB14FzgAQcDRzv7p+Y2cbAMHffvSSWvgNs4O5vmtkRwER3P8XMpgfuB/Zy9+e68jOKiEjvocSS9Bg55zWAXwBrAnMDzwBXAhemlCZVGZuIiIhIRzGzJ4FvufsbZvZX4AfA/sBZwI+ApwEHXgNmdPfzzWxD4BDiHGk64AN336QkluZ094PLth8vu5lY/p0b+LW7X9c1n05ERHobJZakx8k5zwkcCmwJLAf8E/gjcExK6eUqYxMRkfYxs0WJnhcLANMDF7v76Z28z2FAf3e/qIO3O8HdB3bkNqXvMLOliUlMniiLlgNuA/4ErEzUodwVuIbosdQAfAw8C2zt7k+Y2SDgDHcfUhJLA9z9iLL9scC+7v44IiIiHUCJJemxcs79iJnk9idOsj4HngJOB+5IKemXW0SkhzCzx4hhPDeb2UzAHcDp7n5jxaG1mRJLMi3M7DDgNXcfVZ5/A7gBWA14gaiJdJiZPUAkjNY1s1WA64kklBPnQoOaSSwdB8zm7j82s+mAm4H93f2lLv2gIiLSa/SrOgCR9kopfZ5SuimltBmwFnA7sARR4PKJnPOvc86zVxqkiIhMlZkNAT5x95sB3P0j4HDgUDOb38xuMbMHzOxOM1uivGcrM3vMzP5oZqeUZcua2d1mdp+ZPWVmh5TlA81sjJk9amZjzWy7snxYuejGzJY0szvM7H4zu6f0GsHMhpvZEWY2umzzp2X5LGZ2eVn/MTO72sx0XiUdYRvgpsYnpWfRO8AywIvAneWl0cDDZZ3xRDL2CeBuonfTXM1s/zhgZjN7hKjfdK2SSiIiMi3UY0l6lZzzjMA+wM5Ed/HXiZOsY1NK46uMTUREmmZmBwLLuPuPa5bNBvwbuBX4o7ufY2brAT8B9iZ6qK7l7v81swuIXhczAhPKUKDpgPFEj9aTgDfd/RgzWxjY3d1PLEPhBrr7cDN7CPiVu99pZlsCR7v74JJ4Wh/YAjCigPKKxEX+iu5+VYn3UmCUu9+rHksiIiLSlyixJL1WznldohbTGsR0us8Ao4BLU0qTKwxNRERqmNnBwNfqEktzAv8ghvUs7e5vl+VzABsAP3D3xp5HswCTgNmJKdpXBSYTQ4dWJv4OnEMMJ3oCuM3dX29MLAEjgVfcfY6a/b8NfA04CPifu59dlo8hZuV6BTgM2KrEOBA4wt0vU2JJRERE+hJ12ZZeK6X0cEppe+LC4kLiguMMYFzO+ayc80KVBigiIo2eJdrqWqsD44hzFatZ3r+JZVaWnQe8CQxx9yHA34ji3DcRyaZ7id5GT5vZPK2I67Py78S65f2B/YANgW3dfUPgurJcREREpE9RYkl6vZTSGymlBuIiZS9ieNwuwNic8x05501yztbiRkREpDPdA8xhZtsDlOLdvwJOLK/tWJZvCpwNPAKsYWbzlfefC2wMLAXc7e6TzWx9IomEmR0NLO/u1xO9jN4FFm7cubu/DzxnZpuV9b8FjHf391qIeSngEXd/x8wWALaf9q9BREREpOfRUDjpk3LOywJHAoOBhYDniEKZp6WUPqwyNhGRvsjMliISRHMRPX/OdffzzGwR4GKiftLHwPfd/VUz24lIEk0majD9vCSERgBvAWOBFYgi4B+UbU8HzArc6u5H19VYWpbo8WRl/f3c/V+lxtKEmhm6xgDDiOFvVwGfAq8Bfydm8jpVQ+FERESkL1FiSfq0nPMswAHATsSd7f8AjwO/Tim9WGVsIiIiIiIiIt2dEksiQBkKNwRoIOp8zEAU+z4fuCal9Fnz7xYRERERERHpm5RYEqmTc14Q+CWwEVFD42/A3cAJKaXXq4xNREREREREpDtRYkmkGTnnAcSU0nsCg4iaG+OAk1JKD1UZm4iIiIiIiEh3oMSSSCvknFcGjgDWAuYnpsbOwLkppY+qjE1ERERERESkKkosibRBznl24KfAt4nZhl4hpr3+VUrppSpjExEREREREelqvT6xNMLsOGAfYIEG90/buY2fNLif2rGRSU9Win1vARwErEpMjT0eOAu4KaX0eYXhiYiIiIiIiHSvvbLwAAAgAElEQVSJvpBYGlt+/HmD+13t3MaEBveBHReV9CY550WBI4H1gYHAi8Bo4OSU0tsVhiYiIiIiIiLSqXp1YmmE2UrAMcBjwJIN7vuMMBsDDGtwn1DWmdDgPnCE2TBgJWBZYFFgTIP7ISPMbgY2Bx4Fjm5wv7/rP4n0BDnn6YG9gN2AlYG3gSeB41NKT1QZm4iIiIiIiEhn6O2JpaOAfxGJpTHAQsC9NJ9Y+hkwmJj96xFgjwb3F9VjSdoq57wm8AtgDWAuotj3FcCFKaVPqoxNREREREREpKP0qzqATrY9cFuD+/PAe8RQpZbc3uD+fkNk214gZv8SabOU0hMppZ2A1YCzgZmBEcBTOefzy/A5ERERERERkR5tQNUBdJYRZksDiwOXjzCD+KxDgc8BK+vMWPe2iXXP+3dymNLLlRpLv8g5Hw5sB+wP7ABsm3N+GjgN+H1Kqfd2HRQREREREZFeq9cmloCdgIYG91EAI8y+AdwA3AOsCLwE7AtMbsW2Jo4wm7vB/a1OilV6uTJL3I3AjTnnJYGjgHWBa4EXcs63AqeklN6rMEwRERERERGRNunNQ+G2AW5qfNLg/jjwDnAecPQIs/uIHklvtmJbpwMPjjAb3glxSh+TUnoppbQn8HXgCCK5eQgxTO6anPOgSgMUERERERERaaVeXbxbpKfIOa8HHEbUZJoVeAYYBVyWUmpNrzoRERERERGRLqfEUifJOQ8APlPtHGmLnPO8xGxymwLLAP8gZjQ8NqX03wpDExEREREREfkKJZY6QM75XGBpYKbymBmYHVgtpfRalbFJz5Rz7g8k4EfAKsDHwFPASOA+JSxFRERERESkO1BiqQPknDclhi0tUrP4npTSptVEJL1Jznk5otj3N4CFgOeA41JKN1camIiIiIiIiPR5Six1kJzzD4ETgHnKoteIHiano+nkpQPknGcBfkz0ZDohpZQrDklERERERET6OCWWOlDO+Xjiwv/vwDhgHWAx4Hng98CIlNI71UUoIiIiIiIiItJx+lUdQC9zOHAjMDqlNAxYFfglMIlIOD2Vc74x57x2dSGKiIiIiIiIiHQM9VjqYDlnA/qllD6rW74W8HNgdWBe4FngWuCclNJHXR6oiIiIiIiIiMg0UmKpi+Wc5wR+CmwNLA+8CjwKHJNS+muVsYmIiIiIiIiItIUSSxUpPZs2Bw4Cvg7MAIwHLgauSil9WmF4IiIiIiIiIiJTpcRSN5BzXoioxTQEWBr4G3A/MaX8fyoMTURERERERESkWUosdSM55wHAd4G9gEFE0e+ngdOAP6SU9J8lIiIiIiIiIt2GEkvdVM55WeAIYDCwCPACcDswMqX0bpWxiYiIiIiIiIiAEkvdXs55JmBf4DvASsCbwJPAiSmlP1UZm4iIiIiIiIj0bUos9SA557WBw4DVgbmBZ4GrgfNTSh9XGZv0DmZmwHHApsRQzA+BvQEH1nT3G9q53YHAKHcfMg2xLV4bg5mNAYa5+4T2bnNamJkDDxDfDcBb7r5jB2x3ODDB3UdN67ZEREREpGuMMFsUOAdYAJgeuLjB/fQRZqOAUQ3uY+rWHwIMa3Af1rWRinS8flUHIK2XUnospbQjsBpwOjAjcCLwdM55VM556UoDlN5gZ+L3a7C7rw/8BrgC+BqwbZWBdZMY6m3i7kPKY5qTSiIiIiLSY10PXNjg/g1gHWCHEWY7VByTSJdQj6UeLOdswFbAgcCqwADgGeAC4JqU0mcVhic9kJkdTPRW2s7dPyvL1iHuviwIjHP3Lc1sYeBMYC7ijsxh7v5g6W0zGdiAuFtzmbuPLD2WLidqhS0PfAR8391ftriL8ySwK1FXbFzZ3/zE7/TB5d/6GMYANwHblTgO9nInyMyOBDYBZgJGu/vRjb2miIL4i7j70Gn8rhyYzt0/rVs+ocS6rbuvZ2brAscA0wHvAXu7+6tT+Q4nAEsB/3P3M8p2bwd2d/c3pyVuEREREelYpffRMQ1xY7Zx2brASOBFSo+lEWbbA0cD/yau2xZscB82wmwO4AxgUWAW4IQG95tGmA0DViHOnx9ocD+xCz+WSKupx1IPllLylNLtKaUtgDWADMwHXASMyzmfnnNeoNIgpae5jPiD9rKZXW5mewBPEcmd0e6+ZVnvEuA4d98ISMBFZta/vPYNIuG5LvBDM1utLF8TON3dvwncQPzxbLQDsJG730XMgniXu29I1Be7GhjbRAwAC5UYdgFOAjCz3YD5yrC7wcAgM9uirL8G8OC0JpVq3GNmY8pjUFk2PzC5JJXmLJ9zJ3ffgEj6jizrtfQdQvQU27V8pkWASUoqiYiIiHRLg4ibo7WeAVZufDLCbB7ipuIWDe5bEzcdG40Ebmpw3xjYDDiurA9xnnuAkkrSnQ2oOgDpGCmlV4H9cs7TERejw4DdgJ1yzk8DpwJ3p5TURU2a5e5vlkTQ2sDGQAPwU+DQxnXMbBaiR9LIKMkUi4GFys83ld5OE83sjrKd64Fn3X18Wecq4NiaXd/s7hPLz1sAB5R4xpvZe0Th+qZcVv59kUjoQAyXW770aAKYnbjL8wLwmbtf24qvorU2qe+xRPSSurj8vC6wGHBT+a76Af1a8R3i7i+Y2XRmthQwFLiyA+MWERERkY7Tv5llk2uerwuMbXB/rTy/A/he+XlbYNkRZj8uzw1Ypvz8eIP7Pzo4XpEOpcRSL5NSmkwM9xmVc16BGFr0DWLI0PM559uAU1JK71UXpXRXZrYW8Iy7Pwo8amYnAc8BX69ZrT/waVOFuEuSxOoWf17+rU1qflb3/IP6TdU9b25Y5wcA7u42JUPTHzjW3XNdbAOb2E9nadxPf+AJd9+qLpbZafk7bHQlkSj+FjG0T0RERES6n2eBneqWrc6XezHVjxaqPenrDwxtcP9f7QojzJan685fRdpNQ+F6sZTS8ymlXYmkwNHAp8AhRLHv63LOa1QaoHRH2wHHmVlj2zAPUSR+DDETIe7+HvCUme0IYGYLl6Fgje8Zamb9zWxGIiFyf1m+rJktVn7+P+D3zcRwJ+UPs5mtVPb/PDCxMYapGA18t3FYmZmNMrNvtuJ9neERYijewBLLlmZ2Riu+w0ZXETXUnnf3SV0Yt4iIiIi03j3AHKWGEiPMZgJ+RUy01OhRYI0RZo3ns7WlGUYTo00YYTbbCLMHRpjN1vlhi3QMJZb6gJTSxJTSiJTSYGBzonjxWsC9OeeHcs4H5JxnqDZK6SaOAj4EHjKz+4DrgL2Juy3Tm9mjZjaY+MO3t5ndT9QCOszdG3smvQT8gfjj+Tt3f7IsPwU41cz+CHybGGLXlJ8A25dtnw4kd5/cRAzNuYgYGvewmT0EvOjuD7b9q5h27v4GsCdwdRmatzdRyBta/g4b3/9vIql2eZcFLSIiIiJt0hAzYu0A7D/C7AngYeDyhqgf2rjOa8RN/rtHxHnhKzWbOAhYb0ScJ98BnNTg/v7U9lsmcxKpnGaF66NyznMDPyPq2SwH/ItoAI9JKb1UZWzSczXOaObuoyoOpcfIOS8InAdcCIwuw1kBMLOZgbuAb7oaaxEREZE+K+e8PTF784LAHOWxELB+SmlChaGJKLHU1+Wc+xG9Rw4gprLsD4wHzgeuSyk1V9tG5CuUWGq7cgw+DSxL9PZ6Ebhq9913n2/SpEl7Ake7+61VxigiIiIi1co5HwwcyZdLQzyeUlq7opBEvqDEknwh57woUex7feBrwF+Ae4ETUkr/a+m9ItJ+OefLKOPqiw+AI1NKv60oJBERERHpZnLOPwcOA+Ysi14jJtq5EfhdSkmFvqUSSizJV+ScpyOmvhwGDCJq7jwNjATuSynpl0akA+Wc1yaGvDUWacwppZ0rDElEREREuqGc86+JmkwTiUlvVgWWYUqS6XfAzbXlFUQ6mxJL0qKc80rA4USx74WJxuoW4LcppakWlBORqSuFF59kyknBdMCeKaV7Kg1MRERERLqdnPNIYHBKab1SVmEIsC/RKWAgUV5hPHAm8KA6BkhnU2JJWiXnPAtRh2lHYCXi4ncscHxK6akqYxPpDXLO55QfTyBmA5kD2CuldGd1UYmIiIhId5Rznjul9FbdsumBoUSJhRWB+YjyJo8Dp6aUXujyQKVPUGJJ2qT0rFgPOJTodjk78CwxXfrFKaVJFYYn0mPlnPullD4vPy8K/B6YC9g7pXR7pcGJiIiISI+Sc54d+AGwDZFkmh54AbgPOD2l9N8Kw5NeRoklabec8zxE8bhvETNa/RN4CDhWU16KTJuc84LAH4B5gf1SSjdXHJKIiIiI9EA554WJukwbAssDHwHPAzcTnQNU4kSmiRJLMs3KuN7tgP2AVYB+xJjec4EbUkqfVRieSI+Vc56fSC4tCByQUrq+4pBEREREpAfLOa8I/BRYk+gc0Fj0+1Li2k1Fv6XNlFiSDpVzXhw4AvgmUTjur8DdwIkppdcrDE2kR8o5z0sklxYGfpJSurrikEREupSZDQQedPdFa5YNBya4+6hm3vPF62Z2NXAIMTHCKHcf0oZ9T3D3ge0MXUSk2yolToYQnQPqi36fDTygot/SWkosSacoheP2AHYnGqoPgKeAEcD9aqREWq8MO/0DsBhwSErp8opDEhHpMtOaWKrbjhJLIiJ1yrXbjsS1W2PR778ypej38xWGJz2AEkvS6XLOg4BfAmsRQ3qeI8bznpZS+qDK2ER6ipzzXERyaSBwWErpd9VGJCLSNVpKLAEHAxcDOwCzAXu6+zN1PZbGAMPKW0e5+xAzmx+4Efieu//DzH5IzKI0PXEj7EB3/9TMJgArA88AS7v7Z2a2HrCjux/SmZ9bRKQKpej3XsC2wArAjEQ9pjFE0e//VBeddFdKLEmXyTnPCvwY2J7IhP8XeAI4PqU0vsrYRHqCnPMcRHJpKeCXKaULKg5JRKTTTSWxdCRwortfYGZbAru7+3dbSiwBWwO3AT9z9yfMbH2i3shO7v65mZ0KvOju5zb2WCrD6Ua5+2gzOw84193HdfZnFxGpUs55IeL6bQiRZPqYSDLdAlyUUnqvuuikO1FiSbpcGc+7AVHvYDXiDuMzwGXAqJTSJxWGJ9KtlbtIo4lii0emlM6pOCQRkU41lcTScGBFd59Y1rvI3TdpIbF0OVGodjFgcEkknUzc9Pp3WWdm4AF3b6hJLG0LfIeYuvtud1+/sz6viEh3lHNeAfgJMQplWeB1vlz0W9dwfZgSS1KpnPN8wGHApkQDNQF4CDgmpfSvCkMT6bZK77/RxJ2j4SmlMyoOSUSk05jZPMBz7r5AzbKTgUeAUxtrINXWUGohsfQ3YCgwGHje3S8xs5HAa+5+chP7bkwsTVfeexSwkLuf2BmfVUSku6vpJNA4I/iSxDXcM0TR7zGqp9v3KLEk3ULOuR9RH2Ff4OvA58SMBOcAN6WUPq8wPJFuJ+c8C/B7ovbHsSmlUyoOSUSk05jZS0Rdo3FmNhPwAJEgur+5xNJ000338ieffHJRXWLpUnffwMzmJuqFrEVcIB0HbODuH5vZscCz7n51bfFuMzu37HM1d3+5Kz63iEh3lnOejriG24ModbIA8Bei3MkpKaXnKgxPupASS9Lt5JyXIGomrAcsQTROdwEnpZTeqDI2ke4k5zwzcAeRjD0xpXRSxSGJiHSKUgfpZGAy0A84y92vMrMJ11xzzZLAkk8//fQOl156acPIkSOfveyyy74+22yz/evKK69cs6ni3WWbRwEfuvtIM/sJ8H9E/ZCxwCGlUHdtYmlj4Eh336jLPriISA+Rc54N+D6wHdGrfibgBeB+YtKmf7fwdunhlFiSbqtMe/l9YpaWlYH3gXHACOCP6mIpAjnnGYHbgdWB36SUjq84JBGRLlWKy44FFqpZ/BqwQUrpLx21HzM7GHjH3Ud11DZFRHqjnPOCwEFE0e/lgUlEkukW4EIV/e59lFiSHiHn/HXgl8CaRBfLPxPTBJ+RUvqwythEqpZznoGY4Wgt4LcppeHVRiQi0rVyzj8DfkXcIQe4NaW0bUds28xmJ845JhIzx6lArYhIK+Wclydm3qwt+v08MXHTdSr63TsosSQ9SulieTCwDbASMYPLE0SNmT9XGZtIlUoPv1uAdYDTU0pHVhySiEiXyDlvApwFzFMerwMb67xARKT7KEW/vwnsT5RxaCz6/SxTin6rrm4PpcSS9EilYRpCZL9XA2YhZiK4FLgkpTS5uuhEqlEKKN5M1Cc7O6X0i4pDEhHpNOVc4BjgR0QyaT/gN8AbKaWtqoxNRESaV85ZtwP2ZErR778Sw5pPTSk9U2F40g5KLEmPl3OeH/g5sAmwDPAS8CBwTErplSpjE+lq5Q/1DcCGwPkppYaKQxIR6XA55zmBq4lE+kPALimld8oEIANSSn+vNEAREWmVnPOsRIJpeyLJNDNRj+kBosTDqxWGJ62kxJL0Gjnn/sQ0wD8CVgE+A8YTXStvUddK6StyzgOA64CNgYtTSgdXHJKISIfJOa8NXETc4b4AOFwTeoiI9Hw55wWAA4kOA8sDnxD1mG4nbpi+W2F40gIllqRXyjkvCRxJ1JtZAngRuBM4KaX0VpWxiXSFkmjNwLeAS4ADdeElIj1ZGfp2GDHT0HvAASmlu6qNSkREOkPOeTmitu5gouj3G0SS6Qogp5QmVRie1FFiSXq1MlvWD4DvAoOAd4EniRoMD+tCW3qznHM/4CpgK+KP8L76nReRnijnPAtwObApMWlHSim9Xm1UIiLS2cpNhXWBA4ii318D/kkU/T4HuFcjU6qnxJL0GTnn1YFfAGsA8wF/Bq4HzkopTawyNpHOUpJLlxMzKV4D/FDJJRHpSXLOqxCTcyxOTE/905TSZ9VGJSIiXa3UEt0WGEbMEL4g8DemFP0eX110fZsSS9Ln5JxnJ7pVbkMUiPs38DhwbErp+SpjE+kM5U7PpURRxOuBPZVcEpGeIOe8PzFBxyTg0JTSDRWHJCIi3UAp+r0HsCOwAjFL+IvAH4kkkyZx6kJKLEmfVS62NwF+AqwKzAQ8A4wCLk8pTa4uOpGOVX7fLwK+A9wE7KFuwyLSXeWcZwQuBr5NDHdIukgQEZGmlFnCDyCu7VYAPiXqMd0BnJdSeqfC8PoEJZZE+GIGgl8Qs2gtA/ydyHYfqykupbcoyaXzgV2A24BdlVwSke4m57w0UR9uGWKGy311s0dERFqj/A05BFgbWA54k0gyXY6KfncaJZZEapSZtBLwQ2AVYDIwHjgTuE3Dh6SnK8mls4HdgN8Duyi5JCLdRc75e8Ax5elRKaVLq4xHRER6pnLOOxj4MV8u+v1n4Fzgbp0DdxwllkSakXNeCjgCWIcoGPoCMBr4TUrp7SpjE5kW5Q/t6UThwzuJISYqhCsilSkFWc8BhgJ/Bb6bUvpbtVGJiEhvkHMeQNTX/T5RY3chouj3k0Q9pqcrDK9XUGJJZCpKnYcfEsOHBgFvE43QSSmlR6uMTWRa5JxPBfYC7gN2Sil9WnFIItIH5ZwXBTKwMjFM9/sppY+rjUpERHqjnPMswO7ATkQ9ptmIDgQPAaeklF6uMLweS4klkTbIOa9JzE6zBjAvUVD0OuCclNLEKmMTaY+c82+AvYEHgB1Ux0REulLOeUfgZGAG4MSU0lkVhyQiIn1Eznk+YH9gMyLJ9BlfLvqtUSqtpMSSSDvknOcAfgpsTTRC96eUtqo2KpH2yTmfAOxH3KnZPqX0ScUhiUgvV2oangJ8j6h5sUdKaXy1UYmISF9VyqAcQtRlWpYYpfIcMZnE1epJ2zIllkSmQalV8y3grZTSn6qOR6S9cs6/Bg4maogdM7X1RUTaq9whvpbo/XsPMUPlh9VGJSIi8sX13TeIot+rAksB/yKKfp8H3Kmi31+lxJKIiACQc96a+GOp4XAi0ilyzpsBZwBzAL8FTtaMqyIi0h2Vot9bEzVJVwIWJop+n5VSOrfK2LobJZZEREREpNPlnDchinT/B/iBJsAQEZGeIuc8MzF8exfgzJTS9RWH1K0osSQiIiIina7Msvpb4LCU0rtVxyMiIiIdQ4klERERERERERFpl35VByAiIiIiIiIiIj2TEksiIiIiIiIiItIuSiyJiIiIiIiIiEi7KLEk0g2Z2UAze6Vu2XAzG9aObU3oqLhEzMzN7H4zG1MeN1Qdk4i0Tjl+B9QtW9XMzig/t+nvjJkNM7PhHRuliIhI72Zmi5rZrWb2uJk9ZWY/LstHmdmQJtYfYmajujrOthgw9VVERES+ZBN3/7TqIERk2rn7U8CBVcchIiLSh1wPHO/uN5vZTMAdZvZy1UFNC/VYEulhGrPaZnafmT1hZoPK8lnM7HIze6D0JFm1iff+sPQ2ecTMzqm/cy3SXmY2wcwOM7OHyvN1zeye8vt4m5ktUpYvbGY3lN/fh8zsm9VGLtK3lR6yY5pYPtTMrrIwh5ldamb3mtljZrZ93bp7mdnImufnN/5tEhERkSlKj6RP3P1mAHf/CDgcOLRuve3NbJyZ3Q5s0eWBtpESSyI9z2zAR+6+EXAk8Muy/HDgVXffANgDOLf2TWa2PrAVsJG7rwN8DPygy6KW3uSemqFwjReP8wOT3X09M5sTOAPYqfw+XgA0XnReAhxXfn8TcJGZ9e/qDyAizTOzDYEfAcPc3Ynj9yZ33xjYDDjOzOapecv1wA5m1s/MZgSWc/dnujxwERGR7m8QMK5u2TPAyo1Pyt/YM4Et3H1rYLquC6991FtBpOfpD1xRfn6euKAH2JySKHL3f5rZ5nXv2wZYCbjXzABmBj7r9GilN2pqKNxMwMXl53WBxYCbyu9aP6Cfmc0CbACMLMsBDFgI+FJNMRGpzMrEDYuL3H1SWbYtsGxjDQjiuF2m8Q3u/o6ZPQVsBMwL3NqF8YqIiPQkTd1Q7Q9Mrnm+LjDW3V8rz+8AvtfZgU0LJZZEuqf3+WpmembgXQB3n1izvLFx6kec7DeqP777Axe6+8kdGKdIrQ/Kv/2BJ9x9q9oXzWx24FN3H9LVgYlIq20DDAFuN7Pz3P0d4pge6u7/q13RzJaveXoFsCswD7BvF8UqIiLS0zwL7FS3bHW+3IupfmSZ0c1pKJxIN+TubwITzWw1gFLUbSPgyRbedg+wY1l/WeCWutdHA0PLMAXM7Fgz26WjYxcBHgEGmdlAADPb0szOcPf3gKfMrPH3dOEynE5/i0S6j5Pc/d9EF/xflGWjgd0AzGy2Ujtttrr33UYMk5uuvF9ERES+6h5gjsZ6heU671fAiTXrPAqsYWZzl+dDuzbEtlOPJZHua3fgbDObTCSBTylD3Jpb/xiiXs0DwOfA3rUvuvtdZrYy8Ecz+xgYC1zbadFLn+Xub5jZnsDV5XftHaJeC8TF6blmdhDxe3qYu39eUagifdE9Zubl50eA8+pebzweLwHGmtkZwEHAeWa2Q3ntRHd/v/bvkbtPMrP7iBNmERERaYK7e/l7eq6ZHUH0Cj63XKvtWtZ5zcwOAe42s/eAu4Glq4t66ixqMoqIiIiItE/peXgf8G13f7/qeERERKTraPiBiIiIiLRbGd76GHCFkkoiIiJ9j3osiYiIiIiIiIhIu6jHkoiIiIiIiIiItIsSSyIiIiIiIiIi0i5KLImIiIiIiIiISLsosSQiIiIiIiIiIu2ixJKIiIiIiIiIiLSLEksiIiIiIiIiItIuSiyJiIiIiIiIiEi7KLEkIiIiIiIiIiLtosSSiIiIiIiIiIi0ixJLIiIiIiIiIiLSLkosiYiIiIiIiIhIuyixJCIiIiIiIiIi7aLEkoiIiIiIiIiItIsSSyIiIiIiIiIi0i5KLImIiIiIiIiISLsosSQiIiIiIiIiIu2ixJKIiIiIiIiIiLSLEksiIiIiIiIiItIuSiyJiIiIiIiIiEi7KLEkIiIiIiIiIiLtosSSiIiIiIiIiIi0ixJLIiIiIiIiIiLSLkosiYiIiIiIiIhIuyixJCIiIiIiIiIi7aLEkoiIiIiIiIiItIsSSyIiIiIiIiIi0i5KLImIiIiIiIiISLsosSQiIiIiIiIiIu2ixJKIiIiIiIiIiLSLEksiIiIiIiIiItIuSiyJiIiIiIiIiEi7KLEkIiIiIiIiIiLtosSSiIiIiIiIiIi0ixJLIiIiIiIiIiLSLkosiYiIiIiIiIhIuyixJCIiIiIiIiIi7aLEkoiIiIiIiIiItIsSSyIiIiIiIiIi0i5KLPUxZjafmZ1tZv80s4lm9ryZNZiZdYPYRpnZqDa+Z7omtjG6QwMT6UPKMeQ1j4/N7GEzG1x1bK1R3yY08fqwus/3qZm9aGZ7dsb+yz52aeV7J5jZz9uwr+FmNqaNIYp0uZp2ZasmXhtjZkPauL3646xNx05HqD9nqT/Wzaz/tJ5bmdmxZbtDW7HuwLLuwPJ8uJlNmJb9N7GPdrdvIn1NV51PtXTeU7f/+sfwjoxDRImlPsTM+gG3AcsBWwLzAj8G9gVGVhhau5jZ94C/1C5z92HuvkVFIYn0Fn9wd3N3AxYAHgZ+b2bzVBxXi5pqE5oxqebzzQwMB843s02mcf93A4fXLiv7ubo173f3ge5+4rTEINKN/Rs41cymn5aNmNmFwAW1y7rDsVN7rJvZ4sAnwBLTuNkdgHeA7dsRz3B3HziN+//CtLZvIn1Up55PNXVc1mrcd9n/JGDPmmXDOyIGkUZKLPUtywHfAA509+fcfaK73wXsBKw2rSd7FegPVN7TSqQ3c/d3gZ8DE4Fh1UYzVW1uE9z9E3e/CrgROHga9z9gGt8v0puNAOambxxn/ZjGc2wzW4ZITJ0AbD213phdoCd87yLdViedT+m4lG5DiaW+pX/590tdMN39KXffyN0/aWo4WhPdvSeY2bZmls3sQzN7ycx2rnvPN8zsfjP7yMz+UbpzT1+3jQ3N7AYz+8TMvtIwmtk2ZvaImb1rZu+Y2V1mtmx57VHgd8ASpTvn5dnv93AAACAASURBVPWxmtkfzeycum3uUTtUzsx+WuL/0MweN7PNWv91ivQN7v4p8Dfga/DF0JWdzezccowvXZbvZjG89iMzG29m/9e4DTOb38yuNrM3zez9so1v1O6npePRzIaY2X/NbHUzG2sxlPdeM1uivN5km9AGz9d8voXN7Mqyv48shsrtURPLQDP7wMyWNbNxZvY/M3NgQ+Dosv9Ny7puNcN8zOzbZvZs2e7TZrZ5zWsTzGxYzT7czFY0s0dLF/rHzezrzX0AM5vOzI43s1fLd3y/ma3Rxu9BpLO8DRwBHGFmCzW1goWfmdkL5Rj5r5ldZGazmdmM5TjbA9ijHB8/KO/74tgpz1c2sztLO/E/MzvNzGaoeX2Mme1vZieZ2dtm9h8zO6gulj3N7KlyrL9hZteb2YLNfbjGY93MTgReKotfMrNXzGxwOYbnqFl/ztIeztzMJncA7gKuBeYENqrb3yCLc6SPzWwssHrd68OsZihca9oHM/u+mf29fG8Pm9laZrb81No3M1uq/Lx23fbuszJUzswWLO3qu+X/5EozW6C571OkN2rifGpdM7vHzN4qx+UjtcdROQYfqGmHrrY4n2r2uGyrsq07LM69XjazM81s1prXHzSzvc1shMX12L/N7ICa12c1swvM7LWyjces5tymrLNHTbv+tGkIba+kxFIf4u7PAg8BF5Y/9r80s/XNrP/U3tuE44FrgIWIu5BXmtkqAGa2AnA3cDkwP7AJsB5x163WMcADwHylof2Cma1InEydXvbxNaIhPq98lsHAnsA/S3fO3ZqI8Qpgx7rPl4DGrupHA3uVZfMQw2GytXDhJtIXWSR+lwb+UbP4QGKIxmLu/jeLxMvJwP5Er4S9gZPNbJuy/tnADMAKwMLAPcAfGi+0Wnk8zgycCvyAaBdeBS6EVrcJLVm+5vNdD3xOXKjNQfSwOM/Mlqz9WoAziQvlRUs38/uBX5X9312/AzNbiegZdRbRNh4GXGNmy7cQ16nEd7oo8Bxwm5nN2My65wPrA5uW7V8I3G3NXMSLVOAC4O/ASc28/iOibdmDOPYGAysCP3X3j8txdglwSTnOLqzfgJktQhyLY4njZnNgM+JYqvUT4H1gSaLn9nFmtl7ZxubAb4FDibIBKxM35+rPY77C3X9etgmwpLsv6u6PAv8lkkWNvgOMdveJzWxqe+BWd3+JOPa/GA5XkmR3AH8un3FYibUlLbYP5TOfRbRL8xPnW3cAb0ytfXP3vwOPEe13Y4wLAGsAt5Y2617ib8ayRA/614j2TD3Ppc+oPZ8ys7mBO4kyJQOJ85rbmXKd0r+89gfimPw6cR51tbu/0JrzjlbEsyBxLfYgsAiwNjFk76K6VX8GvFHi3Jk4v2tMgB1LnNutCcxX3nuLTan3thdwHLAPcX64H3CKmW3Z1nilm3N3PfrQA5idaACeA7w8XgVSeX0UMKruPV9aBkwATqhbZwxwUc3619a9vi4xtnf6mm1c1dx+gFmBQXWvDwI+qHk+DJjQwjbmIWocbFqez0mcRM5JXKB+CGxd9/7z6z+/Hnr0pUc5hkbXPJ+LuMB6E5i3LBsDPFz3vgnA/nXLfgmMKT8/BZxc9/r85d+pHo/AkNJeDal5/evAezXPv9ImNPH5hgEf1zyfEdgN+BTYuCxbGZit7n1jgaHl54Ellr3r1hkDDK9b9kXMRI+qO+tePxnYueY7HFa3j81r1p2ttGHfK8+H13y/SxLJsJXqtn9nfUx66NHVj9KuDCs/f7P8rq5Tno+pOUYWAL5W994DgdvqtjWqbp3aY+c44hzHal7ftBzjC9bsc0zdNm4kElgQ7d5yda9vAzzbXBx1x3rj8Tuw5vWT+HLbOoa6Nq/mtQWByUxpI08kztWsPN8DeAuYqeY9m9Tuk5r2sDXtA3AfcH7d65cCG9bEO7zu9drP/GPg5ZoY9wOuqYnltbr/kwHEkKAhVf9+6qFHZzyYyvkUkSRare49s5Xjal7iesWBrWpen4ma85OmjssW4vm4sZ2sWXYs8EjdssXLfhctzx8E7q5b51bgx+Xn24ib+bXH9/w1P7/MV8+Xjqrfph49/6EeS32Mu7/n7ke4+4pEr4HdgdeBq8xsUBs29WLd87HEXUWAtYChVjPzANFTanriZKvRgy3E+QGwmMXwt9fN7FNgPDBLawN09zeJLH/jML3tgXvd/R1gJeJi9ra6OH9I3E0T6cs2rzkmXiESOEPc/Y2adR5q/MHM5iNqgZxZdzwdx5Tj6WTgIDP7i8Uw2kOJpA607Xh8rObnd4kkdFvNULOP94k79Du7+73l9beJu2l/txhm4kTvpfr2p9k2rBmrAH+qXeDuh7r7NS2858Wadd8nipOv2MR6axK9qJ6t+w43Q22adCPu/iBxR/70Jnqr/A/Yzsz+VIZMfU70XG71335gNWCce1y9FH8iehytXLPsMb7si/bE3d8G1rYYUv+WmX0G3NLGOOpdDWxiMTvvYsRxfGcz624HPOHu/yvPbyPO2Rp7CAwCnnP3j2re8/cW9t2a9qGp9ml3d7+/dR/vi17s65bnX/QQJ84L5wc+r9n3ZOIiWe2T9GbNnk+5+yTifOQWi+G4k4H3yvtmKdcrZxK9f/5kUepjVyJJ3FHWAgbXtQv/LK/VHpvNtpdEsmxbYIKZ3WhmRxJJNEqPyEWBc+v28St07Pc6Siz1IRb1CeZrfO7u/3H3y4ANiN4CzdUXak3ByH5Maeg+Zkq3zPpH7YxNH7YQ64bAzcAjxInUzMRQlba6khgON4Dodt54Afdx+XfJJmJct8ktifQdf6g5HmbxqMH2TN06tcfvpPLvRk0cTwsDuPuVRPJpOFF/ZBfgeYsaH60+HusupKB9Bfwn1Wx/Oncf5O7Xwxfd1O8mTni+R9x1M+DRJvbVbBvWDGtnvLVq29paH5fl/Zv4Dv+vifVFqvQzYujE9+uWHw4cTVxMLUv0atmHth03H/HV89vG989Yt169fhD14ohhe9cRCZcZgC3aGMeXuPs4YrjtUOC7wPXuPrmZ1bcH1iyJ7Y+JNgmmDKUbQPOfsSmtaR+mqX1y99eIIc47l+E1Xwd+X7P/+5s5Lzy/vfsU6QGaPZ8ys+WIY/vfwMZEb6XGNsoA3P1AIpF8IdHDbzjwlJnN1kHxfUyMOGnq2Ly3Zr1m20uPIXiLA4cQvUU3B54xs62Ycn63fhPbX7yDPoN0E0os9S1HAbebWf3/+wdEF/H3iDv1cza+UMb3DuarVqh7vgpR/BaiZ9FqtS+a2UJmtk8b6jntADzl7ke5+z/c/ROi+3ytz1qxnZuJnlLfIeo83VKW/5VoJOvj/JaZbdDKGEWE6AlJDEWpP54GWxThn9HMdgTedvcr3f0wovbGv4i72h11PLamTZialYkk9n7u/rC7v2dmcxK9qqZ1/88Sn/sLFkUyWypiuULNurMQ9eaeb2K9Z4gT0S/ViDOzncxstSbWF6mMu79K1Cs6nqil1GgocIG7X+Lur7n757T9b//TwKp1vaHWIhIr41oZ4lDgFnc/zd1f8agDWR9HS5qL8Roiqb4rcePrK8xsdmLo76LuPmPjA7iYKXWWngOWr6u31lJ9yNa0D021T9eXG30tfaZaVxDf3XeI+lCNF5XjgZWsZmY7C/tbCwXRRXq5LYGJ7r6Puz9fjpcv2hkzW8DMtiqvnefu+xFt2dI1603reU9T12yzmNmB5ZyjRWbWz8y+DfRz9+vc/XB3/ybwR2DX0vvzlSb2sY6ZbT2NsUs3o8RS3/Ibotvi9RazicxgZosTBRo/IIpl3w9sVg74OYAziDpF9YaZWbKYqWUf4iTotPLaCcC3LGZ2mdPMvkacbGzs7q1tAP8CLGMxu9wsZrY9UbC3MdkFcedvofJZZmyiSz0eRTFvLrHdVYbYURrvk4ARFjOazFQy69cQdahEpG2GA4eb2XZmNrOZrUO0KYsRQx6OJ7pCL2ExQ+RGRPfocR14PE61TWiFV4gk19DS9ixPtF9vM2VmzZb2v67FDClNFdg+lRgK84Oyzg7EcOSxLWzzOItZYeYm2uP3iO/1S9x9AlHU+EIzW6P8H+xO1HVScVzpjkYQ5x6r1iz7C7ChmS1uZnOb2c+IC6naY+8fwBpmNo+ZzdTEds8mhmQda2ZzWUws8lvgwpLQao2/ED2Gljez2c1sT2KoR2tvjv2XaEc2NbPpaxIqVxMFtOek+aG0WwFjSw+gWlcAy1pMbnIFcW52cvmMg4khvU1qZfswEvheacNnNbP9S6zjy+tTa98g6lTNCRzJlB7iAFcRNaF+V/5v5wJOARqYMvRHpK/5CzCXmW1RjskhxHHsRFszK3CdmR1U2sOZicT0JCK5DK07LltyOrCkmZ1sMUx3IaJd2IXoIdWikvxvAEaZ2TKlvVuLGOrbmMgfDhxZbjTObDFJwnVEsXDpRZRY6kNK1ngDYhjKTcTsHA8QJxWD3f1dd7+JOAG7iRiv/ypfPjlodAJRu+h/xEwk/9fYtbMMd9uSmGXltbKPJ4hhJa11PlE08rayjT2I4o8QdQZw94eAc8v2GwvhNeUKYpaCq+uWH0ucaF1GnPAcA+zh7re1IU4RAdz9EuJiYiSRhLmIKCh5dkkob0YMaX2CGJt/BnCYuzcel9N8PLahTWhpG28QJ1Q7l21cR1yojqO0PS04njgRfIOYFbN+2+OIITANZZ2jiILgf21hm0eX/f+H6DX1bY+6DE3Zj7hYvYOonfcDoujnk1OJW6TLld/jn9YtPpj4m/8sUV9sAaJdqT32zibOTV4mjvX67b5BDMXYmEjwjCZuMO3fhvCOIerIPUz0rFy/vH8B+2qv768oQ9z2Ji6o3gO+XZY/Vz7b1e5fqgFVa3viHKzeGCLxvYO7fwhsDaxDfMazgIOmElaL7YO730ycz51OnNt9F9iynDvCVNq3so33iaK+A6ipH1V6nW9O1Kj6M3F+uTCwgTc/K55Ir+budxC1hi5iyjnBAUSbsbDHbIubE6M4JhDnATsSx21jHaSpHpdTieF1ovD/mkRb9xRxDrZVC21UvR2Ycr33HtEb80wieYy7XwT8mri+fIu4xjtcw2B7H2v974xIMLMJxAXjqIpDERHplSym6X2JqDs1odJgRKTDmNkLxGQBT3fyfvYikvcqkCsiIp1uQNUBiIiIiIj0VmUo3ACi9/U7nZlUMrNZiXpSmwIvdNZ+REREamkonIiIiIhI51mdGF52ALBnJ+/rEGJYzDK0UHdJRESkI2konIiIiIiIiIiItIt6LImIiIiIiIiISLsosSQiIiIiIiIiIu2ixJKIiIiIiIiIiLSLEksiIiIiIiIiItIuSiyJiIiIiIiIiEi7KLEkIiIiIiIiIiLtosSSiIiIiIiIiIi0ixJLIiIiIiIiIiLSLkosifQyZraEmd1qZk+Y2Z/M7Bft3M4oMxvSweGJiIiISB9iZm5m95vZA2Y23sx+bWYdeh1qZgua2bUduU0RaT0llkR6kfJH+mbgRndfE1gP2NT+v717j9d0Lhc//rmMcVZIDuUwOzpIW7s2HZwaZlTsKGl/EVtU0km1a6lUfql250eifm1KfiqHfHfJIYeNyRiKiGKLjlLaJSKMRszh+v1x34vVMsPMsp651+Hzfr3Wa57n+9yH6168nvU8131d32/Evt1GJkmSpElsRmbuCDwfeBpwxGgePDNvy8x/XZZ9IuLAiPjyaMYhTVaRmV3HIGmURMTLgC9k5jOGjD0feAYQwKHtv1cCA5m5MCKeBxwFTAHuAd6YmbdHxEnASZk5e/lehSRJkiaKiEhgamYuaJ+vA/wC2ATYCPgSzefQtYFvZOZRETEN+CLwe+A5wH1Aycx7I2Jz4Hge/uz6usy8OyJuycxpEXEgsCXN59+NgNmZ+Z723O8FCpDA3MzceTn8CqQJz4olaWLZArhh6EBmXgtcR3Nn6KXAtsCGwMERMRWowKGZ+RLgfODY5RqxpHGnF5G9iEt7EXN6Edf3Ij7aG+W2hl7EBj3bGiS1IuLjEXFnRKzYPp8dEdPan9mjeJ7ZETFltI6nR8rMu4BbgWcCWwHvz8ydgG2AN0bEyu2mOwNfzMztaT7LDlbgfx3oZeZ0ms+xH1rMaXYD9ge2BraLiGdGxLOAt9B8Fn4hsHJE/EMfLlGadEwsSRPLAmDRYsZnAhdl5n3ZlCmeBuwKPAuYl5k/bbc7FXj5colU0ng3Y6CPbQ0DmbcNLGNbQy/iwJ5tDdJE9XLgFmCnfp4kM6dn5sJ+nkMArEjzmfV7wEER8X3gIuApwHrtNjdk5uAN05uA9SJiTWDLzDy/HT8d+PBijn9uZs5tP/f+rD3mPTSVSqsCKwGr0FTyS3qcVuw6AEmj6nqadreHRMQLBx8O23bhEsYXl5iSpMUayFzQi3gH8ItexGdZTFvDQOZRvSW0NQxk3ttbTFvDQObdvYhbBjKn9RbT1jDQtjX0hrU1DNjWIE04EbEl8Fvgh8BeNAmIoVaMiBNobpjdD7w+M2+NiD2Aw2g+26wJHJaZsx6jVWqwnWp94Cvtfk8A5mTmv7fb/AQ4F9ie5n1rn8z8fd9+ARNM2wq3IfBL4Gs0SaPpmTk/Iq6h+Z0CzBu26xQeWRgRQ7YfanH73g7cTDMlxB+Bz2XmzSO9DkkPs2JJmkAycw5wb0S8AaAtJf4EsAawS0Ss1m66N3AOzR2c1SNii3Z8n3ZckpbawGLaGgaGtDX0hrU1DCyhrWFgGdoaehHP7C2mraFnW4M0Ee0FnN3+7LmYFcW2Bo5tW6bOAL7Qjj8b2L1t938Nf1/Z8ohWqWHHfDbwf9sWra2B50XE09rXtqRJNL0EOAV42yhc46TQtjJ+Hjg+M+cBmwEXt0mlHWh+70uUmfcAv4yIme3Q64APLOXpXwDcmZlbZObOmXnKyK5C0nBWLEkTz17A8RHxFpqqpFMz85iIuAOYHREPAJfSTMydEfFa4KsR8SBwG82XNElaVkPbGj7WizgGmM+wtoaBv29r2LjXtjUM/H1bw5mLOf65A5lzAXoRg20Nv+LhtoYHsK1BmqheBbw0M/8cEfcCOwx7/YbMvL59fBrwH+3j84AvR8RTaaYLeOqQfc7N9j0lHn5P+fmQ168APhwRR9B8nnpWu//NwF8y87/b7W6iSTzp0c1qE4IbAt/l4STf+4EvRsRdwDXAJTTVrn95lGMdABzX/re5q32+NH4O7Ni23T0AzAVOyExvqkqPk4klaYLJzN/RzJ80fPwE4ITFjP+Q5m7/8PED+xGfpImnt4S2hoHM+b3HaGtY581vfspdxx03dOxxtTUM2NYgTSjtCmCbACdHBDTfX14zbLOhy1wvBLKdgPsCmra4C9q5ef5nyHaLe08Z6mPAk4BdMvP+iPg2j96ipSXIzCUm/DPzQppK18WZPmS7k4Y8vhHYcTHHmjZ82/b5gdBMAA8cmZnHR/M/0yY07ZUbLM11SFoyW+EkSdKI9Ya0NQwMaWtok0qP2dawxk47fX2Ftda6p/c42hoGMrcYyNx5wLYGaSLaCxjIzJdn5stp2vn35O+rE58RERu3j19Ls8rtGu3PnHb8kGU872Y0cy/d37bJTR9h/Bo7vgPsHxGX0vx/cTrwkW5DkiYGK5YkSdJIzOo9SltDbynaGlZ44hPXADZa7/DDF/3pQx/6j94I2xp6w9oaBmxrkCaS3YFXDD7JzKsi4m7gxUO2+RxwdDvh9t00VUr3RMR/AFe20wF8HViZpfcx4IR23spf07Torv34LkVdyswf8cg2SkmjIJoVGCVJkpavWuvHeHii7kuAGaWUpf5g0mvaGn43kHl8b0hbw0CmbQ2SJEnLia1wkiSpKzsNefximmXBl8V3gP17tjVIkiR1xoolSZK03NVaNwV+BKxLM9nuH4D7geeUUuZ3GZskSZKWnnMsSZKkLqwJnEQzue5BwD6llB90GpEkaVKptQbwQuDmUsrtXccjjVe2wkmSpOWulHJDKeUw4Ms0S4U/veOQJEmTz1rAOTQLT0gaIRNLkiSpSzfTrOL0/K4DkTQ+1VpXr7Wu0nUcGn9KKX8BbgM26zoWaTyzFU6ahGqtU4BVSil/7ToWSZNbKeWeWuvdwFO7jkXS2Ne2Lm0C7EqzAMAmwNOAlwLXdRiaxq/bgfW6DkIaz6xYkiank4Frug5Cklp30bQjSNJjeT/wY+A/gQK8CPh5KcWkkkbqT8B6tVaLLqQRMrEkTU4LgKntXT9J6trdwNpdByFpXPgs8NMhz/8GfLGjWDQxXE1TsbR514FI45WJJWlyuh1YDXhC14FIEs3d4rVNdktaCvsC04C/tM9vBL7VWTSaCGbTTBGzXcdxSOOW5X7S5HQrTWJpXeCejmORpLcBUUrJrgORNDa1iececBDwa+AI4ESgllIWdRmbxr2baG66bgd8teNYpHHJiiVpcvotDyeWJKlTpZT7Synzuo5D0thUa12NZkn4NwGXATuWUi4A/hU4psvYNP6VUv5Gk1h6StexSOOVFUvS5PS/7b+uwiRJksasWusmwBk0K7/9J/C+werGUsr3u4xNE8rtwIZdByGNV1YsSZPTn4G/0sxRIEmSNObUWncBvkczsfLbSynvtWVWffJ7mpXh1ug6EGk8MrEkTU53AvOAjbsORNLEEhEZEZdGxJyIuD4iPhoRft6QtExqrYcBJ9Os+vaKUsqpHYekie1S4MnAc7oORBqPbIWTJqd7gftxjiVJ/TEjMxdExIrASTST7H6k25AkjQe11hVpJuV+FXAtsGcp5S+Pvpf0uF1J89l45/axpGXgHURpEmrLyOfRTOAtSX2RmQuAdwCHRsRqEfGMiLg4Ii6JiJ9ExHsAImJaRHw3Io6LiMsj4oKIeEL72uYRMSsiZkfEWRGxVju+R1sZdXlEnDa4vaTxq9a6DjCLJqn0bWCmSSUtJ7fQzLP03I7jkMYlE0vS5GViSVLfZeZdwK3AM4GtgPdn5k7ANsAbI2LldtOdgS9m5vbAdcC+7fjXgV5mTgcq8KGI2Ax4L/DSdvsrgQ8sp0uS1Ae11n8C5tC8VxxZSjmolLKg47A0SZRSFtEkltbvOhZpPDKxJE1eJpYkLS8rAotoJuE9KCK+D1xEs7Tzeu02N2TmDe3jm4D1ImJNYMvMPL8dPx34MPAyYBPgvyNiNrA3LkYgjVu11n8DzgZWBvYrpXyu45A0Od1OM4F3dB2INN44x5I0ec3j4S90ktQXEbEOzRLOvwS+RpM0mp6Z8yPiGmBKu+m8YbtO4ZE3wKIdnwJ8NzPf2rfAJfVd+wX+s8AbgF8Bry6l3NptVJrEfg7sAGwA/LHjWKRxxYolafKyYklSX7WTd38eOD4z5wGbARe3SaUdgGc/2v6ZeQ/wy4iY2Q69jqbl7WLgZRGxdnueQyLisH5dh6TRV2tdjaZK6RDgMmBHk0rq2CxgLeCfuw5EGm8mfWIpIjaNiHMi4kcRcXVEHP4Y29+yhPEjI+LAfsQo9clfgdVqrZP+fUDSqJsVEZcBPwPuomlfA3g/8MW2fe1VwCXA2o9xrAOA/xMRlwL/Anw8M28CPgRc2I5vD3xp1K9CUl/UWjehmU9pO+A/gVeWUu7vNiqJHwN30sz5J2kZTOpWuIhYATgLODYzT4yIlYDzI+KWzDyt4/CkfvsTTcXSWjRf/CTpccvMJc5NkZkX0kzgvTjTh2x30pDHNwI7LuZYpwH+rZbGmVrrTOA4YCXg7aWUUzsOSQKglHJnrfUOmupaSctgslcq7AKslpknAmTmg8BhQEbEvhHxg4i4IiKOjogpw3eOiI+1VU7fxklDNf7cSpNYWrfrQCRJ0sRXaz0MOAX4G/AKk0oag27HOUilZTbZE0tbADcMHcjMa2mWOT4CeCmwLc2kowcP3S4iXk5zd/XFNEsiP7P/4Uqj6hZgVUwsSZKkPqq1rlhr/RrN5+ubaOZTur7jsKTF+RPNynCTurNHWlaTPbG0gGb54+FmAhdl5n2ZmTSl9rsO22YX4IzMXNBWOl3c31ClUXcbsBDYqOtAJEnSxFRrXYdmUuQ9gTOAmaUUW/A1Vl1FU7G0edeBSOPJZM/EXg8cOnQgIl44+HDYtguHPV/cEsjSePJnmpXhNu06EEmSNPHUWp9L0/q2LvCRUspRHYckPZZLab4j71ZrnQb8ppTy825Dksa+SV2xlJlzgHsj4g0AEbEy8AlgDWCXiBhcin1v4Jxhu18CvCIiVmi32205hS2NlsHEkhVLkiRpVNVa96P5/LwysJ9JJY11tdYvA58HHgQ+AJyJN2ClpTLZK5YA9gKOj4i30FQlnZqZx0TEHcDsiHiAJnN90tCdMvPsiNgeuBK4G7h8+YYtPW5/Be4HntR1IJIkaWKotQbwGeCNwK+AV5dSbu02Kmmp/I1mft3B78i/oCkmkPQYoplCSNJkVGu9HvhVKeXVXcciSZLGt1rrasA3gZ1ovpDvXUq5v9uopKVTa50CzAa2b4fOL6XYlSIthUndCieJecDqXQchSZLGt1rrxsAcYAfgOOCVJpU0npRSFtJMgfKrduj8DsORxhUTS9LkNg9YtesgJEnS+FVrnQl8D1gfOLSUclgpxbYIjTullD8Ah9PMRXpqx+FI44aJJWlymwes9phbSZIkLUat9T00K789ALyilHJyxyFpkouIjIhLI2JORFwfER+NiKX+3ltK+RbNhPN3jkIs0yJi9uM9jjTWOXm3NLmZWJIkScus1roi8FVgT+Bamkm67+o2KukhMzJzQUSsSLMI0xHAR5Z251LKhf0KTJqIrFiSJrf7gNXayQolSZIeU611HeBimqTSGcBMk0oaizJzAfAO4NCIWC0inhERF0fEJRHxk4h4DzxUWfTdiDguIi6PiAsi4gnta5tHxKyImB0RZ0XEWu34Hm1l1OURcdqQ7beNiKsj4jzgwG6uXFq+TCxJk9ttNHMsrd11IJIkaeyrtW5FM0n3FsBHKZXLZAAAIABJREFUSykHllIWdByWtESZeRdwK/BMYCvg/Zm5E7AN8MaIWLnddGfgi5m5PXAdsG87/nWgl5nTgQp8KCI2A94LvLTd/krgAxExFTgZODAzdwMedzudNB7YCidNbrfStMKtSzNJoSRJ0mLVWvcDPgk8SDMHzcUdhyQtrRWBRTSTzH8sIo4B5gNPAdZrt7khM29oH98EbBwRawJbZubgCnGnA2cCrwM2Af47IgBWAn4HPAuYm5k/bbc/D9irnxcmjQUmlqTJ7RYeTixJkiQ9Qq01gM8AbwB+DexVSvldt1FJSyci1gE2BH4JfI0maTQ9M+dHxDXA4JQQ84btOoVHdvhEOz4F+G5mvnXYuZ67mO2lCc9WOGlyu43mbs2mXQciSZLGnlrrqjQVGm8GLgdeYlJJ40U7effngeMzcx6wGXBxm1TaAXj2o+2fmfcAv4yIme3Q64AP0Mwx9rKIWLs9zyERcRjwM2DtiNi83f41o35R0hhkxZI0uf2Z5u7MJl0HIkmSxpZa68Y0k3NvDhwHvLeUkt1GJS2VWRGxAk2l0neBD7fj7we+GBF3AdcAl9DMNfqXRznWAcBxEXEEcBdwQGbOjYgPARdGxDyaNrg3Z+YDEXEA8M2IuA84tx8XJ401kenfBmmyqrWuBvwCOKuU8rau45EkSWNDrXUGTTJpZeCDpZRvdBySJGmMshVOmsRKKfNoKpZcFU6SJAFQa30PcArNJN27m1SSJD0aW+FGWTu54frA7aWURV3HIy2F+2km8JYkSZNYrXVF4Cs0q1j9GNizlHJXt1FJksY6K5ZG33Y0Kw7s0HUg0lKah4klSZImtVrr2sBFNEmlM4AZJpUkSUvDiqXRtwBYFVuLNH78FRNLkiRNWrXWrWha39YDPlZK+WzHIUmSxhETS6NvHs3y7U/sOhBpKd2PiVBJkialWutrgU/SfH7dr5RyccchSZLGGRNLo+9+mj/MT+g6EGkp2QonSdIk084L+mngjcDNwKtLKb/rNipJ0njkHEuj736adjgTSxov5gKr1Vqndh2IJEnqv1rrqsCZwFuA7wM7mlSSJI2UiaXRZ8WSxps/0FQsrdN1IJIkqb9qrRsDlwI7AscDe5RS5nUblSRpPLMVbvQNzrG0RteBSEvpdzSJpXWBP3UciyRJ6pNa607Al4FVgHeUUr7RcUiSpAnAiqXR9zeaVrjVuw5EWkq30Kxk+OSO45AkSX1Sa303cBrwILC7SSVJ0mixYmmUlVKy1jofcL4ajRe3Aw8Am3QdiCRJGl211hWBrwB7AT8G9iyl3NVtVJKkicTEUn8swMSSxo8/07Rwbtp1IJIkafTUWtcGzgD+GfgO8IZSyoJuo5IkTTQmlvrDiiWNJ3fSJJae2nUgkiRpdNRatwJOAdYDPlZK+WzHIUmSJijnWOqP+Zi00zhRSnmAJrH0xK5jkSRJj1+tdR/gHJo5FPc3qSRJ6ieTH/1hK5zGhVrrc4Bnt0//sdZ6CvCDUsr/7TAsSZLGtV5EAnOApFl59X+BAwYy5z7KPkcCtwxknjTS89ZaA/jUHZ/73LvW3n//n6243np7lFJ+O9LjSZK0NKxY6g9b4TRefBw4HXgGsCVQaKqXJEnS4zNjIHP6QOYLgPuAN4zkIL2Il/Uiznms7Wqtq9LMo/TWJ7/73ReuuN56LzapJElaHqxY6g9b4TRevB64Anh6+/wXwMndhSNJ0sTSi1gZWB/4dfv8loHMae3jacBJA5nTh+1zAPAOYCGwJvCPvYjVgeOBjYFVgN8Arx3IXNSbMuXWtQ888MH7Lrlk03Xf9a4v//Gd73w2sF5v773vWOI+ET8BTgT2bM9x0EDm//TvNyFJmqisWOqPBZhY0jhQSrmT5kPlg+3Q90sp8zsMSZKkiWJWL2I2TTLnDwOZj1l1BNCLWAM4Fpg5kPlC4Hpge2Bz4NyBzJe04w8C02utO01Za60NFtx225M3/NSnDtrvHe9465DDLXaf9rU1gfsHMncCjgA+8HgvWJI0OZn86A9b4TSefIbmbuWmwH90HIskSRPFjIHMBb2IKcDpvYh3DWR+fin2ewD4K7BGL2IuTQIImgTTrr2Iwbmbpq25664rADPJTCIOLqWcPuxYj9gHuKh9bQrNqnEAN9GsHidJ0jKzYqk/TCxp3CilLALeA/yilPK7ruORJGkiGchcCHwd2LUdWtSLiPbxKovZfj5N8mcOMAuYNZB5KfBW4CXAHpucfvqM1V70onunbrLJXsDPF959921zzz33msWc/qF9BjJfAnyLJqE0eK6h8ypOQZKkETCx1B+2wmlcKaVcDuzUdRySJE1Q2wA/ax//jodXZH3j8A17ERsCWwxkPq2d/Ptz7UubAVdscvrpseDPf579wM03b7nwL3+5lqZiadESzrsZcMVA5t29iPWBV43aFUmaVCIiI+LSiLgkIn4YEYe040dGxIEjON77I+JfRj1QdcLkR39YsaRxp5SysOsYJEmaQGb1IpLm8/adPLwq3IeAk3sRd9O0om09bL87gLV7ET+gaYubB3wbOCZWWeWs2z7wgfdNWXvtKVOf8pRL76n17INPP31+b++9lxTDMcBpvYiZwJ+A/wLWHsVrlDS5zMjMBdHMBXddRFw20gNl5qeWZftoFjD4NvB/MvOqkZ5X/RGZ2XUME06t9WTgRaWUzbuORZIkSeNHL+Jg4GkDmYe3z58E/GTDo4768NSNNvowzQ3Mt5ZSLuwyTkmTSzSJ8qmZuaB9XoEzgGcBt2TmSRFxBPAyIGhWtXwt8EzgrZm5V7vfB2nmf/vnIfs9YpXKzPyfiHga8FWaVt31gfdl5pnL7aK11GyF648Hgam1VnvVtdxExKYRcU5E/Cgiro6Iw0d4nJMiYvoI9/33kewnSZIeMgvYphdxWTvp9nmrbbvtLVM32qgH/BmYaVJJUpeiadl9EXD1kLE1aZJJO2TmdsBJwJuAS4DnRcQT2013BS4Ydsg1gfvzkatUfgI4NTN3BN4JvLIvF6THzVa4/phL0wq3KnBfx7FoEoiIFYCzgGMz88SIWAk4PyJuyczTlmMo7wSOXo7nkyRpQtnk9NN/U0qZCVBrXRU4DZgBXAqUUsq8R9tfkvpoVlu5BPD2zPz14FoEmTk3Im6i+Q6yOvAE4EeZuSgizgReExE3AD/JzPkPr2EALHmVyjuAwYTUWn27Kj1uJpb64z6a362JJS0vuwCrZeaJAJn5YEQcBjwjIvYFDqUpSb0SGMjMhRHxPOAomjfye4A3ZubtQw8aEdsCH6NJlN4LHAL8od1vZ5q7ElcC7wXOBTaIiNnAIZn58/5esiRJ41+tdQWaO/9vBF5M83f31FrrRjRtJk8HvgK8p5TiHBaSujRjsBVuuIjYiuYG88zM/FVE7A7s1b58CvBZ4Ebg5MXtn4tfpfIM4DsR8Srgt4DdEWOUiaX+GKxYWq3rQDRpbAHcMHQgM6+NiPtpJrl7AfBXmrueB0fEV4EKvCozfxoRbwGOBfYZ3D8i1gK+QPMH5O6IeCVNQulTwB6ZuXm73WHAOpk5va2Qmt7na5UkadyrtW5B0+bxdGAasDrwa+CsWut04Ms0NynfWUr5ekdhStLSmgbc3CaVpgIH0dyYJjOviYgNgDUz8+3LcMz3Ay/OzJtGPVqNKhNL/XEPD7fCScvDAmBxSw3PBC7KzPsAIuI04PXA94F5mfnTdrtTgU8O23dbYGPgzLZUdYX252YgI+Js4HLg4sy8dXQvR5KkCe924B+ALYeM3UhTufR+4C5g71LKjzuITZKW1fnAfhFxFXA3TbXRrkNe/xZNt8Oy+CFwVkT8kWYe458CH83Mu0YhXo0iE0v9cS8Pt8JJy8P1NO1uD4mIFw4+HLbtwiWMD09MTaHpi95t+MkiYkuaxNPzgNMi4hOZ+bWRBC5J0mRUSrmz1no2zYpKKwN/A1YBPgpcB+xZSrmzwxAl6SGZOfy7w+D4kUOe7j3s5eOGPH4OMLC4/TJz2pDHtwDT23ma9gGe007zMZWme+IA4PMjuQb1j6vC9YcVS1quMnMOcG9EvAEgIlamKa9fA9glIgbbMvcGzgF+BqweEVu04/u040NdAfxjRExrj7lrRHwhIrYBPpKZszPzaJoVH7Zq95kaEa6GKEnSY6i1HkFzU+hq4Jc0N35eBJwNzDCpJGkiiIjtIuIHwA2ZefPS7peZf6WZw/Wydg7Xy4BNaabz0BhjxVJ/3EuTtFuj60A0qewFHN/Ol7SQZmnOYyLiDmB2RDxAs6LMSZmZEfFa4KsR8SBwG/CWoQfLzD9HxEHANyPibzQlrW8C7gRe25a5LqD5//317W7/D7g2Iv4zM4feoZAkSa1a62doFsS4EjicpiUd4OOllE93FpgkjbLM/D5Np8NI9nWy7nEiMl1cYrTVWrcGfgDsX0oxoypJkiRqrQF8Efg3YDbwTZo5DhcAbymlXNhddJIkjYwVS/1xPzAfeELXgUiSJKl7tdYVaCp7Xw2cB/yGJsn0G2CvUsot3UUnSdLIOcdSf9xPc+fJxJIkSdIkV2udQlOd9GrgO8BKwNtoKtx3NKkkSRrPrFjqDyuWJEmSRK11Ks2y2y8B/otmZaSnAycA7y6lOC+FJGlcM7HUH/NoEktrdh2IJEmSulFrXYVm1dUX0Kz2tj3NqsHvKqV8rcvYJEkaLSaW+mOwFc5V4SRJkiahWusaNEtlbwXMAWYAdwH7lFKu7TI2SZJGk4ml/pjf/qzadSCSJElavmqtawEXAJsDNwE7AtcBry6l/LnL2CRpomkT+R8HDi+lzOs6nsnIybv7oO2VXwBM7ToWSZIkLT+11icDs4CnAbfTzKl0DjDDpJIk9cXzgdcBX+06kMnKxFL/zMeKMEmSpEmj1roRTVJpY5o5N9cFPlFK2b+UMr/T4CRpgiqlzKFZHOFfaq0HdB3PZGRiqX/mY8WSJEnSpFBr3Ry4EHgKkMBC4IBSyqc6DUySJoe3AT8HPtwm+bUcmVjqH1vhJEmSJoFa61bAecAGNBXrvwdmllIu6DQwSZokSikP0rTDrQScWmud0nFIk4qJpf6xYkmSJGmCq7W+CPgOzefqL9GsALdjKeU3nQYmSZNMKeVG4FjgeYDVosuRcwD1j3MsSZIkTWC11hnAicD9wJ6llJs6DkmSJrsesBPwulrrWaWUy7sOaDKwYql/FmBiSZIkaUKqte4BfA2YC7zcpJIkda9doX0/4E7gS7XWNToOaVIwsdQ/tsJJkiRNQLXW/YHjgTuAXUopt3QbkSRpUCnlL8C7gacCJ3UbzeRgYql/TCxJkiRNMLXWQ4DPAbfSTND9x45DkiQNU0o5HzgdeGn7vq0+MrHUP7bCSZIkTSC11sOATwC/pEkq3dlxSJKkJXsncCPw/lrrtI5jmdBMLPWPFUuSJEkTRK31SOCDwHU07W/3dhuRJOnRlFLmA/vT5D1OqbVO6TikCcvEUv9YsSRJkjQB1Fp7wL8DVwC7llLmdRySJGkplFJ+BRwFbNX+qz4wsdQ/84GptdboOhBJkiQtu1pr1FqPAw4BvgfsXkp5oOOwJEnLoJRyLM17+P611hldxzMRmVjqn/tpWuFsh5MkSRpnaq0rAF+nWbb6HGCvUsqCbqOSJI3QAcBtwLG11id2HcxEY2Kpf+6jaYVbtetAJEmStPRqrSsC/wW8imZVof1KKYu6jUqSNFKllHuAdwDr0dw00CgysdQ/99FUK63WdSCSJElaOrXWlYCzgF2AE4CDSynZbVSSpMerlPI94BRg51rrO7qOZyIxsdQ/99AklqxYkiRJGgdqrasC5wHbAceUUv7dpJIkTSjvAa4HBmqtT+86mInCxFL/3IutcJIkSUstIj4eEXdGxIrt89kRMa39mT2K55kdEX+37HStdU3gQuD5wCdLKUeM1vkkSWNDKWUh8FpgEfCNWqtzIo8CE0v9cy9WLEmSJC2LlwO3ADv18ySZOT0zFw4+r7WuA1wMPAv4UCnl0/08vySpO6WU3wKfBJ4NHNNxOBOCiaX+uZumYsk5liRJkh5DRGwJ/BaowF6L2WTFiDghIi6PiIsiYuN2vz0i4rKIuDQiro2IGe34gRHx2Yg4KyKuiYijhpzrlvbf9ddbb73//sMf/vDbuXPnPv/b3/72NaWUL/X9YiVJnSqlHE9TpbpPrXW3ruMZ70ws9c99QAJP6DoQSWNTRGwaEedExI8i4uqIOHwZ9n2oLaT98vSGvgUqScvHXsDZ7c+eETH8c+rWwLGZuT1wBvCFdvzZwO6Z+RLgNcCHh+yzG7B/u+92EfHMoQfcbrvtpn/mM595xlOe8pR7V1999VJrXSUinjbqVyZJGosOBH4PHFVrXbvjWMY1E0v9cz8wH3hi14FIGnvaL0xnAd/JzK1pJoqdGRH7LuuxMvOkzPzqMpw7IqJGxCuX9VyS1EevAr6bmTfRTCmww7DXb8jM69vHpw15/TzgyxHxfeD/AU8dss+5mTk3MxP4Gc0y0wDUWp956KGHfmTRokXrHHXUUX/Zd9993wVsMWx/SdIEVUq5D3gL8CTglFprdBzSuGViqX/uBxZgxZKkxdsFWC0zTwTIzAeBw4CMiH0j4gcRcUVEHD04wWxEbNtWNp1Hc4eFdvzAiDiyfTw7Ig6JiFkR8ZOI2Lkdf1JEfDciLgOuA36QmWctzwuWpCWJiM2BTYCTI+ICmukEXjNss6Grsy2keb+cAlwAnJiZ2wGvAIZOyj1v2DGmAGy22WZTge/OnTt3g+OPP37OVVddtU1b8XT5sP0lSRNYKWXwpsQOwEDH4YxbJpb6Zx5WLElasi2AG4YOZOa1NEmfI4CXAtsCGwIHR8RU4GTgwMzcDbjzUY791MycAewLfLwdOwy4LjN3AApgtZKksWQvYCAzX56ZLwf2BvYEht49fsbgvEo0K/qcD6zR/sxpxw95rBPVWrd797vfvR7A0Ucffc1VV131X5l5f9smN31UrkaSNJ4cDvwYeGetdcuugxmPTCz1z2Ar3JpdByJpTFpAs8zpcDOBizLzvrZ14zRgV5qViuZm5k/b7c57lGN/E6BtJxls+7iDhyso13qcsUvSaNsdOHPwSWZeRbMQyouHbPM54Oi28vIVwLsz8x7gP4ArI2IWzXvdyks6ycEHH7w1cNr8+fMT+Jcbb7xxAHhnRFwKvK+NwXk2JGkSKaUsorkh+wBwUq11pY5DGndW7DqACWywFW6NrgORNCZdDxw6dCAiXjj4cNi2C3nkjYBH6wEf2vox2NLxXeCKiNgG+ANw8DJFK0l91E7IPXzsOcOGPrKEfT8DfGbI0Nfa8ZOGbnf66aefTTPh910bbrjhdqWUW0spAP888sglSRNBKeV/a60fAb4IfAl4Y8chjStWLPWPFUuSligz5wD3Dq7mFhErA5+gSUbvEhGrtZvuDZxDM+ns2hGxea31aWuttdaBy3jKdwP7ZeaLMvPVmfmrUbkQSRoHaq2vo/mi8CdgRinl1o5DkiSNMaWUrwPnAnvVWl/VdTzjiYmlPimlLKSpWLKMTtKS7AW8JiJ+RDM/yNmZ+UngKGB22+5xM3BSZj4AHBAR9YMf/OD1z3ve87ZbxnP9EDg+IuZExMUR8ZWI2GQ0L0aSxqJa69tp3ld/S5NUur3jkCRJY9cbaP5efLrWum7XwYwX0UzhoX6otf4M+EkpZZ+uY5E0/rVLoJ5BsyT3rFLKzKXdNyJ+BWydmXdHxIrAu4CNMvNd/YlWkrpXa30fzdxJPwV2K6XM7TgkSdIYV2vdhqZj4HrgZaUUkyaPwYql/poPTO06CEkTxodoVosDmFZrXeIEtYvxFeDCiLgEuKw9zpdHOT5JGjNqrR+nWennWuClJpUkSUujlHI1zWfnbYEPdhzOuODk3f21ABNLkkZBrXUn4C3A4NxLGwMvAS5cmv0z89PAp/sTnSSNHW1159HA62kS6XuWUh7sNipJ0jjzYWA74K211nNKKdd1HdBYZsVSf1mxJGm0bAZcB9wK3EdzY2C/TiOSpDGmTSp9hWaOjAuBV5pUkiQtq1LKImBf4K/AibXWVToOaUwzsdRf87EqTNIoKKWcAOxGk1S6HHg1zaoVkiSg1roCcArNaprfAUopZUG3UUmSxqtSyp+AI2hu8DqFxKMwsdRftsJJGk0bAxsAPyylnFVKqV0HJEljQa11ReDbwO40yaXXtXebJUkasVLKN4GzgT1qrS7KtQQmlvrLiiVJo2k6zRxL53cchySNGe1CBt8FdgaOL6W82RV8JEmj6E3AzcDHaq3rdx3MWGRiqb+cY0nSaJoB3EYz15IkTXq11sFk+4uAo0spAx2HJEmaYEopfwMOAlYHTmtbrzWEv5D+WoAVS5JGz8bAH9s/bpI0qdVanwBcBDwX+Ggp5chuI5IkTVTtqnBfArYBPtJxOGOOiaX+smJJ0qhoVzraEPhj17FIUtdqrU8CZgFPBw4vpXyu45AkSRPfx4ErgINrrdt0HcxYYmKpv6xYkjRaptEkln7ccRyS1Kla64bAxcBGwLtKKa7UI0nqu3b+vtcC9wBfbtuxhYmlfrNiSdJo2RlYGSfuljSJ1Vr/gab97cnAm0opp3YckiRpEiml/Bl4H81N3692G83YYWKpv+YDU53cS9IomE7TBvc/HcchSV06A1gTOKCUck7XwUiSJp9SypnAt4B/qbUe0HU8Y4FtWn3QJpLWABbR/I6fW2tN4DqXv5U0QhvRTNz9QNeBSFKH3gwsLKX8qOtAJEmT2tuAfwI+XGv9Xinl910H1CUrafpjd+CX7b/rA7NpMprRYUySxikn7pakRinlhyaVJEldK6U8CBxIM1XFqZO9S2lSX3wfnQ3cDKxLU7G0JnByKWVRp1FJGq82o0ks+WVKkiRJGgNKKT8FjgWeD3yy43A6ZWKpD9p2t/8D3N0O/QL4THcRSRrnZgArARd0HYgkSZKkh3wWmAMcVGvdrutgumJiqU9KKRcBV7dPLymlzOsyHknj2hzgeOCnXQciSZIkqdEWlewH3An8Z611jVrrvrXWl3Uc2nIVmc4l3S+11i1p5lfaspRye8fhSJIkSZKkUVZr3Q34BnAb8BTgzFLKQd1GtfxYsdRHbc/lziaVJEmSJEmasP4I3AU8G1gL2KDbcJavFbsOYKIrpfxP1zFIkiRJkqTRV2tdGzgHeOqQ4Sd1FE4nrFiSJEmSJEkagVLKX4B/Ak4Cbm2H1621TppCHudYkiRJkiRJepxqrU8FPg9MB7Yvpfy824iWDyuWJKkDEbFpRJwTET+KiKsj4vBl2HdaRMzuY3iSJqCIyIi4NCJmR8RVEfGdiFjzMfY5MiIOHKXzfzMinvrYW0qSND6VUv63lPKvwLbAn7qOZ3mZNKVZkjRWRMQKwFnAsZl5YkSsBJwfEbdk5mkdhydpYpuRmQsAIuIbwBto7qwuk4h4GfD2zNx9affJzH2W9TySJI1HpZRfdh3D8mTFkiQtf7sAq2XmiQCZ+SBwGJARsW9E/CAiroiIoyNiCkBEbNtWNp0HHDh4oIhYKSKOiYjvta8f0sH1SBpnImJlYH3g1+3zW4a8ttiqyIg4oK2y/CFwNPCqiFg9Ik5uK6F+2FYlrTB4zIj4t7Y66sltpdS0R9tHkiSNP/4RfxS9iE17Eef0In7Ui7i69xitKr0hH8qGjR/ZG2EZeS/i30eyn6QxbQvghqEDmXktcB1wBPBSmvLZDYGDI2IqcDJwYGbuBtw5ZNf3Ar/NzJ2B7YADI+LZ/b8ESePUrDZp9BvgD5l5ztLsFBFrAMcCMzPzhcD1wPbA5sC5mfmSdvxBmnklBm2ZmS/IzDuGjD3WPpIkjUm9iOxFXNqLuKQX8cNee1P38XznX8rzntSLmN6v4z9eJpaWoPdwq8p3BjK3pvnCNrMXse9yDuWdy/l8kvpvAbBoMeMzgYsy875sVlY4DdgVeBYwNzN/2m533pB99gD2ab8oXgisDphYkrQkMzJzOrAxsEZEvGsp93sA+Gu7zxRgcG6m64FNI2JORFwKvIS/X275m4s51mPtI0nSWDZjIHMnYAbw3p43dZ1j6VHsAqw20LaqDGQ+2Is4DHhGm1w6FAjgSmBgIHPh0J17ER8DXg78DpgL3NKOP4umfHxVmi+Wbx/IvLE99r/RfHD7Nc2cB6cCG/SaL4wfHsi8tK9XLGl5uZ7mPeQhEfHCwYfDtl3II28CDN1mCvCWzLxqVCOUNKFl5sKI+DrwNpo5lhZFRLRJ7VUWs/38iLgImEPz2ebszLw0It5GkxjaIzPvjoijaN6XBt23mNO/9TH2kSRpzBvIvK8XcQ2w1dDxXsQRwMtoPrMvBF4LPBN460DmXu02H6T5TjAH+AKwEc0N4k8OZJ7ZazoWvgj8I3Az8ITlclEjZMXSkj2iVWXgUVpVhm7Xi3g5TUn3i4F9af4nohexIk07yyEDzd3CdwMn9iLWAo4Eth7I3Aa4GNhsIPOVwG0DmdNNKkkTR2bOAe6NiDfAQ3OdfAJYA9glIlZrN90bOAf4GbB2RGzejr9myOEuAPZvjzM1Is6LiH9YDpchafzbhub9BZpk0eAd1zcO3zAiNgS2yMynZeb0zPxc+9JmwBVtgmh94FVLcd6R7CNJ0pjSa/42vgi4esjYmjTJpB0GMrcDTgLeBFwCPK8X8cR2011pPscfBZw50ExrsQvw8V7Ek2gKTZ4wkLktzQ2Zf1wuFzVCViwt2aO2qgxk3gfQizgNeD1w3JBtdgHOGGhXXelFXNyOP6P9+XovHio4eBJNRdP1wH+32142kHn96F6OpDFmL+D4iHgLzR+fUzPzmIi4A5gdEQ8AlwInZWZGxAHANyPiPuDcIcf5OHBsRFxB8551Qmb+ZvleiqRxZFZEJM1nwDtpPrgCfAg4OSLuBk4Bth623x00Ce4f0FRXzwO+DRwDnBYRM2mWVf4vYO3HiGEk+0iSNFbM6jV/S6HBxPByAAADdElEQVTpQPr14Pf7gcy5vYibgPN7EavTVBr9aCBzUS/iTOA1vYgbgJ8MZM7vRexB0xX1jvZ4ATydJqdwcnvMe3sRly+/y1t2JpaW7BGtKr1Hb1UZakltK1OA37fVSn+nF7E98ALgecBRvYhzBjI/OoK4JY0Dmfk7mjsVw8dPAE5YzPhs/v6L3mfb8XksprpAkobLzOGfX4a+djnNZ5BBJ7TjRwJExMHAtzPz8Pb5k4CfAGdkczd1ccecNuz59CFPF7uPJEnjwIzBIpLhehFb0Ux9M3Mg81e9iN1pbihDc+Pms8CNtEkjmhzBawYybx92nEebCmPMsRVuCQbaVpVe26rSG9aq0ntkq8pQlwCv6EWs0G63Wzv+M2ClXsQL2mNu1Yv4Ti9iU5qKp6sGMr8EfA54brvPvF7EOv25SkmSpKUyC9gmIi6LiDk0iwh8NTPv7jguSZLGkmnAzW1SaSpw0OALA5nXABsALxzIvLIdfmhai17Emr2IOW073SU0i/TQi1iXMb56qhVLj24v4PjekFaVgcxjem2rSm9Iq8rQnQYyz24rkK4E7gYub8fn9yL+FTimF7GIppT87QOZv+1F3Axc2R5zIU0fJTRL+17ei6gD7V1DSZKk5Skzb6aZDkCSJC3Z+cB+vYiraHIBZ/D3XQrf4u87nt5Jk3PYs33+qbad7njguF4z3cUdwPf6H/rIRbP4hyRJkiRJkvqlF3EGzaryN3cdy2iyYkmSJEmSJKlPehHb0cyvdPFESyqBFUuSJEmSJEkaISfvliRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiJhYkiRJkiRJ0oiYWJIkSZIkSdKImFiSJEmSJEnSiPx/gOBXVkR9+YoAAAAASUVORK5CYII=\n",
3213 "text/plain": [
3214 "<Figure size 1152x648 with 12 Axes>"
3215 ]
3216 },
3217 "metadata": {},
3218 "output_type": "display_data"
3219 }
3220 ],
3221 "source": [
3222 "fig, axes = plt.subplots(nrows=3, ncols=4, figsize=(16, 9))\n",
3223 "axes = axes.flatten()\n",
3224 "fc = ec = 'darkgrey'\n",
3225 "for s, (section, result) in enumerate(results.groupby('section')):\n",
3226 " if s > 11:\n",
3227 " continue\n",
3228 " \n",
3229 " df = result.drop('section', axis=1).apply(lambda x: x.map(word2id))\n",
3230 " most_similar_idx = find_most_similar_analogy(df)\n",
3231 " \n",
3232 " best_analogy = result.iloc[most_similar_idx, :4].tolist()\n",
3233 "\n",
3234 " analogy_idx = [words.index(word) for word in best_analogy]\n",
3235 " best_analogy = [a.capitalize() for a in best_analogy]\n",
3236 " \n",
3237 " coords = pd.DataFrame(vectors2D[analogy_idx]) # xy array\n",
3238 " \n",
3239 " xlim, ylim = get_plot_lims(coords)\n",
3240 " axes[s].set_xlim(xlim)\n",
3241 " axes[s].set_ylim(ylim)\n",
3242 "\n",
3243 " for i in [0, 2]:\n",
3244 " axes[s].annotate(s=best_analogy[i], xy=coords.iloc[i+1], xytext=coords.iloc[i],\n",
3245 " arrowprops=dict(width=1,headwidth=5, headlength=5,\n",
3246 " fc=fc, ec=ec, shrink=.1),\n",
3247 " fontsize=12)\n",
3248 " \n",
3249 " axes[s].annotate(best_analogy[i+1], xy=coords.iloc[i+1],\n",
3250 " xytext=coords.iloc[i+1],\n",
3251 " va='center', ha='center',\n",
3252 " fontsize=12, color='darkred' if i == 2 else 'k');\n",
3253 "\n",
3254 " axes[s].axis('off')\n",
3255 " title = ' '.join([s.capitalize()\n",
3256 " for s in section.split('-') if not s.startswith('gram')])\n",
3257 " axes[s].set_title(title, fontsize=16)\n",
3258 "\n",
3259 "fig.tight_layout();"
3260 ]
3261 },
3262 {
3263 "cell_type": "markdown",
3264 "metadata": {},
3265 "source": [
3266 "## Resources\n",
3267 "\n",
3268 "- [Distributed representations of words and phrases and their compositionality](http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)\n",
3269 "- [Efficient estimation of word representations in vector space](https://arxiv.org/pdf/1301.3781.pdf?)\n",
3270 "- [Sebastian Ruder's Blog](http://ruder.io/word-embeddings-1/)"
3271 ]
3272 }
3273 ],
3274 "metadata": {
3275 "celltoolbar": "Slideshow",
3276 "kernelspec": {
3277 "display_name": "Python 3",
3278 "language": "python",
3279 "name": "python3"
3280 },
3281 "language_info": {
3282 "codemirror_mode": {
3283 "name": "ipython",
3284 "version": 3
3285 },
3286 "file_extension": ".py",
3287 "mimetype": "text/x-python",
3288 "name": "python",
3289 "nbconvert_exporter": "python",
3290 "pygments_lexer": "ipython3",
3291 "version": "3.6.8"
3292 },
3293 "toc": {
3294 "base_numbering": 1,
3295 "nav_menu": {},
3296 "number_sections": true,
3297 "sideBar": true,
3298 "skip_h1_title": true,
3299 "title_cell": "Table of Contents",
3300 "title_sidebar": "Contents",
3301 "toc_cell": false,
3302 "toc_position": {
3303 "height": "47px",
3304 "left": "38px",
3305 "right": "1340px",
3306 "top": "66.5px",
3307 "width": "457px"
3308 },
3309 "toc_section_display": true,
3310 "toc_window_display": true
3311 }
3312 },
3313 "nbformat": 4,
3314 "nbformat_minor": 2
3315 }