Assignment/C1_W3_Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 3: Hello Vectors\n",
    "\n",
    "Welcome to this week's programming assignment of the specialization. In this assignment we will explore word vectors.\n",
    "In natural language processing, we represent each word as a vector consisting of numbers.\n",
    "The vector encodes the meaning of the word. These numbers (or weights) for each word are learned using various machine\n",
    "learning models, which we will explore in more detail later in this specialization. Rather than make you code the\n",
    "machine learning models from scratch, we will show you how to use them. In the real world, you can always load the\n",
    "trained word vectors, and you will almost never have to train them from scratch. In this assignment you will\n",
    "\n",
    "- Predict analogies between words.\n",
    "- Use PCA to reduce the dimensionality of the word embeddings and plot them in two dimensions.\n",
    "- Compare word embeddings by using a similarity measure (the cosine similarity).\n",
    "- Understand how these vector space models work.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important Note on Submission to the AutoGrader\n",
    "\n",
    "Before submitting your assignment to the AutoGrader, please make sure you are not doing the following:\n",
    "\n",
    "1. You have not added any _extra_ `print` statement(s) in the assignment.\n",
    "2. You have not added any _extra_ code cell(s) in the assignment.\n",
    "3. You have not changed any of the function parameters.\n",
    "4. You are not using any global variables inside your graded exercises. Unless specifically instructed to do so, please refrain from it and use the local variables instead.\n",
    "5. You are not changing the assignment code where it is not required, like creating _extra_ variables.\n",
    "\n",
    "If you do any of the following, you will get something like, `Grader Error: Grader feedback not found` (or similarly unexpected) error upon submitting your assignment. Before asking for help/debugging the errors in your assignment, check for these first. If this is the case, and you don't remember the changes you have made, you can get a fresh copy of the assignment by following these [instructions](https://www.coursera.org/learn/classification-vector-spaces-in-nlp/supplement/YLuAg/h-ow-to-refresh-your-workspace)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "- [1 - Predict the Countries from Capitals](#1)\n",
    "    - [1.1 Importing the Data](#1-1)\n",
    "    - [1.2 Cosine Similarity](#1-2)\n",
    "        - [Exercise 1 - cosine_similarity (UNQ_C1)](#ex-1)\n",
    "    - [1.3 Euclidean Distance](#1-3)\n",
    "        - [Exercise 2 - euclidean (UNQ_C2)](#ex-2)\n",
    "    - [1.4 Finding the Country of each Capital](#1-4)\n",
    "        - [Exercise 3 - get_country (UNQ_C3)](#ex-3)\n",
    "    - [1.5 Model Accuracy](#1-5)\n",
    "        - [Exercise 4 - get_accuracy (UNQ_C4)](#ex-4)\n",
    "- [2 - Plotting the vectors using PCA](#2)\n",
    "    - [Exercise 5 - compute_pca (UNQ_C5)](#ex-5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1'></a>\n",
    "## 1 - Predict the Countries from Capitals\n",
    "\n",
    "During the presentation of the module, we have illustrated the word analogies\n",
    "by finding the capital of a country from the country. In this part of the assignment\n",
    "we have changed the problem a bit. You are asked to predict the **countries** \n",
    "that correspond to some **capitals**.\n",
    "You are playing trivia against some second grader who just took their geography test and knows all the capitals by heart.\n",
    "Thanks to NLP, you will be able to answer the questions properly. In other words, you will write a program that can give\n",
    "you the country by its capital. That way you are pretty sure you will win the trivia game. We will start by exploring the data set.\n",
    "\n",
    "<img src = './images/map.jpg' width=\"width\" height=\"height\" style=\"width:467px;height:300px;\"/>\n",
    "\n",
    "<a name='1-1'></a>\n",
    "###  1.1 Importing the Data\n",
    "\n",
    "As usual, you start by importing some essential Python libraries and the load dataset.\n",
    "The dataset will be loaded as a [Pandas DataFrame](https://pandas.pydata.org/pandas-docs/stable/getting_started/dsintro.html),\n",
    "which is very a common method in data science. Because of the large size of the data,\n",
    "this may take a few minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to import packages.\n",
    "import pickle\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import w3_unittest\n",
    "\n",
    "from utils import get_vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>city1</th>\n",
       "      <th>country1</th>\n",
       "      <th>city2</th>\n",
       "      <th>country2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Athens</td>\n",
       "      <td>Greece</td>\n",
       "      <td>Bangkok</td>\n",
       "      <td>Thailand</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Athens</td>\n",
       "      <td>Greece</td>\n",
       "      <td>Beijing</td>\n",
       "      <td>China</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Athens</td>\n",
       "      <td>Greece</td>\n",
       "      <td>Berlin</td>\n",
       "      <td>Germany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Athens</td>\n",
       "      <td>Greece</td>\n",
       "      <td>Bern</td>\n",
       "      <td>Switzerland</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Athens</td>\n",
       "      <td>Greece</td>\n",
       "      <td>Cairo</td>\n",
       "      <td>Egypt</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    city1 country1    city2     country2\n",
       "0  Athens   Greece  Bangkok     Thailand\n",
       "1  Athens   Greece  Beijing        China\n",
       "2  Athens   Greece   Berlin      Germany\n",
       "3  Athens   Greece     Bern  Switzerland\n",
       "4  Athens   Greece    Cairo        Egypt"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv('./data/capitals.txt', delimiter=' ')\n",
    "data.columns = ['city1', 'country1', 'city2', 'country2']\n",
    "\n",
    "# print first five elements in the DataFrame\n",
    "data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "#### To Run This Code On Your Own Machine:\n",
    "Note that because the original google news word embedding dataset is about 3.64 gigabytes,\n",
    "the workspace is not able to handle the full file set.  So we've downloaded the full dataset,\n",
    "extracted a sample of the words that we're going to analyze in this assignment, and saved\n",
    "it in a pickle file called word_embeddings_capitals.p\n",
    "\n",
    "If you want to download the full dataset on your own and choose your own set of word embeddings,\n",
    "please see the instructions and some helper code.\n",
    "\n",
    "- Download the dataset from this [page](https://code.google.com/archive/p/word2vec/).\n",
    "- Search in the page for 'GoogleNews-vectors-negative300.bin.gz' and click the link to download.\n",
    "- You'll need to unzip the file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy-paste the code below and run it on your local machine after downloading\n",
    "the dataset to the same directory as the notebook.\n",
    "\n",
    "```python\n",
    "import nltk\n",
    "from gensim.models import KeyedVectors\n",
    "\n",
    "\n",
    "embeddings = KeyedVectors.load_word2vec_format('./GoogleNews-vectors-negative300.bin', binary = True)\n",
    "f = open('capitals.txt', 'r').read()\n",
    "set_words = set(nltk.word_tokenize(f))\n",
    "select_words = words = ['king', 'queen', 'oil', 'gas', 'happy', 'sad', 'city', 'town', 'village', 'country', 'continent', 'petroleum', 'joyful']\n",
    "for w in select_words:\n",
    "    set_words.add(w)\n",
    "\n",
    "def get_word_embeddings(embeddings):\n",
    "\n",
    "    word_embeddings = {}\n",
    "    for word in embeddings.vocab:\n",
    "        if word in set_words:\n",
    "            word_embeddings[word] = embeddings[word]\n",
    "    return word_embeddings\n",
    "\n",
    "\n",
    "# Testing your function\n",
    "word_embeddings = get_word_embeddings(embeddings)\n",
    "print(len(word_embeddings))\n",
    "pickle.dump( word_embeddings, open( \"word_embeddings_subset.p\", \"wb\" ) )\n",
    "```\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will load the word embeddings as a [Python dictionary](https://docs.python.org/3/tutorial/datastructures.html#dictionaries).\n",
    "As stated, these have already been obtained through a machine learning algorithm. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "243"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_embeddings = pickle.load(open(\"./data/word_embeddings_subset.p\", \"rb\"))\n",
    "len(word_embeddings)  # there should be 243 words that will be used in this assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of the word embedding is a 300-dimensional vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dimension: 300\n"
     ]
    }
   ],
   "source": [
    "print(\"dimension: {}\".format(word_embeddings['Spain'].shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Predict relationships among words\n",
    "\n",
    "Now you will write a function that will use the word embeddings to predict relationships among words.\n",
    "* The function will take as input three words.\n",
    "* The first two are related to each other.\n",
    "* It will predict a 4th word which is related to the third word in a similar manner as the two first words are related to each other.\n",
    "* As an example, \"Athens is to Greece as Bangkok is to ______\"?\n",
    "* You will write a program that is capable of finding the fourth word.\n",
    "* We will give you a hint to show you how to compute this.\n",
    "\n",
    "A similar analogy would be the following:\n",
    "\n",
    "<img src = './images/vectors.jpg' width=\"width\" height=\"height\" style=\"width:467px;height:200px;\"/>\n",
    "\n",
    "You will implement a function that can tell you the capital of a country.\n",
    "You should use the same methodology shown in the figure above. To do this,\n",
    "you'll first compute the cosine similarity metric or the Euclidean distance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1-2'></a>\n",
    "### 1.2 Cosine Similarity\n",
    "\n",
    "The cosine similarity function is:\n",
    "\n",
    "$$\\cos (\\theta)=\\frac{\\mathbf{A} \\cdot \\mathbf{B}}{\\|\\mathbf{A}\\|\\|\\mathbf{B}\\|}=\\frac{\\sum_{i=1}^{n} A_{i} B_{i}}{\\sqrt{\\sum_{i=1}^{n} A_{i}^{2}} \\sqrt{\\sum_{i=1}^{n} B_{i}^{2}}}\\tag{1}$$\n",
    "\n",
    "$A$ and $B$ represent the word vectors and $A_i$ or $B_i$ represent index i of that vector. Note that if A and B are identical, you will get $cos(\\theta) = 1$.\n",
    "* Otherwise, if they are the total opposite, meaning, $A= -B$, then you would get $cos(\\theta) = -1$.\n",
    "* If you get $cos(\\theta) =0$, that means that they are orthogonal (or perpendicular).\n",
    "* Numbers between 0 and 1 indicate a similarity score.\n",
    "* Numbers between -1 and 0 indicate a dissimilarity score.\n",
    "\n",
    "<a name='ex-1'></a>\n",
    "### Exercise 1 - cosine_similarity\n",
    "Implement a function that takes in two word vectors and computes the cosine distance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li> Python's<a href=\"https://docs.scipy.org/doc/numpy/reference/\" > NumPy library </a> adds support for linear algebra operations (e.g., dot product, vector norm ...).</li>\n",
    "    <li>Use <a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.dot.html\" > numpy.dot </a>.</li>\n",
    "    <li>Use <a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html\">numpy.linalg.norm </a>.</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C1 GRADED FUNCTION: cosine_similarity\n",
    "\n",
    "def cosine_similarity(A, B):\n",
    "    '''\n",
    "    Input:\n",
    "        A: a numpy array which corresponds to a word vector\n",
    "        B: A numpy array which corresponds to a word vector\n",
    "    Output:\n",
    "        cos: numerical number representing the cosine similarity between A and B.\n",
    "    '''\n",
    "\n",
    "    ### START CODE HERE ###\n",
    "    dot = np.dot(A,B)    \n",
    "    norma = np.sqrt(np.dot(A,A))\n",
    "    normb = np.sqrt(np.dot(B,B))   \n",
    "    cos = dot/(norma*normb)\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "    return cos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6510956"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# feel free to try different words\n",
    "king = word_embeddings['king']\n",
    "queen = word_embeddings['queen']\n",
    "\n",
    "cosine_similarity(king, queen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**:\n",
    "\n",
    "$\\approx$ 0.651095"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Test your function\n",
    "w3_unittest.test_cosine_similarity(cosine_similarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1-3'></a>\n",
    "### 1.3 Euclidean Distance\n",
    "\n",
    "You will now implement a function that computes the similarity between two vectors using the Euclidean distance.\n",
    "Euclidean distance is defined as:\n",
    "\n",
    "$$ \\begin{aligned} d(\\mathbf{A}, \\mathbf{B})=d(\\mathbf{B}, \\mathbf{A}) &=\\sqrt{\\left(A_{1}-B_{1}\\right)^{2}+\\left(A_{2}-B_{2}\\right)^{2}+\\cdots+\\left(A_{n}-B_{n}\\right)^{2}} \\\\ &=\\sqrt{\\sum_{i=1}^{n}\\left(A_{i}-B_{i}\\right)^{2}} \\end{aligned}$$\n",
    "\n",
    "* $n$ is the number of elements in the vector\n",
    "* $A$ and $B$ are the corresponding word vectors. \n",
    "* The more similar the words, the more likely the Euclidean distance will be close to 0. \n",
    "\n",
    "<a name='ex-2'></a>\n",
    "### Exercise 2 - euclidean\n",
    "Implement a function that computes the Euclidean distance between two vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>    \n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>Use <a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html\" > numpy.linalg.norm </a>.</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C2 GRADED FUNCTION: euclidean\n",
    "\n",
    "def euclidean(A, B):\n",
    "    \"\"\"\n",
    "    Input:\n",
    "        A: a numpy array which corresponds to a word vector\n",
    "        B: A numpy array which corresponds to a word vector\n",
    "    Output:\n",
    "        d: numerical number representing the Euclidean distance between A and B.\n",
    "    \"\"\"\n",
    "\n",
    "    ### START CODE HERE ###\n",
    "\n",
    "    # euclidean distance    \n",
    "    d = np.sqrt(np.sum((A-B)**2))\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "\n",
    "    return d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.4796925"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Test your function\n",
    "euclidean(king, queen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output:**\n",
    "\n",
    "2.4796925"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Test your function\n",
    "w3_unittest.test_euclidean(euclidean)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1-4'></a>\n",
    "### 1.4 Finding the Country of each Capital\n",
    "\n",
    "Now, you  will use the previous functions to compute similarities between vectors,\n",
    "and use these to find the capital cities of countries. You will write a function that\n",
    "takes in three words, and the embeddings dictionary. Your task is to find the\n",
    "capital cities. For example, given the following words: \n",
    "\n",
    "- 1: Athens 2: Greece 3: Baghdad,\n",
    "\n",
    "your task is to predict the country 4: Iraq.\n",
    "\n",
    "<a name='ex-3'></a>\n",
    "### Exercise 3 - get_country\n",
    "**Instructions**: \n",
    "\n",
    "1. To predict the capital you might want to look at the *King - Man + Woman = Queen* example above, and implement that scheme into a mathematical function, using the word embeddings and a similarity function.\n",
    "\n",
    "2. Iterate over the embeddings dictionary and compute the cosine similarity score between your vector and the current word embedding.\n",
    "\n",
    "3. You should add a check to make sure that the word you return is not any of the words that you fed into your function. Return the one with the highest score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C3 GRADED FUNCTION: get_country\n",
    "\n",
    "def get_country(city1, country1, city2, embeddings, cosine_similarity=cosine_similarity):\n",
    "    \"\"\"\n",
    "    Input:\n",
    "        city1: a string (the capital city of country1)\n",
    "        country1: a string (the country of capital1)\n",
    "        city2: a string (the capital city of country2)\n",
    "        embeddings: a dictionary where the keys are words and values are their emmbeddings\n",
    "    Output:\n",
    "        country: a tuple with the most likely country and its similarity score\n",
    "    \"\"\"\n",
    "    ### START CODE HERE ###\n",
    "\n",
    "    # store the city1, country 1, and city 2 in a set called group\n",
    "    group = set((city1, country1, city2))\n",
    "\n",
    "    # get embeddings of city 1\n",
    "    city1_emb = embeddings[city1]\n",
    "\n",
    "    # get embedding of country 1\n",
    "    country1_emb = embeddings[country1]\n",
    "\n",
    "    # get embedding of city 2\n",
    "    city2_emb = embeddings[city2]\n",
    "\n",
    "    # get embedding of country 2 (it's a combination of the embeddings of country 1, city 1 and city 2)\n",
    "    # Remember: King - Man + Woman = Queen\n",
    "    vec = country1_emb-city1_emb+city2_emb\n",
    "\n",
    "    # Initialize the similarity to -1 (it will be replaced by a similarities that are closer to +1)\n",
    "    similarity = -1\n",
    "\n",
    "    # initialize country to an empty string\n",
    "    country = ''\n",
    "\n",
    "    # loop through all words in the embeddings dictionary\n",
    "    for word in embeddings.keys():\n",
    "\n",
    "        # first check that the word is not already in the 'group'\n",
    "        if word not in group:\n",
    "\n",
    "            # get the word embedding\n",
    "            word_emb = embeddings[word]\n",
    "\n",
    "            # calculate cosine similarity between embedding of country 2 and the word in the embeddings dictionary\n",
    "            cur_similarity = cosine_similarity(vec,word_emb)\n",
    "\n",
    "            # if the cosine similarity is more similar than the previously best similarity...\n",
    "            if cur_similarity > similarity:\n",
    "\n",
    "                # update the similarity to the new, better similarity\n",
    "                similarity = cur_similarity\n",
    "\n",
    "                # store the country as a tuple, which contains the word and the similarity\n",
    "                country = (word,similarity)\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "\n",
    "    return country"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Egypt', 0.7626822)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Testing your function, note to make it more robust you can return the 5 most similar words.\n",
    "get_country('Athens', 'Greece', 'Cairo', word_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output: (Approximately)**\n",
    "\n",
    "('Egypt', 0.7626821)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Test your function\n",
    "w3_unittest.test_get_country(get_country)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1-5'></a>\n",
    "### 1.5 Model Accuracy\n",
    "\n",
    "Now you will test your new function on the dataset and check the accuracy of the model:\n",
    "\n",
    "$$\\text{Accuracy}=\\frac{\\text{Correct # of predictions}}{\\text{Total # of predictions}}$$\n",
    "\n",
    "<a name='ex-4'></a>\n",
    "### Exercise 4 - get_accuracy\n",
    "**Instructions**: Implement a program that can compute the accuracy on the dataset provided for you. You have to iterate over every row to get the corresponding words and feed them into you `get_country` function above. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>Use <a href=\"https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.iterrows.html\" > pandas.DataFrame.iterrows </a>.</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C4 GRADED FUNCTION: get_accuracy\n",
    "\n",
    "def get_accuracy(word_embeddings, data, get_country=get_country):\n",
    "    '''\n",
    "    Input:\n",
    "        word_embeddings: a dictionary where the key is a word and the value is its embedding\n",
    "        data: a pandas DataFrame containing all the country and capital city pairs\n",
    "\n",
    "    '''\n",
    "\n",
    "    ### START CODE HERE ###\n",
    "    # initialize num correct to zero\n",
    "    num_correct = 0\n",
    "\n",
    "    # loop through the rows of the dataframe\n",
    "    for i, row in data.iterrows():\n",
    "\n",
    "        # get city1\n",
    "        city1 = row['city1']\n",
    "\n",
    "        # get country1\n",
    "        country1 = row['country1']\n",
    "\n",
    "        # get city2\n",
    "        city2 = row['city2']\n",
    "\n",
    "        # get country2\n",
    "        country2 = row['country2']\n",
    "\n",
    "        # use get_country to find the predicted country2\n",
    "        predicted_country2, _ = get_country(city1,country1,city2,word_embeddings)\n",
    "\n",
    "        # if the predicted country2 is the same as the actual country2...\n",
    "        if predicted_country2 == country2:\n",
    "            # increment the number of correct by 1\n",
    "            num_correct += 1\n",
    "\n",
    "    # get the number of rows in the data dataframe (length of dataframe)\n",
    "    m = len(data)\n",
    "\n",
    "    # calculate the accuracy by dividing the number correct by m\n",
    "    accuracy = num_correct /m\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "    return accuracy\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE: The cell below takes about 30 SECONDS to run.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy is 0.92\n"
     ]
    }
   ],
   "source": [
    "accuracy = get_accuracy(word_embeddings, data)\n",
    "print(f\"Accuracy is {accuracy:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output:**\n",
    "\n",
    "$\\approx$ 0.92"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Test your function\n",
    "w3_unittest.test_get_accuracy(get_accuracy, data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2'></a>\n",
    "## 2 - Plotting the vectors using PCA\n",
    "\n",
    "Now you will explore the distance between word vectors after reducing their dimension.\n",
    "The technique we will employ is known as\n",
    "[*principal component analysis* (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis).\n",
    "As we saw, we are working in a 300-dimensional space in this case.\n",
    "Although from a computational perspective we were able to perform a good job,\n",
    "it is impossible to visualize results in such high dimensional spaces.\n",
    "\n",
    "You can think of PCA as a method that projects our vectors in a space of reduced\n",
    "dimension, while keeping the maximum information about the original vectors in\n",
    "their reduced counterparts. In this case, by *maximum infomation* we mean that the\n",
    "Euclidean distance between the original vectors and their projected siblings is\n",
    "minimal. Hence vectors that were originally close in the embeddings dictionary,\n",
    "will produce lower dimensional vectors that are still close to each other.\n",
    "\n",
    "You will see that when you map out the words, similar words will be clustered\n",
    "next to each other. For example, the words 'sad', 'happy', 'joyful' all describe\n",
    "emotion and are supposed to be near each other when plotted.\n",
    "The words: 'oil', 'gas', and 'petroleum' all describe natural resources.\n",
    "Words like 'city', 'village', 'town' could be seen as synonyms and describe a\n",
    "similar thing.\n",
    "\n",
    "Before plotting the words, you need to first be able to reduce each word vector\n",
    "with PCA into 2 dimensions and then plot it. The steps to compute PCA are as follows:\n",
    "\n",
    "1. Mean normalize the data\n",
    "2. Compute the covariance matrix of your data ($\\Sigma$). \n",
    "3. Compute the eigenvectors and the eigenvalues of your covariance matrix\n",
    "4. Multiply the first K eigenvectors by your normalized data. The transformation should look something as follows:\n",
    "\n",
    "<img src = './images/word_embf.jpg' width=\"width\" height=\"height\" style=\"width:800px;height:200px;\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex-5'></a>\n",
    "### Exercise 5 - compute_pca\n",
    "\n",
    "**Instructions**: \n",
    "\n",
    "Implement a program that takes in a data set where each row corresponds to a word vector. \n",
    "* The word vectors are of dimension 300. \n",
    "* Use PCA to change the 300 dimensions to `n_components` dimensions. \n",
    "* The new matrix should be of dimension `m, n_components`. \n",
    "\n",
    "* First de-mean the data\n",
    "* Get the eigenvalues using `linalg.eigh`.  Use 'eigh' rather than 'eig' since R is symmetric.  The performance gain when using eigh instead of eig is substantial.\n",
    "* Sort the eigenvectors and eigenvalues by decreasing order of the eigenvalues.\n",
    "* Get a subset of the eigenvectors (choose how many principle components you want to use using n_components).\n",
    "* Return the new transformation of the data by multiplying the eigenvectors with the original data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>\n",
    "    <font size=\"3\" color=\"darkgreen\"><b>Hints</b></font>\n",
    "</summary>\n",
    "<p>\n",
    "<ul>\n",
    "    <li>Use <a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.mean.html\" > numpy.mean(a,axis=None) </a> which takes one required parameter. You need to specify the optional argument axis for this exercise: If you set <code>axis = 0</code>, you take the mean for each column.  If you set <code>axis = 1</code>, you take the mean for each row.  Remember that each row is a word vector, and the number of columns are the number of dimensions in a word vector. </li>\n",
    "    <li>Use <a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.cov.html\" > numpy.cov(m, rowvar=True) </a> which takes one required parameter. You need to specify the optional argument rowvar for this exercise. This calculates the covariance matrix.  By default <code>rowvar</code> is <code>True</code>.  From the documentation: \"If rowvar is True (default), then each row represents a variable, with observations in the columns.\"  In our case, each row is a word vector observation, and each column is a feature (variable). </li>\n",
    "    <li>Use <a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.eigh.html\" > numpy.linalg.eigh(a, UPLO='L') </a> </li>\n",
    "    <li>Use <a href=\"https://docs.scipy.org/doc/numpy/reference/generated/numpy.argsort.html\" > numpy.argsort </a> sorts the values in an array from smallest to largest, then returns the indices from this sort. </li>\n",
    "    <li>In order to reverse the order of a list, you can use: <code>x[::-1]</code>.</li>\n",
    "    <li>To apply the sorted indices to eigenvalues, you can use this format <code>x[indices_sorted]</code>.</li>\n",
    "    <li>When applying the sorted indices to eigen vectors, note that each column represents an eigenvector.  In order to preserve the rows but sort on the columns, you can use this format <code>x[:,indices_sorted]</code></li>\n",
    "    <li>To transform the data using a subset of the most relevant principle components, take the matrix multiplication of the eigenvectors with the original data.  </li>\n",
    "    <li>The data is of shape <code>(n_observations, n_features)</code>.  </li>\n",
    "    <li>The subset of eigenvectors are in a matrix of shape <code>(n_features, n_components)</code>.</li>\n",
    "    <li>To multiply these together, take the transposes of both the eigenvectors <code>(n_components, n_features)</code> and the data (n_features, n_observations).</li>\n",
    "    <li>The product of these two has dimensions <code>(n_components,n_observations)</code>.  Take its transpose to get the shape <code>(n_observations, n_components)</code>.</li>\n",
    "</ul>\n",
    "</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# UNQ_C5 GRADED FUNCTION: compute_pca\n",
    "\n",
    "\n",
    "def compute_pca(X, n_components=2):\n",
    "    \"\"\"\n",
    "    Input:\n",
    "        X: of dimension (m,n) where each row corresponds to a word vector\n",
    "        n_components: Number of components you want to keep.\n",
    "    Output:\n",
    "        X_reduced: data transformed in 2 dims/columns + regenerated original data\n",
    "    pass in: data as 2D NumPy array\n",
    "    \"\"\"\n",
    "\n",
    "    ### START CODE HERE ###\n",
    "    # mean center the data\n",
    "    X_demeaned = X - np.mean(X,axis=0)\n",
    "\n",
    "    # calculate the covariance matrix\n",
    "    covariance_matrix = np.cov(X_demeaned, rowvar=False)\n",
    "\n",
    "    # calculate eigenvectors & eigenvalues of the covariance matrix\n",
    "    eigen_vals, eigen_vecs = np.linalg.eigh(covariance_matrix, UPLO='L')\n",
    "\n",
    "    # sort eigenvalue in increasing order (get the indices from the sort)\n",
    "    idx_sorted = np.argsort(eigen_vals)\n",
    "    \n",
    "    # reverse the order so that it's from highest to lowest.\n",
    "    idx_sorted_decreasing = idx_sorted[::-1]\n",
    "\n",
    "    # sort the eigen values by idx_sorted_decreasing\n",
    "    eigen_vals_sorted = eigen_vals[idx_sorted_decreasing]\n",
    "\n",
    "    # sort eigenvectors using the idx_sorted_decreasing indices\n",
    "    eigen_vecs_sorted = eigen_vecs[:,idx_sorted_decreasing]\n",
    "\n",
    "    # select the first n eigenvectors (n is desired dimension\n",
    "    # of rescaled data array, or n_components)\n",
    "    eigen_vecs_subset = eigen_vecs_sorted[:,0:n_components]\n",
    "\n",
    "    # transform the data by multiplying the transpose of the eigenvectors with the transpose of the de-meaned data\n",
    "    # Then take the transpose of that product.\n",
    "    X_reduced = np.dot(eigen_vecs_subset.transpose(),X_demeaned.transpose()).transpose()\n",
    "\n",
    "    ### END CODE HERE ###\n",
    "\n",
    "    return X_reduced\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your original matrix was (3, 10) and it became:\n",
      "[[ 0.43437323  0.49820384]\n",
      " [ 0.42077249 -0.50351448]\n",
      " [-0.85514571  0.00531064]]\n"
     ]
    }
   ],
   "source": [
    "# Testing your function\n",
    "np.random.seed(1)\n",
    "X = np.random.rand(3, 10)\n",
    "X_reduced = compute_pca(X, n_components=2)\n",
    "print(\"Your original matrix was \" + str(X.shape) + \" and it became:\")\n",
    "print(X_reduced)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "# Test your function\n",
    "w3_unittest.test_compute_pca(compute_pca)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output:**\n",
    "\n",
    "Your original matrix was: (3,10) and it became:\n",
    "\n",
    "<table>\n",
    "    <tr>\n",
    "        <td>\n",
    "           0.43437323\n",
    "            </td>\n",
    "                <td>\n",
    "            0.49820384\n",
    "            </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>\n",
    "            0.42077249\n",
    "            </td>\n",
    "                <td>\n",
    "           -0.50351448\n",
    "            </td>\n",
    "    </tr>\n",
    "    <tr>\n",
    "        <td>\n",
    "            -0.85514571\n",
    "            </td>\n",
    "                <td>\n",
    "           0.00531064\n",
    "            </td>\n",
    "    </tr>\n",
    "</table>\n",
    "\n",
    "Now you will use your pca function to plot a few words we have chosen for you.\n",
    "You will see that similar words tend to be clustered near each other.\n",
    "Sometimes, even antonyms tend to be clustered near each other. Antonyms\n",
    "describe the same thing but just tend to be on the other end of the scale\n",
    "They are usually found in the same location of a sentence,\n",
    "have the same parts of speech, and thus when\n",
    "learning the word vectors, you end up getting similar weights. In the next week\n",
    "we will go over how you learn them, but for now let's just enjoy using them.\n",
    "\n",
    "**Instructions:** Run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You have 11 words each of 300 dimensions thus X.shape is: (11, 300)\n"
     ]
    }
   ],
   "source": [
    "words = ['oil', 'gas', 'happy', 'sad', 'city', 'town',\n",
    "         'village', 'country', 'continent', 'petroleum', 'joyful']\n",
    "\n",
    "# given a list of words and the embeddings, it returns a matrix with all the embeddings\n",
    "X = get_vectors(word_embeddings, words)\n",
    "\n",
    "print('You have 11 words each of 300 dimensions thus X.shape is:', X.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAD8CAYAAABDwhLXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnt0lEQVR4nO3deXRV1fn/8fdjQIhCjQICQSQOlDJlkDAUGlAQUEtlLiIog4q0VTt8m29x0VVRcWml3zq0VItVUKGCBEjR2uKA/ATFSiIJo1S0UUkQIghCCRXh+f2RmzRgQgK5Ofcm+bzWuosz7HP2cy5ZebL32Wcfc3dERERq2hmRDkBEROoHJRwREQmEEo6IiARCCUdERAKhhCMiIoFQwhERkUDU+4RjZglmtinScYiI1HX1PuGIiEgwGkQ6gHAxs7OB54ELgBjgXqAD8D0gFngLuNXd3cy6AU+FDn05AuGKiNQ7Fs0zDTRv3twTEhKqVPbzzz/niy++oF27dgAcPXoUd6dBg+Kc+q9//Ytzzz2XuLg4tmzZQtu2bWnatCk7duxg//79dO7cuaYuQ0QkMNnZ2Z+5e4tIx1GeqG7hJCQkkJWVVaWy//znPxk0aBADBgxgyJAhpKWlsWTJEh588EEOHTpE48aNufnmm5k6dSqJiYls27YNgA0bNnD99ddXuR4RkWhmZh9FOoaKRHXCORXf/OY3effdd3nppZf45S9/yYABA5g9ezZZWVm0bduWGTNmcPjw4UiHKSJSb9WZQQMFBQWcddZZjB8/nvT0dN59910AmjdvzsGDB8nIyAAgLi6OuLg41qxZA8CCBQsiFrOISH1SZ1o4GzduJD09nTPOOIOGDRvy2GOPkZmZSZcuXWjVqhXdu3cvLTt37lwmT56MmTFo0KAIRi0iUn9E9aCB1NRU170VEZGqM7Nsd0+NdBzlqTNdaiIiEt3qTJdaicz1+cxasY2CfUXEx8WSPrgDw1LaRDosEZF6r04lnMz1+dy5dCNFR44CkL+viDuXbgRQ0hERibA61aU2a8W20mRToujIUWat2BahiEREpESdSjgF+4pOabuIiASnTiWc+LjYU9ouIiLBqVMJJ31wB2Ibxhy3LbZhDOmDO0QoIhERKVGnBg2UDAzQKDURkehTpxIOFCcdJRgRkehTp7rUREQkeinhiIhIIJRwREQkEPUy4fTu3fu0jhs7diyJiYk89NBDFZZZtWoVQ4YMOd3QRETqrDo3aKAq3nrrrVM+5tNPP2XdunVs3769BiISEan76mULp0mTJrg76enpdOnSha5du7Jo0SIAbrzxRjIzM0vLjhs3jr/85S8MGjSI/Px8kpOTWb16NZdffnnpa6k/++wzEhISInAlIiK1R71s4QAsXbqUnJwccnNz+eyzz+jevTt9+/blpptu4qGHHmLYsGHs37+ft956i6effpqkpCSGDBlCTk5OpEMXEamVwtLCMbOnzGy3mW2qYP/lZrbfzHJCn1+Fo97qWLNmDWPHjiUmJoaWLVvSr18/1q1bR79+/Xj//fcpLCzkueeeY+TIkTRoUG/zsohI2ITrN+k84PfAMycps9rda8Xd9BtvvJH58+ezcOFC5s6dW26ZBg0acOzYMQAOHz4cZHgiIrVSWFo47v4GsDcc5wpKWloaixYt4ujRoxQWFvLGG2/Qo0cPACZOnMjDDz8MQKdOnco9PiEhgezsbAAyMjICiVlEpDYLctDAt80s18z+ZmadA6z3a8yM4cOHk5iYSFJSEv379+fBBx+kVatWALRs2ZKOHTsyadKkCs/x85//nMcee4yUlBQ+++yzoEIXEam1zN3DcyKzBOBFd+9Szr5vAMfc/aCZXQM84u7tKzjPFGAKwIUXXtjto48+Ckt8Jfbs2cNll11GyXnz8vIYMmQImzb99/bToUOH6Nq1K++++y7nnHNOWOsXEalJZpbt7qmRjqM8gbRw3P0Ldz8YWn4JaGhmzSsoO8fdU909tUWLFmGNo6CggG9/+9v8/Oc/r7DMq6++SseOHbn99tuVbEREwiiQ4Vdm1grY5e5uZj0oTnR7gqi7rPj4eP75z38CkLk+n1krtvHRR3ns3X2AgSOup2BbLm3atOG9995j/vz5dO/enS+//JJLL72UZ599lrPOOouJEyfSuHFjsrKy+OKLL/jtb3/LkCFDmDdvHsuWLWP//v3k5+czfvx47rrrLn71q19x3nnn8ZOf/ASA6dOnc/755/PjH/846MsXEYmocA2Lfg5YC3Qwsx1mdpOZTTWzqaEio4BNZpYLPApc5+HqyzsNmevzuXPpRvJDr54u+mwH25v14b75LxMXF8eSJUsYMWIE69atIzc3l44dO/Lkk0+WHp+Xl8c777zDX//6V6ZOnVo6Su2dd95hyZIlbNiwgcWLF5OVlcXkyZN55pniwXvHjh1j4cKFjB8/PviLFhGJsLC0cNx9bCX7f0/xsOmoMGvFNoqOHC1dbxDXEm+WwKwV27i2Wzfy8vLYtGkTv/zlL9m3bx8HDx5k8ODBpeW///3vc8YZZ9C+fXsuvvhi3nvvPQAGDhxIs2bNABgxYgRr1qzhJz/5Cc2aNWP9+vXs2rWLlJSU0jIiIvVJvXyisSDUsilhMQ1Lt8c0j6GoqIiJEyeSmZlJUlIS8+bNY9WqVf8tb3b88aH1irbffPPNzJs3j08//ZTJkyeH+3JERGqFejmXWnxcbKXbDxw4QOvWrTly5AgLFiw4rtzixYs5duwYH3zwAR9++CEdOnQA4JVXXmHv3r0UFRWRmZlJnz59ABg+fDh///vfWbdu3XEtJRGR+qRetnDSB3fgzqUbj+tWi20YQ/rgDmx/7V0A7r33Xnr27EmLFi3o2bMnBw4cKC174YUX0qNHD7744gsef/xxGjduDECPHj0YOXIkO3bsYPz48aSmFo9MPPPMM7niiiuIi4sjJiYmwCsVEYke9TLhDEtpAxTfyymgJd3/Zy7pgzsUb0/575DpH/zgB+Uef+WVV/L4449/bfsFF1xw3EzTJY4dO8bbb7/N4sWLw3MBIiK1UL1MOFCcdEoST03asmULQ4YMYfjw4bRvX+6zriIi9ULYZhqoCampqV7yzpmaVPJMTsG+IuLjYv/b2hERqWWieaaBetvCKVHyTE7J/Zz8fUXcuXQjgJKOiEgY1ctRamWd+EwOQNGRo8xasS1CEYmI1E31PuGc+ExOZdtFROT01PuEU5VnckREpPrqfcJJH9yB2IbHPxtT8kyOiIiET70fNHDcMzkapSYiUmPqfcKB4J7JERGpz+p9l5qIiARDCUdERAKhhCMiIoFQwhERkUAo4YiISCCUcEREJBBKOCIiEgglHBERCURYEo6ZPWVmu81sUwX7zcweNbPtZrbBzC4LR70iIlJ7hKuFMw+46iT7rwbahz5TgMfCVK+IiNQSYUk47v4GsPckRYYCz3ixt4E4M2sdjrpFRKR2COoeThvgkzLrO0LbvsbMpphZlpllFRYWBhKciIjUvKgbNODuc9w91d1TW7RoEelwREQkTIJKOPlA2zLrF4S2iYhIPRFUwlkO3BgardYL2O/uOwOqW0REokBY3odjZs8BlwPNzWwHcBfQEMDdHwdeAq4BtgOHgEnhqFdERGqPsCQcdx9byX4HfhSOukREpHaKukEDIiJSNynhiIhIIJRwREQkEEo4IiISCCUcEREJhBKOiIgEQglHREQCoYQjIiKBUMIREZFAKOGIiEgglHBERCQQSjgiIhIIJRwREQmEEo6IiARCCUdERAKhhCMiIoFQwhERkUAo4YiISCCUcEREJBBKOCIiEoiwJBwzu8rMtpnZdjObVs7+iWZWaGY5oc/N4ahXRERqjwbVPYGZxQCzgYHADmCdmS139y0nFF3k7rdVtz4REamdwtHC6QFsd/cP3f1LYCEwNAznFRGRCsybN4+CgoJTPs7M8syseQ2EVKlwJJw2wCdl1neEtp1opJltMLMMM2tb0cnMbIqZZZlZVmFhYRjCExGpe06WcEI9T1EnqEEDLwAJ7p4IvAI8XVFBd5/j7qnuntqiRYuAwhMRiay8vDy+9a1vMW7cODp27MioUaM4dOgQ2dnZ9OvXj27dujF48GB27txJRkYGWVlZjBs3juTkZIqKikhISOAXv/gFQEdgtJmNNbONZrbJzH5dXp1mNt7M3gndW/9jSaIys4Nlyowys3mh5Xlm9piZvW1mH5rZ5Wb2lJltLSlzMuFIOPlA2RbLBaFtpdx9j7v/J7T6J6BbGOoVEalTtm3bxg9/+EO2bt3KN77xDWbPns3tt99ORkYG2dnZTJ48menTpzNq1ChSU1NZsGABOTk5xMbGAtCsWTOArcAbwK+B/kAy0N3MhpWty8w6AmOAPu6eDBwFxlUhzHOBbwM/BZYDDwGdga5mlnyyA8ORcNYB7c3sIjM7E7guFEQpM2tdZvVair8QEZE6o3fv3kBxS6VLly6ndY62bdvSp08fAMaPH8+KFSvYtGkTAwcOJDk5mZkzZ7Jjx44Kjx8zZkzJYndglbsXuvtXwAKg7wnFB1D8x/86M8sJrV9chTBfcHcHNgK73H2jux8DNgMJJzuw2qPU3P0rM7sNWAHEAE+5+2YzuwfIcvflwB1mdi3wFbAXmFjdekVEoslbb71V7XOY2XHrTZs2pXPnzqxdu7ZKx5999tmnVB3wtLvfWc4+L7Pc+IR9Jb1Vx8osl6yfNKeE5R6Ou7/k7t9090vc/b7Qtl+Fkg3ufqe7d3b3JHe/wt3fC0e9IiKR8Nvf/pYuXbrQpUsXHn74YQCaNGlS7fN+/PHHpcnlz3/+M7169aKwsLB025EjR9i8eTNQnIwOHDhQ0aneAfqZWfPQfZmxwP87ocxrwCgzOx/AzM4zs3ahfbvMrKOZnQEMr/aFhWimARGRU5Cdnc3cuXP5xz/+wdtvv80TTzzB+vXrw3LuDh06MHv2bDp27Mjnn39eev/mF7/4BUlJSSQnJ5e2pCZOnMjUqVNLBw2U5e47gWnA60AukO3ufzmhzBbgl8DLZraB4gFdJbc/pgEvAm8BO8NycYShS01EpD5Zs2YNw4cPL+2+GjFiBKtXrw7LuRs0aMD8+fOP25acnMwbb7zxtbIjR45k5MiRpet5eXnH7Xf354DnTjzO3RPKLC8CFpVTJgPIKGf7xDLLeUCX8vZVRC0cEREJhBKOiMgpSEtLIzMzk0OHDvHvf/+bZcuWkZaWVu3zJiQkMPPZFfR5YCUXTfsrfR5YSeb6/MoPrEXUpSYicgouu+wyJk6cSI8ePQC4+eabSUlJqfZ5M9fnc+fSjRQdOQpA/r4i7ly6EYBhKeVN3lL7WPFw6uiUmprqWVlZkQ5DRKTG9XlgJfn7ir62vU1cLG9O61/l85hZtrunhjO2cFGXmohIFCgoJ9mcbHttpC41EZHTkLk+n1krtlGwr4j4uFjSB3eoVtdXfFxsuS2c+LjY6oQZVdTCERE5RSX3W/L3FeH8935LdW7ypw/uQGzD4yd5jm0YQ/rgDtWMNnoo4YiInKJZK7aV3twvUXTkKLNWbDvtcw5LacP9I7rSJi4Wo/jezf0jutaZAQOgLjURqQfuvfde5s+fT4sWLWjbti3dunXjnHPOYc6cOXz55ZdceumlPPvss5x11lksXryYu+++m5iYGM4555xyH7qsqfstw1La1KkEcyK1cESkTlu3bh1LliwhNzeXv/3tb5SMfB0xYgTr1q0jNzeXjh078uSTTwJwzz33sGLFCnJzc1m+fHm556zovkpdut9SE5RwRKROe/PNNxk6dCiNGzemadOmfO973wNg06ZNpKWl0bVrVxYsWFA6KWafPn2YOHEiTzzxBEePHi33nPXhfktNUMIRkXpp4sSJ/P73v2fjxo3cddddHD58GIDHH3+cmTNn8sknn9CtWzf27NnztWPrw/2WmqCEIyJ1Wp8+fXjhhRc4fPgwBw8e5MUXXwTgwIEDtG7dmiNHjrBgwYLS8h988AE9e/bknnvuoUWLFnzyySflnndYShvenNaffz3wXd6c1l/Jpgo0aEBE6rTu3btz7bXXkpiYSMuWLenatSvnnHMO9957Lz179qRFixb07Nmz9N0y6enpvP/++7g7AwYMICkpKcJXUHdoahsRqfMOHjxIkyZNOHToEH379mXOnDlcdtllkQ6rRkTz1DZq4YhInTdlyhS2bNnC4cOHmTBhQp1NNtFOLRwRqRfCPRVNtFILR0QkgurD1P+1QVhGqZnZVWa2zcy2m9m0cvY3MrNFof3/MLOEcNQrIlIVNTEVjZy6aiccM4sBZgNXA52AsWbW6YRiNwGfu/ulwEPAr6tbr4hIVdWHqf9rg3C0cHoA2939Q3f/ElgIDD2hzFDg6dByBjDAzCwMdYuIVEpT0USHcCScNkDZJ6N2hLaVW8bdvwL2A83KO5mZTTGzLDPLKiwsDEN4IlLfaSqa6BB1Mw24+xx3T3X31BYtWkQ6HBGpAzQVTXQIxyi1fKBtmfULQtvKK7PDzBoA5wBfn6BIRKSG1PWp/2uDcLRw1gHtzewiMzsTuA44cU7v5cCE0PIoYKVH8wNAIiISdtVu4bj7V2Z2G7ACiAGecvfNZnYPkOXuy4EngWfNbDuwl+KkJCIi9UhYHvx095eAl07Y9qsyy4eB0eGoS0REaqeoGzQgIiJ1kxKOiIgEQglHREQCoYQjIiKBUMIREZFAKOGIiEgglHAk7B5++GEOHToU6TBEJMoo4UjYnSzhHD16tNztIlL3KeHUU8888wyJiYkkJSVxww03kJeXR//+/UlMTGTAgAF8/PHHAEycOJGMjIzS45o0aQLAqlWruPzyyxk1ahTf+ta3GDduHO7Oo48+SkFBAVdccQVXXHFF6TH/8z//Q1JSEvfddx/Dhg0rPd8rr7zC8OHDg7twEYkcd4/aT7du3VzCb9OmTd6+fXsvLCx0d/c9e/b4kCFDfN68ee7u/uSTT/rQoUPd3X3ChAm+ePHi0mPPPvtsd3d//fXX/Rvf+IZ/8sknfvToUe/Vq5evXr3a3d3btWtXem53d8AXLVrk7u7Hjh3zDh06+O7du93dfezYsb58+fKavWCReoTiKcUi/vu7vI9aOPXQypUrGT16NM2bNwfgvPPOY+3atVx//fUA3HDDDaxZs6bS8/To0YMLLriAM844g+TkZPLy8sotFxMTw8iRIwEwM2644Qbmz5/Pvn37WLt2LVdffXV4LkxEolpY5lKTuqtBgwYcO3YMgGPHjvHll1+W7mvUqFHpckxMDF999VW552jcuDExMf99+dWkSZP43ve+R+PGjRk9ejQNGujHUKQ+UAunHurfvz+LFy9mz57iVxLt3buX3r17s3DhQgAWLFhAWloaAAkJCWRnZwOwfPlyjhw5Uun5mzZtyoEDByrcHx8fT3x8PDNnzmTSpEnVvRwRqSX0p2U91LlzZ6ZPn06/fv2IiYkhJSWF3/3ud0yaNIlZs2bRokUL5s6dC8Att9zC0KFDSUpK4qqrruLss8+u9PxTpkzhqquuIj4+ntdff73cMuPGjaOwsJCOHTuG9dpEJHqZR/F70FJTUz0rKyvSYUgNuO2220hJSeGmm26KdCgidYqZZbt7aqTjKI+61KTG5OXl8ec//7l0PSsrizvuuINu3bqxYcMGxo8fH5Z6MjMz2bJlS1jOJSI1R11q9Vjm+nxmrdhGwb4i4uNiSR/cIazvfC9JOCWj31JTU0lNDf8fXpmZmQwZMoROnTqF/dwiEj5q4dRTmevzuXPpRvL3FeFA/r4i7ly6kcz1+aVlTuXh0DvuuIPevXtz8cUXlz4oOm3aNFavXk1ycjIPPfQQq1atYsiQIQDMmDGDyZMnc/nll3PxxRfz6KOPltY7f/58evToQXJyMrfeemvp7ARNmjRh+vTpJCUl0atXL3bt2sVbb73F8uXLSU9PJzk5mQ8++CCgb1BETpUSTj01a8U2io4cP81M0ZGjzFqxDYDNmzczc+ZMVq5cSW5uLo888gi33347EyZMYMOGDYwbN4477rij9NidO3eyZs0aXnzxRaZNmwbAAw88QFpaGjk5Ofz0pz/9WgzvvfceK1as4J133uHuu+/myJEjbN26lUWLFvHmm2+Sk5NDTEwMCxYsAODf//43vXr1Ijc3l759+/LEE0/Qu3dvrr32WmbNmkVOTg6XXHJJTX1lIlJN6lKrpwr2FZ10e0UPhy5duhQofjj0f//3f0uPGzZsGGeccQadOnVi165dVYrhu9/9Lo0aNaJRo0acf/757Nq1i9dee43s7Gy6d+8OQFFREeeffz4AZ555ZmkLqVu3brzyyiunceUiEilKOPVUfFws+eUknfi42NM6X9mHQKs68rG8B0fdnQkTJnD//fd/rXzDhg0xs+PKi0jtUa0uNTM7z8xeMbP3Q/+eW0G5o2aWE/osr06dEh7pgzsQ2zDmuG2xDWNIH9wBOLWHQytS2QOg5RkwYAAZGRns3r27tN6PPvoo7PWISPCqew9nGvCau7cHXgutl6fI3ZNDn2urWaeEwbCUNtw/oitt4mIxoE1cLPeP6Fo6Sq3sw6FJSUn87Gc/43e/+x1z584lMTGRZ599lkceeeSkdSQmJhITE0NSUhIPPfRQleLq1KkTM2fOZNCgQSQmJjJw4EB27tx50mOuu+46Zs2aRUpKigYNiESxaj34aWbbgMvdfaeZtQZWuXuHcsoddPcmp3p+PfgpInJq6vKDny3dveTPz0+BlhWUa2xmWWb2tpkNO9kJzWxKqGxWYWFhNcMTEZFoUemgATN7FWhVzq7pZVfc3c2souZSO3fPN7OLgZVmttHdy+37cPc5wBwobuFUFp/UrJp+OFRE6o9KE467X1nRPjPbZWaty3Sp7a7gHPmhfz80s1VACqDO9ihX8nBoyfM6JQ+HAko6InLKqtulthyYEFqeAPzlxAJmdq6ZNQotNwf6AJr4qhao7OFQEZFTUd2E8wAw0MzeB64MrWNmqWb2p1CZjkCWmeUCrwMPuLsSTi1Q2cOhIiKnoloPfrr7HmBAOduzgJtDy28BXatTj0RGuB8OFZH6TXOpSYUqezhURORUaGobqVDJwACNUhORcFDCkZMaltJGCUZEwkJdaiIiEgglHBERCYQSjoiIBEIJR0REAqGEIyIigVDCERGRQCjhiIhIIJRwREQkEEo4IiISCCUcEREJhBKOiIgEQglHREQCoYRTCzz++OM888wzAMybN4+CgoIIRyQicuo0W3QtMHXq1NLlefPm0aVLF+Lj4yMYkYjIqVPCiULPPPMMv/nNbzAzEhMTueSSS2jSpAkJCQlkZWUxbtw4YmNjue+++3jiiSfIzMwE4JVXXuEPf/gDy5Yti+wFiIiUQ11qUWbz5s3MnDmTlStXkpubyyOPPFK6b9SoUaSmprJgwQJycnK45ppreO+99ygsLARg7ty5TJ48OVKhi4iclBJOlFm5ciWjR4+mefPmAJx33nkVljUzbrjhBubPn8++fftYu3YtV199dVChioickmolHDMbbWabzeyYmaWepNxVZrbNzLab2bTq1CnHmzRpEvPnz+e5555j9OjRNGigXlIRiU7VbeFsAkYAb1RUwMxigNnA1UAnYKyZdapmvXVW//79Wbx4MXv27AFg7969x+1v2rQpBw4cKF2Pj48nPj6emTNnMmnSpEBjFRE5FdX6c9jdt0Jx185J9AC2u/uHobILgaHAlurUXVd17tyZ6dOn069fP2JiYkhJSSEhIaF0/8SJE5k6dSqxsbGsXbuW2NhYxo0bR2FhIR07doxc4CIilQii/6UN8EmZ9R1Az4oKm9kUYArAhRdeWLORRakJEyYwYcKEcveNHDmSkSNHHrdtzZo13HLLLUGEJiJy2ipNOGb2KtCqnF3T3f0v4Q7I3ecAcwBSU1M93Oeva7p168bZZ5/N//3f/0U6FBGRk6o04bj7ldWsIx9oW2b9gtA2CYPs7OxIhyAiUiVBdKmtA9qb2UUUJ5rrgOsDqLdWy1yfz6wV2yjYV0R8XCzpgzswLKVNpMMSETlt1R0WPdzMdgDfBv5qZitC2+PN7CUAd/8KuA1YAWwFnnf3zdULO/L27dvHH/7whxo5d+b6fO5cupH8fUU4kL+viDuXbiRzvRqGIlJ7VSvhuPsyd7/A3Ru5e0t3HxzaXuDu15Qp95K7f9PdL3H3+6obdDSoyYQza8U2io4cPW5b0ZGjzFqxrUbqExEJgmYaOE3Tpk3jgw8+IDk5mfT0dNLT0+nSpQtdu3Zl0aJFAPzoRz9i+fLlAAwfPrx02pmnnnqK6dOnk5eXR8eOHbnlllvo3LkzgwYNoqioiIJ9ReXWWdF2EZHaQAnnND3wwANccskl5OTk0KtXL3JycsjNzeXVV18lPT2dnTt3kpaWxurVqwHIz89ny5biR49Wr15N3759AXj//ff50Y9+xObNm4mLi2PJkiXEx8WWW2dF20VEagMlnDBYs2YNY8eOJSYmhpYtW9KvXz/WrVtXmnC2bNlCp06daNmyJTt37mTt2rX07t0bgIsuuojk5GSgeIhzXl4e6YM7ENsw5rg6YhvGkD64Q9CXJiISNpp4qwa1adOGffv28fe//52+ffuyd+9enn/+eZo0aULTpk3Zs2cPjRo1Ki0fExNDUVFR6Wg0jVITkbpECec0lZ3TLC0tjT/+8Y9MmDCBvXv38sYbbzBr1iwAevXqxcMPP8zKlSvZs2cPo0aNYtSoUZWef1hKGyUYEalTlHBOU7NmzejTpw9dunTh6quvJjExkaSkJMyMBx98kFatiidnSEtL4+WXX+bSSy+lXbt27N27l7S0tAhHLyISPHOP3tljUlNTPSsrK9JhiIjUGmaW7e4Vvi4mkjRoQEREAqEutWrSFDQiIlWjhFMNJVPQlMwKUDIFDaCkIyJyAnWpVYOmoBERqTolnGrQFDQiIlWnhFMNmoJGRKTqlHCqQVPQiIhUnQYNVIOmoBERqTolnGrSFDQiIlWjLjUREQmEEs5JFBQUlE60uWrVKoYMGQLAvHnzuO222yIZmohIraOEcxLx8fFkZGREOgwRkTpBCSdk2rRpzJ49u3R9xowZ/OY3v6FLly4nPe6FF16gZ8+epKSkcOWVV7Jr1y4ACgsLGThwIJ07d+bmm2+mXbt2fPbZZwDMnz+fHj16kJyczK233srRo0dPVoWISJ1QrYRjZqPNbLOZHTOzCmcnNbM8M9toZjlmFpXTP48ZM4bnn3++dP3555+nZ8+elR73ne98h7fffpv169dz3XXX8eCDDwJw9913079/fzZv3syoUaP4+OOPAdi6dSuLFi3izTffJCcnh5iYGBYsWFAzFyUiEkWqO0ptEzAC+GMVyl7h7p9Vs74ak5KSwu7duykoKKCwsJBzzz2Xtm3bVnrcjh07GDNmDDt37uTLL7/koosuAopfO71s2TIArrrqKs4991wAXnvtNbKzs+nevTsARUVFnH/++TV0VSIi0aNaCcfdtwKYWXiiibDRo0eTkZHBp59+ypgxY6p0zO23387PfvYzrr32WlatWsWMGTNOWt7dmTBhAvfff38YIhYRqT2CuofjwMtmlm1mU05W0MymmFmWmWUVFhYGFF6xMWPGsHDhQjIyMhg9enSVjtm/fz9t2hQ/h/P000+Xbu/Tp09pF93LL7/M559/DsCAAQPIyMhg9+7dAOzdu5ePPvoonJchIhKVKk04ZvaqmW0q5zP0FOr5jrtfBlwN/MjM+lZU0N3nuHuqu6e2aNHiFKqovs6dO3PgwAHatGlD69atq3TMjBkzGD16NN26daN58+al2++66y5efvllunTpwuLFi2nVqhVNmzalU6dOzJw5k0GDBpGYmMjAgQPZuXNnTV2SiEjUCMsrps1sFfBzd690QICZzQAOuvtvKitbm18x/Z///IeYmBgaNGjA2rVr+cEPfkBOTk6kwxKROi6aXzFd41PbmNnZwBnufiC0PAi4p6brjbSPP/6Y73//+xw7dowzzzyTJ554ItIhiYhEVLUSjpkNB34HtAD+amY57j7YzOKBP7n7NUBLYFloYEED4M/u/vdqxl1jwvXK6Pbt27N+/foaiFBEpHaq7ii1ZcCycrYXANeElj8EkqpTT1D0ymgRkZqjmQbK0CujRURqjhJOGXpltIhIzVHCKUOvjBYRqTlKOGXoldEiIjVHb/wsQ6+MFhGpOUo4J9Aro0VEaoa61EREJBBKOCIiEgglHBERCYQSjoiIBEIJR0REAhGW1xPUFDMrBCLxdrLmQNS+DrsctSne2hQrKN6aVJtihdoTbzt3D/ZlYlUU1QknUswsK1rfJ1Ge2hRvbYoVFG9Nqk2xQu2LNxqpS01ERAKhhCMiIoFQwinfnEgHcIpqU7y1KVZQvDWpNsUKtS/eqKN7OCIiEgi1cEREJBBKOCIiEgglHMDMRpvZZjM7ZmYVDns0s6vMbJuZbTezaUHGeEIc55nZK2b2fujfcysod9TMckKf5QHHeNLvyswamdmi0P5/mFlCkPGVE09l8U40s8Iy3+fNkYgzFMtTZrbbzDZVsN/M7NHQtWwws8uCjrFMLJXFermZ7S/zvf4q6BhPiKetmb1uZltCvxN+XE6ZqPl+ax13r/cfoCPQAVgFpFZQJgb4ALgYOBPIBTpFKN4HgWmh5WnArysodzBC8VX6XQE/BB4PLV8HLIrg/39V4p0I/D5SMZ4QS1/gMmBTBfuvAf4GGNAL+EcUx3o58GKkv9My8bQGLgstNwX+Wc7PQtR8v7XtoxYO4O5b3X1bJcV6ANvd/UN3/xJYCAyt+ejKNRR4OrT8NDAsQnFUpCrfVdlryAAGmJkFGGNZ0fR/Wyl3fwPYe5IiQ4FnvNjbQJyZtQ4muuNVIdao4u473f3d0PIBYCtw4guyoub7rW2UcKquDfBJmfUdfP0HMSgt3X1naPlToGUF5RqbWZaZvW1mw4IJDajad1Vaxt2/AvYDzQKJ7uuq+n87MtSFkmFmbYMJ7bRE089qVXzbzHLN7G9m1jnSwZQIdfOmAP84YVdt+36jRr1546eZvQq0KmfXdHf/S9DxVOZk8ZZdcXc3s4rGtrdz93wzuxhYaWYb3f2DcMdaT7wAPOfu/zGzWylunfWPcEx1wbsU/5weNLNrgEygfWRDAjNrAiwBfuLuX0Q6nrqi3iQcd7+ymqfIB8r+VXtBaFuNOFm8ZrbLzFq7+85QU353BefID/37oZmtovivtSASTlW+q5IyO8ysAXAOsCeA2MpTabzuXja2P1F8Hy1aBfqzWh1lf5m7+0tm9gcza+7uEZsk08waUpxsFrj70nKK1JrvN9qoS63q1gHtzewiMzuT4hvdgY78KmM5MCG0PAH4WgvNzM41s0ah5eZAH2BLQPFV5bsqew2jgJUeuiMbAZXGe0If/bUU9+1Hq+XAjaHRVL2A/WW6YKOKmbUquXdnZj0o/p0UqT88CMXyJLDV3X9bQbFa8/1GnUiPWoiGDzCc4n7Y/wC7gBWh7fHAS2XKXUPxqJUPKO6Ki1S8zYDXgPeBV4HzQttTgT+FlnsDGykecbURuCngGL/2XQH3ANeGlhsDi4HtwDvAxRH+Gags3vuBzaHv83XgWxGM9TlgJ3Ak9HN7EzAVmBrab8Ds0LVspIKRl1ES621lvte3gd4R/jn4DuDABiAn9LkmWr/f2vbR1DYiIhIIdamJiEgglHBERCQQSjgiIhIIJRwREQmEEo6IiARCCUdERAKhhCMiIoH4/1img1fgibz1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# We have done the plotting for you. Just run this cell.\n",
    "result = compute_pca(X, 2)\n",
    "plt.scatter(result[:, 0], result[:, 1])\n",
    "for i, word in enumerate(words):\n",
    "    plt.annotate(word, xy=(result[i, 0] - 0.05, result[i, 1] + 0.1))\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What do you notice?**\n",
    "\n",
    "The word vectors for gas, oil and petroleum appear related to each other,\n",
    "because their vectors are close to each other.  Similarly, sad, joyful\n",
    "and happy all express emotions, and are also near each other."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "encoding": "# -*- coding: utf-8 -*-"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}