```{r global-options, include=FALSE}
#knitr::opts_chunk$set(eval = FALSE)
library(tidyverse)
library(tidymodels)
library(kableExtra)
library(reticulate)
```

**Purpose:** In this notebook we'll consider a powerful technique falling under the umbrella of *word embeddings*. In particular, we'll look at a measure called *term frequency, inverse document frequency* as a way to measure how important a term is to one category of documents versus other categories.

**Data Used:** We'll use excerpts from *Spooky Authors* (Edgar-Allen Poe, Mary Shelley, and HP Lovecraft), which appeared as part of a [Kaggle Competition](https://www.kaggle.com/competitions/spooky-author-identification/overview) in 2018. For convenience, I've added the training data to my GitHub repository [here](https://raw.githubusercontent.com/agmath/agmath.github.io/master/data/classification/spooky_authors.csv).

## The Big Idea

Are tokens more frequently utilized in one type of document versus another? Here, we'll look at a measure of how frequent a token is to a particular class is and how *unique* it is to that class, relative the other classes. The *term frequency, inverse document frequency* of a token is defined as follows: $\displaystyle{\text{tf_idf}\left(\text{token}\right) = f_{\text{Token}}\cdot\ln\left(\frac{n_{\text{Documents}}}{n_{\text{Documents Containing Token}}}\right)}$, where $f_{\text{Token}}$ denotes the frequency of the $\tt{Token}$ across the entire corpus.

In the formula for $\text{tf_idf}\left(\text{Token}\right)$, if $\tt{Token}$ appears across all of the documents, then the ration $\frac{n_{\text{Documents}}}{n_{\text{Documents Containing Token}}}$ will evaluate to $1$ and the logarithm will be $0$. That is, a *token* appearing across all documents cannot be useful in differentiating the document. That logarithm will be largest when the *token* in question appears in just one of the documents and none of the others. This means that *tokens* with the highest *term frequency, inverse document frequency* will be those unique to their document and having a high frequency of usage within that document.

Let's try this out with our spooky author excerpts. You can see the first few rows of the spooky author data below.

```{python}
import pandas as pd

from sklearn.model_selection import train_test_split
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import confusion_matrix, accuracy_score

spooky_authors = pd.read_csv("https://raw.githubusercontent.com/agmath/agmath.github.io/master/data/classification/spooky_authors.csv")
```

```{r}
py$spooky_authors %>%
  head() %>%
  kable() %>%
  kable_styling(bootstrap_options = c("hover", "striped"))
```

Let's compute *term-frequency*, *inverse document frequency* for our excerpted text from our spooky authors. We'll start by splitting our `spooky_authors` data into training and test sets. We'll then begin by isolating the text column, creating a `TfidfVectorizer()`, fitting it to the spooky author text, and then using it to transform the text. 

```{python}
train, test = train_test_split(spooky_authors, train_size = 0.75, random_state = 123)

spooky_train_text = train["text"].values

tfidf_vec = TfidfVectorizer(analyzer = "word", stop_words = "english")

tfidf_vec.fit(spooky_train_text)
spooky_train_tfidf = tfidf_vec.transform(spooky_train_text)
spooky_train_tfidf
```

The `spooky_authors_tfidf` object is stored as a *sparse* matrix. This means that most of the entries in the matrix are 0's, and so it is more efficient to store the size of the matrix, the locations of non-zero entries, and their values in a different data structure (*sparse matrix*) than it is to store the entire matrix in memory. 

It makes sense that the resulting matrix would be sparse. Each excerpt makes use of very few words in the entire lexicon present across the union of the excerpts. Still, we can see part of the sparse matrix if we use `.toarray()` to force the matrix to be printed out.

```{python}
spooky_train_tfidf.toarray()
spooky_train_tfidf.toarray().shape
```

Since the matrix is sparse, we aren't seeing anything useful! There are many non-zero entries in here though -- just the proportion of non-zero entries is very tiny.

Now that we've got our text converted into usable numeric features, let's build a model. I'll use a random forest since we have so many features here!

```{python}
rf_clf = RandomForestClassifier()
rf_clf.fit(spooky_train_tfidf, train["author"])
```

Now that our model is fit, let's use it on a spooky sentence!

```{python}
predictions = rf_clf.predict_proba(tfidf_vec.transform(['I laid down on the floor and stretched my arm towards the corner of the draped comforter. I lifted it slightly and...']))
predictions
```

Let's see how the forest actually performs on our unseen `test` excerpts from our `spooky_authors` data. We'll first need to transform the text using our `tfidf_vec()` transformer -- we won't fit it again, because fitting it was part of our *training* process.

```{python}
test_text_tfidf = tfidf_vec.transform(test["text"].values)
test_author_preds = rf_clf.predict(test_text_tfidf)
cm = confusion_matrix(test["author"], test_author_preds)
test_accuracy = accuracy_score(test["author"], test_author_preds)
print("Order of Classes: ", rf_clf.classes_)
cm
print("Model (Test) Accuracy: ", test_accuracy)
```

Note that the model above takes quite some time to fit and also to predict. This is because each distinct word in the training set (aside from stopwords) has become a feature that the model can use. Additionally, we're fitting a random forest consisting of 100 individual trees. That being said, it is performing reasonably well. It has an accuracy of `r py$test_accuracy` and identifies excerpts from Edgar Allen Poe quite well. However, it has a difficult time differentiating the excerpts from the other authors from Edgar Allen Poe.

***

## Summary

Over these past three topics, you've learned quite a bit about extracting insights from text data. We've engaged in basic text analysis through tokenization, looked at using regular expressions for extracting features from text using pattern matching, and now we've seen how to use a simple word embedding -- *tf-idf* scores -- as a component of a predictive model.