Transfer Learning with GloVe

Transfer Learning with GloVe

Table of Contents

Transfer Learning

Transfer learning is one of the most important breakthroughs in machine learning! It helps us to use the models created by others.

Since everyone don’t have access to billions of text snippets and GPU’s/TPU’s to extract patterns from it. If someone can do it and pass on the learnings then we can directly use it and solve business problems.

When someone else creates a model on huge generic dataset and passes only the model to other for use. This is known as transfer learning, because everyone don’t have to train the model on such huge amount of data, hence, they “transfer” the learnings from others to their system.

Transfer learning is really helpful in NLP. Specially vectorization of text, because converting text to vectors for 50K records also is slow. So if we can use the pre-trained models from others, that helps to resolve the problem of converting the text data to numeric data, and we can continue with the other tasks, such as classification or sentiment analysis etc.

Stanford’s GloVe and Google’s Word2Vec are two really popular choices in Text vectorization using transfer learning.


GloVe

Global Vectors for Word Representation GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space.

Glove vectors are generated based on co-occurance matrix of words using a context window.

Glove Word vectors are basically a form of word representation that bridges the human understanding of language to that of a machine. 

They have learned representations of text in an n-dimensional space where words that have the same meaning have a similar representation. Meaning that two similar words are represented by almost similar vectors that are very closely placed in a vector space.

For example, look at the below diagram, the words King and Queen appear closer to each other. Similarly the words Man and Woman appears closer to each other due to the kind of numeric vectors assigned to these words by GloVe. If you compute the distance between two words using their numeric vectors, then those words which are related to each other with a context, will have less distance between them.

GloVe word representation
GloVe word representation

So how these numeric vectors are generated for each word? Based on Co-occurence matrix.


Context Window

Similar words tend to occur together and will have similar context for example – Apple is a fruit. Mango is a fruit. Apple and mango tend to have a similar context i.e fruit.

Co-occurrence β€“ For a given corpus, the co-occurrence of a pair of words say w1 and w2 is the number of times they have appeared together in a Context Window.

Context Window β€“ Context window is specified by a number and the direction. Basically, how many words before and after a given word should be considered? So what does a context window of 2 (around) means? Let us see an example below

The context window of 2 is the green words for word highlighted in Red
The context window of 2 is the green words for word highlighted in Red


Co-occurence matrix

To understand the co-occurence matrix, consider the simple sentence below and the co-occurence matrix formed using a context window of 2 words.

Co-occurence matrix example for one sentence
Co-occurence matrix example for one sentence with context window=2 words

context window of 2 words
The words “He” and “is” appear 4-times together using a context window of 2-words

Let’s say there are 5000 unique words in the corpus. So Vocabulary size = 5000. A co-occurrence matrix of size 5000 X 5000 will be formed. 

Now, for even a decent corpus the size of all the unique words can easily get very large and difficult to handle. Imagine 1000000’s of words!!. So generally, this architecture is never preferred in practice.

Hence, instead of using the full 5000 X 5000 Co-occurrence matrix it is decomposed using techniques like PCA, SVD etc. into factors and combination of these factors forms the word vector representation.

For example, you perform PCA on this matrix of size 5000 X 5000. You will obtain 5000 principal components. Let’s say you selected 300 components out of these 5000 components. So, the new matrix will be of the form 5000 X 300.

From Text Corpus to GloVe vectors
From Text Corpus to GloVe vectors

Hence, a single word, instead of being represented in 5000 numbers will be represented in 300 numbers while still capturing almost the same semantic meaning.

This 300 number can vary from model to model.


Case Study: Support Ticket Classification using Glove

In a previous case study I showed you how can you convert Text data into numeric using TF-IDF. And then use it to create a classification model to predict the priority of support tickets.

In this case study I will use the same dataset and show you how can you use the numeric representations of words from Glove and create a classification model.

Problem Statement: Use the Microsoft support ticket text description to classify a new ticket into P1/P2/P3.

You can download the data required for this case study here.

Reading the support ticket data

This data contains 19,796 rows and 2 columns. The column”body” represents the ticket description and the column “urgency” represents the Priority.

Support Ticket Data


Visualising the distribution of the Target variable

Now we try to see if the Target variable has balanced distribution or not? Basically each priority type has enough number of rows to be learned.

If the data would have been imbalanced, for example very less number of rows for P1 category, then you need to balance the data using any of the popular techniques like over-sampling, under-sampling or SMOTE.

Support ticket data target variable

The above bar plot shows that there are enough rows for each ticket type. Hence, this is balanced data for classification.


Count Vectorization: converting text data to numeric

This step will help to remove all the stopwords and create a document term matrix.

We will use this matrix to do further processing. For each word in the document term matrix, we will use the GloVe numeric vector representation.

Count Vectorization of Text Data for Glove


GloVe conversion:

More information and model download: https://nlp.stanford.edu/projects/glove/

Please download “glove.6B.zip” from the above link, that is the GloVe model trained on 6Billion text tokens. All other files are >5GB and will probably hang your laptop!

Now we will use the GloVe representation of words to convert the above document term matrix to a smaller matrix, where the columns are the sum of the vectors for each word present in the document.

For example, look at the below diagram. The flow is shown for one sentence, the same happens for every sentence in the corpus.

  • The numeric representation of each word is taken from GloVe.
  • All the vectors are added, hence producing a single vector
  • That single vector represents the information of the sentence, hence treated as one row.

Glove vectors for a single sentence
Glove vectors for a single sentence

Note: If you feel that your laptop is hanging due to the processing required for the below commands, you can use google colab notebooks!

From the glove.6B.zip folder I am choosing the version of glove which represents each word with 50-numbers.

sample glove vector

sample glove words


Converting every sentence to a numeric vector

For each word in a sentence we extract the numeric form of the word and then simply add all the numeric forms for that sentence to represent the sentence.

Preparing Data for ML

glove data for machine learning


Splitting the data into training and testing

glove data for ml shape


Standardization/Normalization

This is an optional step. It can speed up the processing of the model training.

glove data for ml shape


Training ML classification models

Now the data is ready for machine learning. There are 50-predictors and one target variable. We will use below algorithms and select the best one out of them based on the accuracy scores you can add more algorithms to this list as per your preferences.

  • Naive Bayes
  • KNN
  • Logistic Regression
  • Decision Trees
  • AdaBoost


Naive Bayes

This algorithm trains very fast! The accuracy may not be very high always but the speed is guaranteed!

I have commented the cross validation section just to save computing time. You can uncomment and execute those commands as well.

Naive Bayes Glove model results


KNN

This is a distance based supervised ML algorithm. Make sure you standardize/normalize the data before using this algorithm, otherwise the accuracy will be low.

KNN model output for GloVe


Logistic Regression

This algorithm also trains very fast. Hence, whenever we are using high dimensional data, trying out Logistic regression is sensible. The accuracy may not be always the best.

Logistic Regression output for GloVE


Decision Tree

This algorithm trains slower as compared to Naive Bayes or Logistic, but it can produce better results.

Decision Tree accuracy for Glove data


Adaboost

This is a tree based boosting algorithm. If the data is not high dimensional, we can use this algorithm. otherwise it takes lot of time to train.

Adaboost accuray for Glove data


Training the best model on full data

KNN algorithm is producing the highest accuracy on this data, hence selecting it as final model for deployment.


Making predictions on new cases

To deploy this model, all we need to do is write a function which takes the new data as input, performs all the pre-processing required and passes the data to the Final model.

Glove text classification results


Conclusion

Transfer learning has made NLP research faster by providing an easy way to share the models produced by big companies and build on top of that. Similar to GloVe we have other algorithms like Word2Vec, Doc2Vec and BERT which I have discussed in separate case studies.

I hope this post helped you to understand how GloVe vectors are created and how to use them to convert any text into numeric form.

Consider sharing this post with your friends to spread the knowledge and help me grow as well! πŸ™‚

Author Details
Lead Data Scientist
Farukh is an innovator in solving industry problems using Artificial intelligence. His expertise is backed with 10 years of industry experience. Being a senior data scientist he is responsible for designing the AI/ML solution to provide maximum gains for the clients. As a thought leader, his focus is on solving the key business problems of the CPG Industry. He has worked across different domains like Telecom, Insurance, and Logistics. He has worked with global tech leaders including Infosys, IBM, and Persistent systems. His passion to teach inspired him to create this website!

Leave a Reply!

Your email address will not be published. Required fields are marked *