Word2Vec and GLoVe
Word2Vec is a set neural network algorithms that have gotten a lot of attention in recent years as part of the re-emergence of deep learning in AI.
The idea that one can represent words and concepts as vectors is not new. The ability to do it effectively and generate noteworthy results is.
Word2Vec algorithms are especially interesting because they allow us to perform arithmetic on the word vectors that yield both surprising and satisfying results. We call these “word analogies”.
Some popular word analogies Word2Vec is capable of finding:
“King” is to “Man” as “Queen” is to “Woman”.
“France” is to “Paris” as “Italy” is to “Rome”.
“December” is to “November” as “July” is to “June”.
Not only can we cluster similar words together, we can make all these clusters have the same “structure”, all by using Word2Vec.
Word2Vec was created by a team led by Tomas Mikolov at Google and has many advantages over earlier algorithms that attempt to do similar things, like Latent Semantic Analysis (LSA) or Latent Semantic Indexing (LSI).
In this book we cover the GLoVe algorithm, which is an alternative to Word2Vec that both runs faster and yields superior results. It was invented by the guys at Stanford, who have created and published many NLP tools throughout the years that are now considered standard.
I show you both their derivations in math, and how to implement them in code. We’ll do 3 different versions. The first version will show you how to solve the problem with vanilla gradient descent and Numpy. The second version will show you a simpler way of doing gradient descent using Theano’s automatic differentiation capabilities. The third version will make use of techniques found in recommender systems and matrix factorization. In fact, I devote an entire chapter to showing you how to use matrix factorization to recommend movies to users. We then take the main algorithm for solving matrix factorization problems, called alternating least squares, and apply it to solve GLoVe.
Amazingly, all the technologies we discuss in this book can be downloaded and installed for FREE. That means all you need to invest after purchasing this book is your effort and your time. The only prerequisites are that you are comfortable with Python , Numpy, and Theano coding and you know the basics of deep learning.
“Hold up... what’s deep learning and all this other crazy stuff you’re talking about?”
If you are completely new to deep learning, you might want to check out my earlier books and courses on the subject, since they are required in order to understand this book:
Deep Learning in Python https://www.udemy.com/data-science-deep-learning-in-python
Deep Learning in Python Prerequisities https://www.udemy.com/data-science-logistic-regression-in-python
Much like how IBM’s Deep Blue beat world champion chess player Garry Kasparov in 1996, Google’s AlphaGo recently made headlines when it beat world champion Lee Sedol in March 2016.
What was amazing about this win was that experts in the field didn’t think it would happen for another 10 years. The search space of Go is much larger than that of chess, meaning that existing techniques for playing games with artificial intelligence were infeasible. Deep learning was the technique that enabled AlphaGo to correctly predict the outcome of its moves and defeat the world champion.
Deep learning progress has accelerated in recent years due to more processing power (see: Tensor Processing Unit or TPU), larger datasets, and new algorithms like the ones discussed in this book.
Book 1 in the series can be found at: https://www.amazon.com/dp/B01KQ0ZN0A
Book 3 in the series can be found at: https://www.amazon.com/dp/B01KS5AEXO
