Word vectors are numeric representations of words, learned from the words around them, in which words with similar meanings sit close together. Patent US 9,037,464 B1, "Computing numeric representations of words in a high-dimensional space," describes how Google learns them. Google filed it on March 15, 2013, from a provisional application of January 15, 2013, entitled "Computing Vector Representations," and the patent was granted on May 19, 2015.
What does patent US 9,037,464 describe?
Patent US 9,037,464 describes a system that trains an embedding function and a classifier to predict words from their context, then keeps the embedding as a numeric representation of every word. Its 4 inventors are Tomas Mikolov, Kai Chen, Gregory S. Corrado and Jeffrey A. Dean, and the original assignee is Google Inc. The same 4 authors signed the first Word2vec research paper in 2013, "Efficient Estimation of Word Representations in Vector Space," and Jeffrey Dean co-signed the anchor text indexing patent as well. The patent itself never uses the name Word2vec. It holds 18 claims and 5 drawing sheets.
The abstract sets out the method: obtain "sequences of words," train "a classifier and an embedding function," then process "each word in the vocabulary" to associate it "with the respective numeric representation of the word in the high-dimensional space."
What is a word vector?
A word vector is a list of floating-point numbers that places a word in a high-dimensional space. The patent's simplified example: the embedding function "may map the word 'cat' to a vector [0.1, 0.5, 0.2] and the word 'tablet' to a vector [0.3, 0.9, 0.0]." The patent adds that words can be represented in very high-dimensional spaces, "e.g., spaces that have on the order of 1000 dimensions, resulting in higher quality representations than when words are represented in relatively lower-dimensional spaces."
The key property: "Positions of representations in the high-dimensional space can reflect semantic similarities, syntactic similarities, or both, between words represented by the representations."
How are word vectors learned?
Word vectors are learned by predicting words from their neighbors. The patent describes 2 architectures:
- From the surrounding words to the missing word: the system receives the words at positions t − N to t + N around an unknown word at position t, where N is a predetermined integer, maps each one to a vector (for example through a lookup table) and merges them into a single vector. The merge can be a sum, a weighted linear combination, a component-wise maximum or a norm-constrained linear combination. The patent's example: "Atlanta" maps to [0.1, 0.2, 0.3] and "Hotel" to [0.4, 0.5, 0.6], and their sum is [0.5, 0.7, 0.9]. A classifier then gives a score to each word of "a pre-determined set of words," and each score "represents a predicted likelihood that the corresponding word is the unknown word."
- From one word to its neighbors: the system receives one word, maps it to a vector and uses "a plurality of classifiers," each one corresponding to "a respective position surrounding the input word," to predict the words around it. Each classifier generally has its own parameter values and produces its own score vector for its position.
The classifier can be any multiclass or multilabel classifier, for example a multiclass logistic regression, a multiclass support vector machine or a Bayesian classifier. In some implementations, a ranking function (for example a hinge-loss or pairwise ranking function) replaces the classifier and orders the words by predicted likelihood.
How does training adjust the vectors?
Training adjusts the vectors by backpropagation, on sequences in which every word is known. When the scores of a classifier "do not align with the actual value of the unknown word," the classifier adjusts its parameters "using conventional gradient based methods," then "sends an error signal to the embedding function," which adjusts its own parameters "through successive stages of backpropagation."
Input words can be tokenized first, "so that known compounds, e.g., 'New York City' and other entity names, are treated as a single word by the system." For large training sets, the training process can be parallelized, for example with the techniques of "Large Scale Distributed Deep Networks," a 2012 paper by Jeffrey Dean and others. The patent states: "By using a word prediction system having a two-layer architecture and by parallelizing the training process, the word prediction system can be effectively trained on very large word corpuses, e.g., corpuses that contain on the order of 200 billion words."
Once training is done, the system processes each word of the vocabulary with the trained embedding function and stores the associations, "e.g., as a data set of words and their associated representations." These representations "can be used for a variety of purposes other than as input to a classifier."
What do word vectors capture?
Word vectors capture similarities and relationships between words. The patent gives 3 kinds of examples:
- Syntactic similarity: "words that are similar to the word 'small' include the words 'smaller,' 'tiny,' 'smallest.'"
- Semantic similarity: "the word 'queen' is similar to the words 'king' and 'prince.'"
- Analogies: "the word 'king' is similar to the word 'queen' in the same sense as the word 'prince' is similar to the word 'princess.'"
Relationships become arithmetic. "Vector subtraction and vector addition operations" can find a word with a given relationship: to find a word that has a similar relationship to a word A as a word B has to a word C, the system computes vector(A) − vector(B) + vector(C). The patent's example: "the operation vector('King') − vector('Man') + vector('Woman') may result in a vector that is closest to the vector representation of the word 'Queen.'"
What else can the trained system do?
The trained system can also use its predictions directly. The patent gives 2 uses: predicting "missing words at various positions in sentences," and a question/answer system. In that case, the system "can be trained on known question-answer pairs, and can then be used to predict a question if all or part of an answer is known or an answer if all or part of a question is known."
What do the claims protect?
The claims protect the 2 architectures, each as a system and as a method. The 18 claims contain 4 independent claims:
- Claim 1: a system whose embedding layer maps the words that surround an unknown word into a numeric representation, a classifier layer that scores the likely unknown word, and instructions to compute and associate the representation of each word of the vocabulary.
- Claim 5: the same kind of system with one input word and a plurality of classifiers, one per surrounding position.
- Claim 9: the method that trains a plurality of classifiers and an embedding function on one input word, then assigns a point in the high-dimensional space to each word of the vocabulary.
- Claim 14: the method that trains a classifier and an embedding function on the words surrounding an unknown word, then assigns a point to each word of the vocabulary.
The dependent claims add the floating-point representations, positions that reflect semantic or syntactic similarities, the merged vector built from the vectors of several words, and training by backpropagation.
Why does this patent matter for search?
This patent matters because vectors let a system measure meaning, not just matching words. Two texts can be close in vector space without sharing a word. Google confirmed in October 2015 that RankBrain, a machine learning system, helps it interpret queries, and later systems such as generative summaries compare embeddings to decide whether a passage verifies a sentence. This patent does not describe those systems; it describes the foundation they build on: words as positions in a space of meaning.
Older methods approached the same goal with statistics: phrase-based indexing learned which phrases predict each other, and contextual synonyms learned which words users swap. Vectors pursue the same goal in a more general way, with one learned representation per word.
What do word vectors change for your SEO?
Word vectors mean that a page can be understood through the meaning of its vocabulary, so covering a topic with its natural, related terms matters more than repeating one keyword. 5 consequences follow:
- Cover the vocabulary of the topic. Vectors are learned by predicting each word from the words around it: words used in the same contexts end up close together, so a page that uses the terms experts use together sits close to the topic.
- Stop repeating the exact keyword. "Small," "tiny" and "smaller" sit close together: varied, natural wording can be understood as the same concept.
- Name entities precisely. Relationships like king and queen are encoded, and known compounds such as "New York City" and other entity names can be treated as a single word: write the full, standard name of the entities linked to your subject.
- Write clear, self-contained passages. Systems built on embeddings compare passages with queries or sentences: a passage whose meaning is unambiguous on its own is easier to match.
- Think in concepts, not strings. Optimize for what a query means, since a query and a page can be close in meaning without a shared word.
Our patent-based SEO briefs apply this: each one lists the terms experts use around the topic and asks for passages whose meaning stands on its own.
The patent describes what Google's system can do. It does not confirm how Google uses word vectors in its ranking today.
