Continuous Bag of Words

The continuous Bag of Words (CBOW) model aims to predict the target word based on the context provided by the surrounding words. The working of the CBOW is as follows:

1. Step 1: Take the context of surrounding words as input. This is the window of words around the word.
2. Step 2: Each word is represented as a vector in the embedding layer in the given context. The dimensions of the vector stores the information on the semantic and syntactic features of the word.
3. Step 3: Aggregate the individual word ( or vectors) to get a single vector that represents the context. It will become our input.
4. Step 4: Finally predict the output using the input in the previous step. We get a probability distribution over the vocabulary and the word with the highest probability is considered the target word.

CBOW is adjusted using the neural network to minimize the difference between the predicted word and the actual word in the target.

• Sentence given: “w3wiki is a nice website”
• Window size: 2 (within the window of 2 around the target word “nice”)
• Let we want to predict word: nice, so considering 1 word on both sides of the target word, we have
• Input: The context words are “a”, and “website”.
• Each word is represented through a vector of embedding. Then the embeddings are aggregated to create a single vector of context (e.g., by averaging) to create a single vector representing the context.
• Using the aggregated context vector to predict the target word, The model outputs a probability distribution over the vocabulary.

Ouput: The output should predict the word “nice”.

CBOW is trained to efficiently predict target words based on their surrounding context. The neural network learns to represent words in a way that captures their semantic and syntactic relationships, optimizing the overall predictive performance.

Word2Vec is a modeling technique used to create word embeddings. It creates a vector of words where it assigns a number to each of the words. Word embeddings generally predict the context of the sentence and predict the next word that can occur. In R Programming Language Word2Vec provides two methods to predict it:

• CBOW (Continuous Bag of Words)
• Skip Gram

The basic idea of word embedding is words that occur in similar contexts tend to be closer to each other in vector space. The vectors are used as input to do specific NLP tasks. The other tasks are text summarization, text matching, etc.