What is transformer model in NLP?
One kind of deep learning model that was first shown in 2017 is the transformer model. These models have been used for a variety of machine learning and artificial intelligence tasks and have swiftly established themselves as foundational in natural language processing (NLP).
In 2017, Ashish Vaswani, a team from Google Brain, and a group from the University of Toronto published a paper titled “Attention is All You Need” that initially outlined the approach. Because transformers are now widely employed in applications like training LLMs, the publication of this study is seen as a turning point in the field.
Near-real-time speech and text translation is possible with these models. Tourists can now converse with locals in their native tongue on the street, for instance, with applications. They expedite medication design and aid researchers in their understanding of DNA. They can assist in identifying irregularities and stopping financial and security fraud. Computer vision tasks also make use of vision transformers.
Because they enable the model to concentrate on the most pertinent portions of input text, transformer designs are used for prediction, summarization, question answering, and other purposes in OpenAI‘s well-known ChatGPT text creation tool. The tool’s different versions use the term “GPT,” which stands for “generative pre-trained transformer” (e.g., GPT-2, GPT-3). Because transformer models can more easily anticipate the next word in a text sequence based on huge, complicated data sets, text-based generative AI systems like ChatGPT benefit from them.
The Bidirectional Encoder (BERT) model. Transformer representations are predicated on the architecture of the transformer. Nearly all English-language Google search results as of 2019 employed BERT, and it has since been expanded to more than 70 other languages.
How does transformer models work?
Transformer models handle incoming data which may include token sequences or other structured data through a number of layers that include feedforward neural networks and self-attention techniques. There are a few essential steps that make up the fundamental concept of how transformer models operate.
Suppose you have to translate a sentence from English to French. To complete this operation using a transformer model, you would need to follow these steps.
Input embeddings
First, the input sentence is converted into embeddings, which are numerical representations. These encapsulate the input sequence’s tokens’ semantic meaning. These embeddings can be acquired from pre-trained word embeddings or learnt during training for word sequences.
Positional encoding
Prior to being fed into the transformer model, positional encoding is usually introduced as a collection of extra values or vectors added to the token embeddings. The position information is encoded using particular patterns in these positional encodings.
Multi-head attention
Self-attention uses several “attention heads” to record various kinds of token associations. The self-attention mechanism computes attention weights using softmax functions, a kind of activation function.
Layer normalisation and residual connections
To stabilise and expedite training, the model makes use of layer normalisation and residual connections.
Feedforward neural networks
Feedforward layers receive the output from the self-attention layer. By applying non-linear changes to the token representations, these networks enable the model to identify intricate relationships and patterns within the data.
Stacked layers
Transformers usually have several layers stacked on top of one another. The representations are progressively improved as each layer refines the output of the one before it. The model may capture abstract and hierarchical aspects in the data by stacking numerous layers.
Output layer
A distinct decoder module can be placed on top of the encoder to produce the output sequence in sequence-to-sequence tasks such as neural machine translation.
Training
Supervised learning is used to train transformer models, which learn to minimize a loss function that measures the discrepancy between the model’s predictions and the task’s ground truth. Stochastic gradient descent (SGD) and Adam are two optimization algorithms commonly used in training.
Inference
The model can be applied to new data after training. The pre-trained model is fed the input sequence during inference, and it produces representations or predictions for the task at hand.
How transformer models are different?
The transformer model’s primary innovation is its independence from neural network techniques with serious limitations, such as recurrent neural networks (RNNs) and convolutional neural network (CNN). Transformers are fast for training and inference since they handle input sequences in parallel since GPUs can’t be added. Transformer models train faster than LSTM structures.
The 1990s and 1920s saw the invention of LSTM and RNN. Computation can take a long time with these methods since they calculate each input component sequentially, word by word, for example. Furthermore, when there is a significant “distance” between bits of information in an input, both methods struggle to maintain context.
Two big innovations
Transformer models contribute to two main innovations. Think about these two developments in relation to text prediction.
Positional encoding: Each word is given a distinct number rather than being examined in the order that it occurs in a phrase. This enables the model to take into account the sequential information of the sequence by giving information on the location of each token (parts of the input, such as words or subword fragments in NLP) in the sequence.
Self-attention: In order to forecast words that are likely to be used in sequence, the model uses attention, a mechanism that determines weights for each word in a phrase based on how those weights relate to each other. As a model is trained on a large amount of data, this knowledge is acquired over time.
Each word in the sequence can pay attention to every other word in parallel with the self-attention mechanism, which weighs each word’s significance for the current token. In this sense, it may be claimed that machine learning models are able to “learn” grammar rules by using statistical probabilities of the ordinary linguistic usage of terms.
