LLaMA: Concepts Explained (Summary)

4 min readMar 2, 2023

Pre-normalization, SwiGLU, Rotary Embeddings

In this article we would understand the why LLaMA is able to achieve comparable performance with smaller model size.

LLaMA-13B outperforms GPT-3(175B) on most benchmark despite being 10× smaller. It is possible to train state-of-the-art models using publicly available datasets.

The best performances are not achieved by the largest models, but by smaller models trained on more data. https://arxiv.org/abs/2203.15556

Pre-training data is mixture of many open datasets of diverse domain. It is helping LLaMA to achieve few shot capabilities.

https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/

T5 success can also be credited C4(Colossal Clean Crawled Corpus) dataset. Its cleaning process includes de-duplication, discarding incomplete sentences, and removing offensive or noisy content.

Now let's jump on the most important concepts which LLaMA has included.

1. Pre-normalization Using RMSNorm

RMSNorm : Root Mean Square Layer Normalization [1]

LLaMA normalizes the input of each transformer sub-layer, instead of normalizing the output.

Inspiration of including pre-normalization is taken from GPT3.

RMSNorm is extension of Layer Normalization (LayerNorm). Reason behind using RMSNorm is the computational overhead in LayerNorm. This makes improvements slow and expensive. RMSNorm achieves comparable performance against LayerNorm but reduces the running time by 7%∼64%.

Let first understand LayerNorm, It has two properties.

a. re-centring : It make model insensitive to shift noises on both inputs and weights.

b. re-scaling: It keeps the output representations intact when both inputs and weights are randomly scaled.

RMSNorm claims that most of the benefits comes from re-scaling.

RMSNorm does re-scaling invariance and regularizes the summed
inputs simply according to the root mean square (RMS) statistic.

a_i : activation of ith neuron

g ∈ Rn is the gain parameter used to re-scale the standardized summed inputs

Intuitively, RMSNorm simplifies LayerNorm by totally removing the mean statistic in LayerNorm.

Feel free to take a look into the implementation of RMSNorm : https://github.com/bzhangGo/rmsnorm/blob/master/rmsnorm_torch.py

2. SwiGLU

To understand SwiGLU activation function we need to understand Swish activation function.

Inspiration of using SwiGLU in LLaMA is taken from PaLM.

def sigmoid(x):
  return  1/(1 + np.exp(-x))

def swish(x):
  return x*sigmoid(x)

https://arxiv.org/pdf/2002.05202v1.pdf

Python Implementation of SwiGLU.[2]

class SwiGLU(tf.keras.layers.Layer):
    def __init__(self, bias=True, dim=-1, **kwargs):
        super(SwiGLU, self).__init__(**kwargs)
        self.bias = bias
        self.dim = dim
        self.dense = tf.keras.layers.Dense(2, use_bias=bias)

    def call(self, x):
        out, gate = tf.split(x, num_split=2, axis=self.dim)
        gate = tf.keras.activations.swish(gate)
        x = tf.multiply(out, gate)
        return x

3. Rotary Embeddings (RopE)

RoPE, is a type of position embedding which encodes absolute positional information with rotation matrix and naturally incorporates explicit relative position dependency in self-attention formulation.

Advantage of RoPE

Can be expanded to any sequence lengths
Decaying inter-token dependency with increasing relative distances.
Capability of equipping the linear self-attention with relative position encoding.

The key idea is to encode relative position by multiplying the context
representations with a rotation matrix.

RoPE decays with the relative distance increased, which is desired for natural language encoding.