Depths of DeepSeek - MLA and MoE

21 Oct, 2026

The ridiculously large LLMs of the present time are quite different from models like GPT 3 or Llama 3 which came out a few years ago. There has been a shift from the standard MHA and MLP to architectures like Multi-head Latent Attention (MLA) and Mixture of Experts (MoE) in models like Deepseek V2 and V3, Kimi K2, Mistral Large 3 and many others.

Deepseek V2 was the first one to use MLA attention mechanism which if implemented correctly, can be far more efficient than MHA. It is based on the idea that keys and values can be stored in the cache in the form of a single compressed matrix and during the forward pass, these can be projected to higher dimensions by using learned weights, creating the real keys and values used in attention. This helps reduce the cache size by a lot which was the primary goal of adopting this technique.

Mixture of Experts is nothing but using only a specfic set of model parameters relevant to the inputs instead of letting a big MLP eat a lot more resources. This involves creating multiple (say n_experts) small experts and having a router (linear) produce logits of dimension n_experts. Then, we just have to get the probabilities and pick the top K experts. This way, only the experts capable of solving the problem will work on it. Overall, MoE can potentially reduce the economical cost of training models by a great margin.

What to expect?

In this article, we're gonna learn about how MLA and MoE architectures work and also look at the math behind them. By the end of this, you'll know everything (conceptually) you need to make your own homemade DeepSeek V2, load weights and verify the architecture (so, this one's inference only).

Prerequisite knowledge:

• Multi Head Attention (MHA) and Decoder-only Transformer
• RMSNorm
• Rotary Positional Encodings (RoPE)
• SwiGLU
• KV cache

In the transformer block, we change only the attention and the MLP but the normalizations and everything else can stay identical.
Obviously, a faithful implementation of DeepSeek V2 demands a lot more work and verification.

Multi-head Latent Attention

I hope you do realize that MHA has a huge KV cache. The main goal of MLA was to reduce this cache size in order to make models more memory-efficient. This might also result in insane performance gains as the number of operations during attention computation can decrease by a lot with the right configuration.

Basically, attention calculates three things first -

1. Query (Q)
2. Key (K)
3. Value (V)

In MLA, we don't jump to these directly. First a compressed C^KV matrix is created using the input X.
C^KV = W^DKVX
In simpler terms, C^KV = linear^DKV(X).
D is for projection to lower dimensions whereas U is for projection to higher dimensions.

This replaces the KV cache of the normal MHA. K and V are calculated using this C^KV.
K = W^UKC^KV
V = W^UVC^KV

Suppose C^KV is B x T x d_c. The normal KV cache would've been B x T x Hd_h where H is the number of heads.
Compression ratio r = Hd_h/d_c
In Deepseek V3, H = 128, d_h = 128 and d_c = 512 (= 4d_h). So, r = 32.
The ratio for operations during matrix multiplications MLA ops/MHA ops can be calculated in terms of r and we find that r > 1.5 is required for performance gains.

taken from Lior Sinai's blog

For r = 32, MLA ops/MHA ops is approximately 0.047. This means that MLA operations are roughly 21 times less than MHA operations. That's a near 20x speedup btw...

Further, Q can also be made from a compressed C^Q in a way similar to K and V. This is done to save memory during training and looks like -

C^Q = W^DQX
Q = W^UQC^Q

So, overall, creating Q, K and V looks like -

C^Q = linear^DQ(X)
C^KV = linear^DKV(X)

Q = linear^UQ(C^Q)
K = linear^UK(C^KV)
V = linear^UV(C^KV)

Since we're compressing Q too, we now have to do five matrix multiplications. As a result, compression ratio r of a minimum of 1.67 is required for performance gains.

The rest of the process involving applying RoPE, calcuating scores and output projection can be identical to MHA. You can also choose to get more benefits by leveraging the opportunity to absorb weight matrices to be more efficient with the operations.

Simply stated, you can combine the weight matrices as follows -
S = K^TQ (attention scores)
  = (W^UKC^KV)^T(W^UQX)
  = (C^KV)^T((W^UK)^TW^UQ)X
  = (C^KV)^T(W^KQ)X (where W^KQ = (W^UK)^TW^UQ)

This way, we do not need to compute keys and values separately. Absorption on value and output matrices works in a similar way.

I've made it in my repo lantern. There, I've implemented a small Deepseek V2 for learning purposes. I also loaded weights from an implementation on hugging face and found that I got really close when I verified the logits.

I'm not gonna go into more detail about absorption but here are some caveats that I want to mention.

• In some cases, it might not be worth all the effort. Even though it looks easy on paper, there's a lot of problems concerning dimensions of the tensors. For example, there are multiple heads and each head has its own W^UK matrix.
• Absorption cannot be done with RoPE. S_ij becomes (C_i^KV)^T(W_i^UK)^T(R_i^TR_j)(W_j^UQ)(C_j^Q) with RoPE applied to K and Q. You can see that R_i^TR_j comes between the two weight matrices which breaks the absorption property. To overcome this, RoPE has to be decoupled.
  
  These extra complexities might make absorption a no-go if you simply want to make a small model for experimentation but don't want to get your head too messed up in all these details. Though, there are clear benefits that absorption offers during inference.
  

The biggest advantage is that it allows us to skip full K and V reconstruction as we've seen above. Without it, MLA needs to compute K and V from C^KV which adds two matrix multiplications whereas MHA directly uses K and V from the cache. For this reason, if you want performance gains, absorption with decoupled RoPE is totally worth going for!

You can read about the absorption math including dimensional analysis and broadcasting and also on how decoupled RoPE is implemented if you're planning to use it. For that, I recommend you to checkout this blog. This section on DeepSeek's MLA was inspired by it too.

simple MLP Mixture of Experts

A standard MLP works like:

• pass the attention output (say X) into the MLP
• MLP input X goes through some linear layers or SwiGLU depending on the model
• the output is of the shape B x T x d_model

In MoE, there are some shared experts and some routed experts. One expert is just a small MLP/FFN e.g. SwiGLU.

The inputs have to go through s shared experts as they represent knowledge learned for general purposes. A router selects top K routed experts out of all the r routed experts and only those K selected ones work on the inputs as they represent specialized knowledge for producing output on related problems. This is one way to form an intuition about the division of a huge number of parameters into experts.

An MoE works as:

• X goes into the router which is a linear that maps d_model to r where each score is the weight for the router of the same index
• the logits (router's outputs) are softmax-ed and made into probabilities
• top K routed experts are selected and are assigned weights depending on the logits the router gave us and now we begin processing the inputs X
• X goes through the shared experts and the outputs are added
• X goes through the selected routed experts and the weighted outputs are added
• Finally, the contributions from the two types of experts are added together and this is the output of the MoE

MoE(X) = ${\sum }_{i=1}^{s}SharedExper{t}_i(X)$ + ${\sum }_{i=1}^K(W_i*RoutedExper{t}_i(X))$

Again, I've implemented this in lantern.

There are some awesome performance gains that this architecture offers while allowing us to keep a huge model size. This is because there are only a small set of activated experts per input.

As an example, DeepSeek V2 uses roughly 31% of V1's active parameters per token, yet scores 7.2 points higher on MMLU (note that this higher score is not solely because of MoE but depends on several other factors). This is because in V2, only 21B parameters out of the total 236B are active per token compared to V1's 67B which are involved all the time.

And as you can see, this approach allows us to increase the model capacity by a pretty huge factor. For instance, V2 is around 3.5x the size of V1. DeepSeek also says that the training costs of V2 were 42.5% lower than those of V1.

This is obviously not solely because of MoE. You can imagine how controlled and sophisticated the process of training a model like DeepSeek's is on a production level. So, if you want these gains, you need to make sure your model's attention, MoE and optimizations all work in harmony to contribute to the performance speedup as well as the reduction of training and inference costs all while still being able to produce meaningful responses.

Outro

Yep...these are the main advancements that DeepSeek V2 made. When this model came out, it became a topic of discussion everywhere in the ML community because of its speedups and cost efficiency. Indeed, it is a great innovation and studying its architecture and training methodology can be a pretty amazing learning experience.

That said, not every technique or optimization is practical for small experiments and it is advised that you do your own investigation on the parts that you want to include in your implementations and experiments and don't trust everything that's written in their paper.

Also, thank you DeepSeek for being the real OPEN AI.

Feel free to reach out and tell me if you spot any inconsistencies or if you feel that something can be improved.
I would love to hear your feedback on this!
See you in the next one!

Arnav
𝕏 @__iamarnav