Increasing inference pipeline stages reduces weights memory footprint but leaves KV cache (activation) memory constant.

Running a frontier-scale model of a few trillion parameters on Groq LPUs would require thousands of chips just to hold the weights, due to insufficient on-chip SRAM.

“If you think about it to run a frontier level model of like let's say a few trillion parameters you need thousands and thousands of Grock LPUs just to hold the weights.”

Increasing pipeline stages reduces weights memory footprint but does not reduce KV cache (activation) memory footprint, which remains constant.

“If you increase the number of pipeline stages, the memory footprint for the number of weights keeps going down and down and down, but the memory footprint for the number of activations stays constant.”