Using a Transformer Model: From Training to Inference
Short excerpt below. Click through to read at the original source.
This chapter is divided into four parts; they are: • Autoregressive Generation • Prefill and Decode • A Simple KV Cache • Memory Usage of the KV Cache A decoder-only transformer model predicts the next token from the tokens that came before it.