Inference
Start with the model, then see what happens when it runs.
Inference is using an already trained model to produce an output from an input. For a text-generating language model, this usually means processing a prompt and generating tokens in sequence.
[Hugging Face]In practice
A support application supplies a question to its model and receives a draft answer. That model run is inference.
[Hugging Face]A little deeper
In text generation, prefill processes the supplied input; decoding then produces further tokens. This is the model running, rather than a training step that adjusts its weights. [Hugging Face]
A common mix-up
Every answer trains the model.
Producing an answer does not by itself update the model’s weights. [Hugging Face]
Helpful to know: Language Model