Correct me if I am wrong, to use LORA fine-tuned model in inference you would still need the original model + trained additional layers, right?

If we can perfect methods to fine-tune large models for specific task while reducing the overall model size, then it can fit into more consumer grade hardware for inference and can be broadly used. The objective is to prune unnecessary trivia and memorization artifacts from the model and leverage LLMs purely for interpreting natural language inputs.

> to use LORA fine-tuned model in inference you would still need the original model + trained additional layers, right?

You don't need additional layers. After training, the product of the two matrices is added to the original weights matrix, so the model size remains the same as the original during inference.

Yes you still require the original model weights to use LoRA layers. For many LLaMA based models you need to find the original weight yourself and then apply the LoRA diff on top of that.

I had to read the paper first, but yeah, that diagram is shockingly simple once you get it.

Some annotations:

- The labels in the orange boxes mean "A is initialized with random weights (in a gaussian distribution, B is initialized with weights set to zero".

- d is the number of values of the layer's input and output. (The width of the input and output vectors, if you will.)

- r is the number of "intermediary values" between A and B. It's expected to be a lot smaller than d, hence "Low Rank" (apparently LoRa even works with r = 3 or so), but it can be equal to d, though you lose some of the perf benefits.