~7.5GB - it'll be the same as running inference with a full context. That's for ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		eulers_secret on May 4, 2023 \| parent \| context \| favorite \| on: Google “We have no moat, and neither does OpenAI” ~7.5GB - it'll be the same as running inference with a full context. That's for 4-bit quantization, the 8-bit quantization uses more RAM than my 3080 has...

seydor on May 4, 2023 [–]

I wonder how much it would take to train the 4b 13B

MacsHeadroom on May 4, 2023 | [–]

About 15GB training it in the webui.

If you use https://github.com/johnsmith0031/alpaca_lora_4bit then 30B only needs 24GB, and works on a single 3090 or $200 P40.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact