I'm using https://github.com/ggerganov/llama.cpp and currently mistral 7b (on a m1 macbook pro). I'm sure with some prompt examples you can get pretty good results on a smaller model.

At the moment I don't have it open sourced due to it being part of a larger project that I'm working on that contains tailwindui licensed components.

A cool feature that I'm working on is creating a firefox plugin so you can save/index job postings from other sites and extract out meta information via an LLM. Very similar to this chrome plugin. https://chromewebstore.google.com/detail/huntr-job-search-tr...

Love the extension idea, definitely useful for tracking opportunities

Thank you for the perspective on running local LLMs

Do you think that saas or local LLMs will be the most popular in the next 5 years?

Hoping it's local :)