In the most recent session from the Responsible AI Programme (presented by ustwo and CAT), Evan Hahn, a self-proclaimed “fan, not expert”, presented what he knows about local large language models.
Local LLMs are like their more famous cloud counterparts—ChatGPT, Claude, Gemini, Mistral—but instead of running in the cloud, they live entirely on your device. When you use cloud models, your query gets processed in a data center before being sent back to your device. By contrast, local ones do all the thinking on your laptop, phone, or smart fridge.
Local models have two significant advantages over their cloud counterparts:
- Offline: cloud models require the internet. Local models don’t, so you can use them if you’re traveling without a connection.
- Private: many people—Evan included—don’t trust cloud providers with every query. Local LLMs process everything on your device, so you can ask questions freely, without worry of surveillance or interference.
But they have two big disadvantages:
- Worse results: your laptop is probably less powerful than the beefy GPUs in a data center. That means that the results are generally worse for the same queries.
- Harder to use: it’s not too bad, but setting up a local model is harder than simply visiting a website and getting started.
There are also two parts that are sometimes better with cloud models and sometimes better with local ones:
- Environmental impact: any LLM usage is harmful to the environment, but small models are greener because they need less power, and local models are more likely to be small. But that if you’re running the same model locally versus the cloud, the cloud is probably more efficient per query.
- Cost: local models cost electricity. Depending on your usage, it might be cheaper to use a cloud model, but it might be more expensive.
Evan gave a demo showing things that local models are good at: summarization, proofreading, basic coding, and more. He also showed some specialized models, like one tuned for translation. For mainstream queries, local LLMs may be good enough.
For more resources, including the slides, see Evan’s page for this talk.