Using LLM Locally

The talk argues that running language models on your own computer keeps your data away from online services and allows more customization than a hosted chat API. The speaker introduces LM Studio, an open-source desktop application available on several platforms. It offers an interface for finding, downloading and chatting with models, though it can feel complex for someone new to language models.

The demo starts with searching for a model and downloading a small model from Microsoft directly inside the app. The app estimates whether a given model will run well on the machine, using color indicators to show poor or workable fits. The speaker notes that lightweight models can run on modest hardware, such as a laptop with 8 GB of RAM, though more slowly. The chat view exposes settings such as how strict or creative the output should be, and models can be loaded from a list once downloaded.

The last part covers the local server, which lets code or other applications call a loaded model through an API. A planned demonstration of chatting with your own documents, using retrieval to add context to questions, was not shown because of the time limit and a software crash. The session closed with an offer to answer questions about installation and what hardware is needed.