Running and Fine-tuning Open Source LLMs on Google Kubernetes Engine

### Overview In this presentation, speaker N Frey discusses the architecture, benefits, and best practices for running and fine-tuning open-source large language models (LLMs) on Google Kubernetes Engine (GKE). Frey outlines why organizations choose open-source LLMs over closed-source alternatives—focusing on cost-effectiveness, customization, community support, and regulatory compliance. The talk covers how Kubernetes acts as a portable, scalable control plane, how GKE integrates hardware accelerators like GPUs and TPUs, and how to execute model loading, serving, and parameter-efficient fine-tuning (PEFT).

### Key Topics - **Why Open Source LLMs**: Benefits including cost control through infrastructure-as-code (e.g., shutting clusters down after use), tailoring models to specialized industry vocabularies (e.g., healthcare), and meeting strict compliance or privacy requirements (such as in financial services) where data cannot be shared with third parties. - **Kubernetes and GKE Capabilities**: Portability across multi-cloud and on-premise environments, dynamic autoscaling, and managed Google Cloud integrations including native support for frameworks like Ray, Spark, TensorFlow, PyTorch, Jax, and Cuda. - **Accelerator Provisioning and Sharing**: Utilizing Node Auto-Provisioning (NAP) to automatically spin up TPU or GPU node pools (e.g., Nvidia L4) when workload demand spikes. Three primary GPU sharing strategies are evaluated: Multi-Instance GPU (MIG) for hardware isolation, Multi-Process Service (MPS) for software isolation, and GPU time-sharing for batch or sequential workloads. - **Serving Frameworks**: Contrasting Hugging Face Text Generation Inference (TGI) with vLLM regarding throughput, memory utilization, and fine-tuning options. - **Customization Techniques**: Comparing prompt engineering (few-shot learning, chain-of-thought), prompt learning (p-tuning, prompt tuning via LSTMs), and Parameter-Efficient Fine-Tuning (PEFT, including adapters and LoRA) versus full-parameter fine-tuning.

### Technical Details - **Model Storage and Checkpointing**: Demonstrating the use of GKE GCS Fuse (`gke-gcsfuse`) to mount Google Cloud Storage buckets directly into pods/containers as volumes, avoiding slower Cloud Storage API calls and supporting high-throughput read operations and training checkpoint management. - **Job Orchestration**: Employing Kueue, a Kubernetes-native job queuing system, to handle job prioritization, resource allocation, and scheduling for batch fine-tuning tasks. - **Demo Walkthrough**: Frey shows an implementation utilizing a Google Cloud GKE cluster (`AI on GKE`) with Nvidia L4 GPUs. A Kubernetes model-loader job pulls weights from Hugging Face into a GCS bucket, and a pod runs Llama 2 (evaluated up to 70B parameters with 4-bit/8-bit quantization options) served via TGI and vLLM. Frey connects via port-forwarding to execute sample inference requests.

### Takeaways Running open-source models on GKE provides enterprises with deep control over sensitive data, hardware costs, and model architectures. Leveraging GCS Fuse for storage, Kueue for scheduling, and techniques like LoRA for parameter-efficient adaptation creates a scalable, resilient platform for hosting and fine-tuning LLMs on cloud infrastructure.