Offline RAG with Small Language Models Optimized for Resource-Constrained Environments: Natural Disaster Safety Advice

We present an offline RAG system combined with small language models (sLMs) below 1B parameters designed to function on consumer-grade hardware with CPU only. Frontier AI models require continuous internet access, which is incompatible with resource-constrained environments with unreliable internet access. Additionally, dependence on clouds raises privacy concerns for personal information or proprietary data. Local models can address these resource bottlenecks since they do not require internet access and user data remains within the device. However, in resource-constrained use cases, memory constraints limit user options to small language models with significantly fewer parameters and higher hallucination risks. For domains where accuracy is critical, these drawbacks compromise the viability of local models as an alternative. To optimize for resource-constraints while mitigating the limitations of sLMs, we employed the following methods. First, we offer users a choice between 3 sub 1B parameter sLMs (qwen2:0.5b, qwen3:0.6b, qwen3.5:0.8b). We also leveraged RAG to ground sLM output with retrieved context and utilized a system prompt with guidelines for safety and accuracy. Additionally, we visualized and logged performance data such as response time and documents retrieved from similarity search to create an auditable trail. To prevent resource barriers, user-facing parts of this architecture is built with open-source frameworks only. Initial qualitative testing of our proof-of-concept architecture surfaced two failure modes. Few-shot examples initially included in the system prompt caused severe context contamination. This was mitigated by transitioning to a zero-shot system prompt with basic safety guidelines only. Additionally, persisting the vector index across different queries reduced end-to-end user response latency from over 60 seconds to 5-9 seconds. We then developed a prototype set to deliver emergency safety advice during extreme weather conditions and natural disasters. Vulnerable groups such as children and elderly people face disproportionate risks from natural disasters, which highlights the need for a reliable, offline source of safety guidance. We aim to develop our current prototype into a scalable solution that can be distributed for extreme weather and disaster relief.