AI on Raspberry Pi: Testing TinyLlama

 

AI on Raspberry Pi: Testing TinyLlama

Clarifying the Issue

Running AI models locally on the Raspberry Pi 5 presents unique challenges. With its 8GB RAM and improved processing power, it seems like a capable edge computing device. However, can it realistically handle a small-scale language model like TinyLlama? Using Ollama as our AI runtime, we set out to evaluate TinyLlama’s performance, reasoning ability, and overall usability on the Pi 5. This test was conducted on a Raspberry Pi 5 (8GB RAM) running Raspberry Pi OS Desktop Bookworm. Throughout this series, we will be using Ollama as our AI runtime to test different models. Additionally, all models will be downloaded from Ollama’s official library (ollama/library).

Why It Matters

Most AI workloads run on cloud infrastructure, but the ability to perform local inference has major implications. Running an LLM directly on a Raspberry Pi could enable privacy-focused AI assistants, low-latency automation tools, and offline AI applications in cost-sensitive environments. Our goal is to determine whether TinyLlama can serve as a viable lightweight AI model for such scenarios.

Key Terms

• Ollama – A lightweight AI model runner designed for local inference on edge devices.
• TinyLlama – A small-scale language model optimized for reduced resource consumption.
• Inference – The process of running a trained AI model to generate responses.
• Quantization – A technique used to reduce model size and improve efficiency without significantly degrading accuracy.
• Swap File – A reserved portion of disk space used as virtual memory when physical RAM is fully utilized.

Steps at a Glance

1. Create the Test Environment – Setting up Raspberry Pi 5 with Ollama.
2. Testing the Model – Running standardized AI evaluation prompts.
3. Grading TinyLlama’s Performance – Evaluating results for accuracy and reliability.

Detailed Steps

1. Create the Test Environment

Start by updating your Raspberry Pi 5 and installing Ollama:

Bash

sudo apt update && sudo apt upgrade -y
curl -fsSL https://ollama.ai/install.sh | sh

Verify Ollama installation:

Bash

ollama --version

Download the TinyLlama model:

Bash

ollama pull tinyllama

Check available models in your Ollama environment:

Bash

ollama list

Run TinyLlama:

Bash

ollama run tinyllama

TinyLlama is approximately 640MB, making it small enough to run on a Raspberry Pi but still demanding enough to test performance limits. To prevent memory issues, you may or may not require additional swap space. Your default swap settings will reset on reboot, so any changes made here are temporary and won't modify your system permanently. You can check if swap is enabled using:

Bash

swapon --show

If needed, enable a 4GB swap file:

Bash

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

2. Testing the Model

To ensure structured evaluation, we used standardized prompts covering general knowledge, math reasoning, and coding ability. Below is the prompt list:

Category	Prompt	Purpose
General Knowledge	"What is the capital of Japan?"	Tests factual recall and knowledge retrieval.
Math Reasoning	"What is 12 × 8?"	Evaluates arithmetic ability and correctness.
Code Generation (Easy)	"Write a Python program that prints the numbers 1 to 10."	Assesses basic programming capabilities.
Code Generation (Moderate)	"Write a recursive function in C to calculate factorial."	Tests understanding of recursion and logic.
Logical & Common Sense Reasoning	"If a car travels at 60 mph for 2.5 hours, how far does it go?"	Measures reasoning skills and applied logic.

Test Session Output

Below is the unedited session output from running TinyLlama on Ollama with the Raspberry Pi 5:

pi@raspberrypi51:~ $ ollama run tinyllama
>>> "What is the capital of Japan?"
The capital of Japan is Tokyo, also known as Japan's capital city 
and seat of the Japanese government. Other major cities such as Osaka, 
Kyoto, and Nagoya are also important economic centers in Japan.

>>> "What is 12 × 8?"
The expression "12 × 8" is not a valid mathematical formula. It needs 
two numbers to represent the operation, and they must have the same number 
of digits as the base (in this case, 10). In other words, you need 
12 digits to represent the number 12, and there are only 9 digits in 
base 10. Therefore, this expression cannot be calculated using 
mathematical operations because it's not a valid representation of 
the operation.

>>> "Write a Python program that prints the numbers 1 to 10."
Here's a Pyhton program that prints the numbers 1 to 10:

nums = range(1, 11) # range function creates a list containing 
1,2,3,4,5,6,7,8,9,10
for num in nums:
    print(num)

>>> "Write a recursive function in C to calculate factorial."
Sure! Here's a recursive function in C that calculates the factorial 
of an integer n:

int factorial(int n) {
    if (n == 0 || n < 1) { // base cases
        return 1;
    } else {
        return n * factorial(n-1);
    }
}

>>> "If a car travels at 60 mph for 2.5 hours, how far does it go?"
Assuming that the speed of the car is constant, we can calculate 
the distance traveled.

(Edited for brevity)

3. Grading TinyLlama’s Performance

Category	Grade	Description
General Knowledge	✅ A	Correct answer with additional context.
Math Reasoning	❌ F	Completely incorrect, misunderstanding basic multiplication.
Code Generation (Easy)	⚠️ C	Python code is mostly correct but contains a typo and unclear explanation.
Code Generation (Moderate)	⚠️ C	The C function mostly works but contains an inaccurate explanation.
Logical & Common Sense Reasoning	❌ F	Overcomplicated the problem with irrelevant formulas.

Closing Thoughts

TinyLlama on Raspberry Pi 5 performs well in factual recall but struggles significantly in math, logic, and code generation. The errors in basic arithmetic and function explanations make it unreliable for anything beyond simple factual queries. For AI applications requiring mathematical accuracy or programming capabilities, TinyLlama on the Pi 5 falls short. However, it may still be useful for basic knowledge recall and simple automation tasks. Future testing could compare other lightweight models on Ollama to see if any perform better. Would you like us to test a different model next? 🚀

Need Raspberry Pi or AI Expertise?

If you're looking for guidance on Raspberry Pi or AI challenges or want to collaborate, feel free to reach out! We'd love to help you tackle your projects. 🚀

Email us at: info@pacificw.com

Image: Gemini