Cleaning Dirty Raspberry Pi Sensor Data with Python and Pandas
Cleaning Dirty Raspberry Pi Sensor Data with Python and Pandas
Sensor data rarely arrives perfectly clean.
A Raspberry Pi might spend hours collecting temperature readings, quietly adding one measurement after another to a CSV file. Eventually somebody needs that data. A manager may want to open it in Excel. A developer may want to feed it into another program.
Before either happens, we need to clean it.
For this example, we'll work entirely from the Linux terminal in
Raspberry Pi OS, using Python and Pandas. Pandas provides
read_csv()
for bringing CSV data into a DataFrame and
to_csv()
and
to_json()
for writing the results back out.
Our basic pipeline is simple:
dirty CSV → Pandas → clean data → CSV + JSON
The Dirty Sensor Data
Suppose a temperature sensor produced a file called
temperature.csv:
timestamp,temp
2026-11-02 08:00:00,72.4
2026-11-02 08:05:00,72.7
2026-11-02 08:10:00,
2026-11-02 08:15:00,73.1
2026-11-02 08:15:00,73.1
bad_timestamp,74.0
2026-11-02 08:25:00,999.0
2026-11-02 08:30:00,73.6
There are only eight readings, but several things have gone wrong.
One reading has no temperature. Another is duplicated. One timestamp isn't
really a timestamp at all. And
999.0
is clearly not a believable room-temperature reading.
That's dirty data.
Installing Pandas
From the Raspberry Pi terminal, we'll create a virtual environment and install Pandas:
python3 -m venv sensor-env
source sensor-env/bin/activate
python3 -m pip install pandas
Now create a program:
nano clean_sensor.py
Cleaning the Data
Here's our program:
import pandas as pd
# INPUT
df = pd.read_csv("temperature.csv")
# PROCESSING
df["timestamp"] = pd.to_datetime(
df["timestamp"],
errors="coerce"
)
df["temp"] = pd.to_numeric(
df["temp"],
errors="coerce"
)
# Remove missing or invalid values
df = df.dropna(subset=["timestamp", "temp"])
# Remove duplicate readings
df = df.drop_duplicates()
# Keep only plausible room-temperature readings
df = df[df["temp"].between(40, 120)]
# Put the readings in time order
df = df.sort_values("timestamp")
# OUTPUT
df.to_csv(
"temperature_clean.csv",
index=False
)
df.to_json(
"temperature_clean.json",
orient="records",
date_format="iso",
indent=2
)
print(df)
print("\nCreated temperature_clean.csv")
print("Created temperature_clean.json")
Run it:
python3 clean_sensor.py
The program first loads the CSV into a Pandas DataFrame. The timestamp column is converted into actual datetime values, while the temperature column is converted into numbers.
The important part is
errors="coerce". Bad values become missing values rather than crashing the entire pipeline.
We can then remove those bad records deliberately.
We also remove duplicates and reject temperatures outside the range we decided makes sense for this particular sensor application.
That range is a rule we chose for this example, not something Pandas automatically knows.
One Dataset, Two Audiences
After cleaning, the useful readings look like this:
2026-11-02 08:00:00 72.4
2026-11-02 08:05:00 72.7
2026-11-02 08:15:00 73.1
2026-11-02 08:30:00 73.6
But we're saving them twice.
temperature_clean.csv
is useful for someone who wants to open the information in Excel and work with
rows and columns.
temperature_clean.json
is aimed at software. JSON gives another Python program, web application, API,
or other system a structured representation of the same cleaned observations.
The important point isn't the file format.
It's the pipeline.
We started with messy measurements from the physical world. Python and Pandas converted them into clean, predictable data. Then we produced two outputs for two different consumers.
Sensor → dirty CSV → Python/Pandas → clean data → CSV for people + JSON for software
That's a small data pipeline running on a Raspberry Pi.
And that clean data is now ready for whatever comes next.
.jpeg)
