From Noise to Signal: Cleaning Industrial Sensor Data with Python and Pandas

In a perfect world, sensors provide a continuous stream of clean, predictable measurements. On a real plant floor, the physical environment is hostile. Electromagnetic interference, transient voltage drops, and packet drops turn telemetry streams into ragged logs littered with missing timestamps, phantom spikes, and missing payloads.

Feed raw sensor telemetry directly into an operational dashboard or downstream model, and you aren't monitoring machinery—you're charting sensor failure modes.

Here is a straightforward pipeline to clean noisy motor telemetry. We take a dirty JSON extract, enforce physical operational boundaries in Pandas, interpolate genuine signal gaps, and export dual deliverables: flat CSV for reporting and structured ISO-8601 JSON for application services.

The Raw Telemetry: Dirty Input

Hardware failure modes leave distinct fingerprints in JSON streams. Notice three classic telemetry faults below: an orphaned reading with a missing timestamp, an impossible 999.9 hardware fault code, and a dropped packet represented by a null reading.

motor_sensor_dirty.json

[
  {"timestamp": "2026-11-08T10:00:00", "temp": 42.5},
  {"timestamp": "2026-11-08T10:01:00", "temp": 43.1},
  {"timestamp": null, "temp": 42.8},
  {"timestamp": "2026-11-08T10:03:00", "temp": 999.9},
  {"timestamp": "2026-11-08T10:04:00", "temp": 44.2},
  {"timestamp": "2026-11-08T10:05:00", "temp": null},
  {"timestamp": "2026-11-08T10:06:00", "temp": 45.0}
]

The Pipeline: Clean, Enforce, Interpolate

Rather than scattering ad-hoc cleanup steps, the script treats the telemetry as a structured pipeline with explicit physical constraints (a motor operating between 0°C and 150°C).

import json
import numpy as np
import pandas as pd

# --- 1. INGEST ---
with open('motor_sensor_dirty.json', 'r') as f:
    df = pd.DataFrame(json.load(f))

# --- 2. VALIDATE TIMESTAMPS ---
# Unparseable or null timestamps invalidate time-series alignment; drop them first
df['timestamp'] = pd.to_datetime(df['timestamp'], errors='coerce')
df = df.dropna(subset=['timestamp']).sort_values('timestamp')

# --- 3. ENFORCE PHYSICAL BOUNDARIES ---
# Clamp to realistic motor operating envelope (0°C to 150°C); map spikes to NaN
TEMP_MIN, TEMP_MAX = 0.0, 150.0
df.loc[(df['temp'] < TEMP_MIN) | (df['temp'] > TEMP_MAX), 'temp'] = np.nan

# --- 4. RECONSTRUCT SIGNAL ---
# Linear interpolation bridges brief packet drops without flattening trends
df['temp'] = df['temp'].interpolate(method='linear').round(2)

# --- 5. EMIT DELIVERABLES ---
df.to_csv('motor_sensor_clean.csv', index=False)
df.to_json('motor_sensor_clean.json', orient='records', date_format='iso')

The Audit Trail: Handling Fault Modes

Instead of hiding data transformations, a reliable pipeline accounts for every dropped or modified value:

Orphaned Row (null timestamp): Action: Dropped immediately via dropna(subset=['timestamp']). Rationale: Sensor values without a verified temporal coordinate cannot be aligned in a time series.
Hardware Error Code (999.9 temp): Action: Converted to NaN via boundary check, then interpolated to 43.65°C. Rationale: Prevents a diagnostic error code from blowing out rolling motor averages.
Transient Packet Loss (null temp): Action: Linearly interpolated between 10:04 and 10:06 to 44.60°C. Rationale: Thermal mass does not jump instantaneously; linear estimation preserves physical continuity across single dropped cycles.

Stakeholder Outputs

Downstream consumers need different contracts. The pipeline produces two distinct formats from the same validated DataFrame:

1. Operations & Reporting (`motor_sensor_clean.csv`)

A compact, two-column flat file that opens natively in spreadsheets or business intelligence tools:

timestamp,temp
2026-11-08 10:00:00,42.5
2026-11-08 10:01:00,43.1
2026-11-08 10:03:00,43.65
2026-11-08 10:04:00,44.2
2026-11-08 10:05:00,44.6
2026-11-08 10:06:00,45.0

2. Applications & APIs (`motor_sensor_clean.json`)

A standardized ISO-8601 payload ready for microservices or database ingestion:

[
  {"timestamp": "2026-11-08T10:00:00.000Z", "temp": 42.5},
  {"timestamp": "2026-11-08T10:01:00.000Z", "temp": 43.1},
  {"timestamp": "2026-11-08T10:03:00.000Z", "temp": 43.65},
  {"timestamp": "2026-11-08T10:04:00.000Z", "temp": 44.2},
  {"timestamp": "2026-11-08T10:05:00.000Z", "temp": 44.6},
  {"timestamp": "2026-11-08T10:06:00.000Z", "temp": 45.0}
]

Takeaway

Industrial data plumbing is never about accepting numbers at face value. By pairing real-world mechanical domain knowledge (valid operating temperatures) with robust time-series interpolation, you convert raw operational static into dependable telemetry your downstream systems can trust.

Search This Blog

Tech-Reader.blog