The Secret Life of JavaScript: IndexedDB

How store gigabytes of data client-side with IndexedDB

#JavaScript #IndexedDB #WebPerformance #Storage

Margaret is a senior software engineer. Timothy is her junior colleague. They work in a grand Victorian library in London — the kind of place where code quality is the unspoken objective, and craftsmanship is the only thing that matters.

Episode 38

Timothy watched the red error text bloom across his developer console:

Uncaught DOMException: Failed to execute 'setItem' on 'Storage': Setting the value of 'username' exceeded the quota.

"The infinite scroll and the streaming NDJSON engine are handling the data beautifully," Timothy told Margaret as she walked up, sliding a fresh mug of dark roast onto his desk. "But the moment a user reloads the application, we have to pull all 100,000 diagnostic records across the network all over again. I tried to cache the parsed log array inside localStorage so it would persist across sessions, but it crashed the moment the dataset crossed 5 megabytes."

The String Trap

Margaret looked at his failing code, which was forcing a massive array of JavaScript objects through JSON.stringify().

"Why did it crash at 5 megabytes, Timothy?" Margaret asked.

"Because that is the hard storage limit enforced by the browser for localStorage," Timothy sighed.

"And how does localStorage store that data under the hood?" Margaret pressed.

Timothy thought for a second. "As a raw, synchronous string."

"Exactly," Margaret said. "You are taking a highly optimized, asynchronous data stream, blocking the main thread to convert it into a single gargantuan string, and trying to force it into a synchronous key-value store built for simple user preferences. You are treating the browser like a basic cookie jar when you should be treating it like a production database server."

The Transactional Store

"But what other choice do I have if I want data to survive a page refresh?" Timothy asked.

Margaret stepped to the whiteboard and wrote indexedDB.open().

"Welcome to IndexedDB," Margaret explained. "Unlike localStorage, which blocks execution and caps you at a meager 5 megabytes, IndexedDB is a fully asynchronous, transactional, object-oriented database built right into the browser. It doesn't care about strings—it stores raw JavaScript objects natively. More importantly, its storage limit isn't a hard-coded 5MB; it can scale to consume up to 50% or more of the user's free hard drive space. Gigabytes of data, stored safely on the client side without blocking a single frame of your UI."

The Streaming Marriage

Following Margaret’s architectural blueprint, Timothy realized he didn’t need to wait for his NDJSON stream to finish completely before saving it. He could marry the Streams API directly to an IndexedDB transaction, writing objects to the local database as they arrived through the network pipe.

He wrapped the native, event-driven IndexedDB boilerplate into a clean, modern Promise-based utility, ensuring proper database schema creation and transaction lifecycle management.

// log-storage.js - Native IndexedDB Orchestration
const DB_NAME = 'TechReaderStorage';
const STORE_NAME = 'diagnostic-logs';

// 1. Initialize and upgrade the database schema natively
export function initDB() {
  return new Promise((resolve, reject) => {
    const request = indexedDB.open(DB_NAME, 1);

    request.onupgradeneeded = (event) => {
      const db = event.target.result;
      // Create an object store using the log's unique ID as the primary key
      db.createObjectStore(STORE_NAME, { keyPath: 'id' });
    };

    request.onsuccess = () => resolve(request.result);
    request.onerror = () => reject(request.error);
  });
}

// 2. Stream-aware bulk insertion using a high-performance transaction
export function createWriteStream(db) {
  // Open a readwrite transaction across the object store
  const transaction = db.transaction(STORE_NAME, 'readwrite');
  const store = transaction.objectStore(STORE_NAME);

  // Return an interface for the stream writer to pump data into
  return {
    write(record) {
      store.put(record); // Natively stores raw JS objects asynchronously
    },
    commit() {
      return new Promise((resolve) => {
        transaction.oncomplete = () => resolve();
      });
    }
  };
}

Timothy integrated the database writer directly into his response stream reader from Volume 31:

// main.js - Direct Stream-to-DB Pipeline
import { initDB, createWriteStream } from './log-storage.js';

async function streamLogsToLocalDatabase() {
  const db = await initDB();
  const dbStream = createWriteStream(db);

  const response = await fetch('/api/logs-stream');
  const reader = response.body
    .pipeThrough(new TextDecoderStream())
    .pipeThrough(new NDJSONParseStream()) // From Vol. 31
    .getReader();

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;

    // Pump the parsed object straight into the database mid-transit
    dbStream.write(value);
  }

  await dbStream.commit();
  console.log('All logs streamed and safely committed to IndexedDB.');
}

Timothy saved the files, cleared his network cache, and triggered the pipeline. As the NDJSON chunks streamed in, they bypassed localStorage entirely, writing thousands of raw objects silently into IndexedDB.

He opened the browser's Application tab. Under IndexedDB, a beautifully indexed table of tens of thousands of diagnostic records sat completely intact. He refreshed the page—the application immediately queried the local database instead of hitting the network, populating his infinite scroll grid instantly with zero network latency.

By marrying the asynchronous power of the Streams API with the transactional capacity of IndexedDB, Timothy broke through the 5-megabyte storage wall, providing a lightning-fast, offline-capable data architecture entirely on the client side.

Aaron Rose is a software engineer and technology writer at tech-reader.blog.

Catch up on the latest explainer videos, podcasts, and industry discussions below.

Search This Blog

Tech-Reader.blog