The Secret Life of JavaScript: Handling Streaming JSON

How to use the Accumulator Pattern to handle NDJSON chunks

#JavaScript #WebDev #StreamsAPI #NDJSON

---

Margaret is a senior software engineer. Timothy is her junior colleague. They work in a grand Victorian library in London — the kind of place where code quality is the unspoken objective, and craftsmanship is the only thing that matters.

Episode 32

The Suspended Fragment

Timothy leaned closer to his monitor, tracing the lines of the NDJSON accumulator they had written the day before. The progressive rendering was working flawlessly, but one specific line of logic was gnawing at him.

"I understand why we split the buffer by the newline character," Timothy said to Margaret as she passed by. "But I don't understand this exact sequence. We split the string into an array of lines, and then immediately call lines.pop() to remove the last item and shove it right back into the buffer."

Margaret looked at the code on his screen:

const lines = buffer.split('\n');
buffer = lines.pop(); 
for (const line of lines) {
  const user = JSON.parse(line);
  renderUserGrid(user);
}

"It feels like we are losing data," Timothy continued. "If we take the last item out of the array before the loop runs, doesn't that mean we are skipping a user?"

The Sentence and the Period

"Are we skipping a user, or are we simply waiting for them to finish speaking?" Margaret asked.

Timothy frowned. "What do you mean?"

"Think of the newline character \n as a period at the end of a sentence," Margaret explained. "You wouldn't try to read a sentence that doesn't have a period yet, would you?"

"No, because the thought isn't complete," Timothy answered.

"Exactly," Margaret smiled. "When you stream data over a network, the chunks arrive arbitrarily. The network does not care about your JSON objects. It will mercilessly slice a chunk right in the middle of a word. If chunk one ends with {"i, what happens when you try to parse it?"

"The parser crashes," Timothy said, "because it's an incomplete fragment."

"Right. So when you split your incoming chunk by the newline character, what do you know with absolute certainty about the very last string in that resulting array?"

Timothy thought about the math. The split() function separates strings at the delimiter. That means everything before a delimiter is complete, but whatever trails after the final delimiter has not been closed out yet.

"The last item doesn't have a newline after it," Timothy realized. "It's an incomplete fragment."

The Accumulator Pattern

"Precisely," Margaret said. "The lines.pop() command isn't throwing data away. It is an act of preservation. It carefully removes that incomplete, trailing fragment from the array so your for loop doesn't attempt to parse it. It holds that fragment safely in the buffer until the next network chunk arrives."

Margaret picked up a dry-erase marker and illustrated the step-by-step flow on the whiteboard:

Chunk 1:  {"id":1}\n{"id":2}\n{"i
          ↓ split by \n
          ["{"id":1}", "{"id":2}", "{"i"]
          ↓ pop() → buffer = "{"i"
          Render user 1 and 2 ✅

Chunk 2:  d":3}\n
          ↓ append to buffer → "{"id":3}\n"
          ↓ split by \n
          ["{"id":3}", ""]
          ↓ pop() → buffer = "" (empty)
          Render user 3 ✅

"If chunk one arrives," she explained, pointing to the top block, "your split array gives you two complete users and one fragment. The pop() method safely extracts {"i and stores it. You successfully render users one and two."

"And then chunk two arrives with d":3}\n," Timothy finished the thought, tracing the logic down to the second block. "We append it to the buffer, which gives us {"id":3}\n. Now it has a period. It's complete."

The Trailing Requirement

"You've nailed it," Margaret said. "The logic is highly deterministic. If there is a newline, the object is complete. If there is no newline, keep it in the buffer and wait. No complex bracket counting. Just a simple character check."

"Wait," Timothy paused, looking back at the whiteboard. "If pop() always removes the last item to wait for the next chunk... what happens at the very end of the entire download? Won't the final user get stuck in the buffer forever?"

"Only if the backend engineers made a critical mistake," Margaret noted. "For NDJSON streaming to work perfectly, the server must send a trailing newline at the absolute end of the file. If that final period is there, the last item popped off the array will just be a harmless empty string, and every single user object will have been safely processed."

---

Aaron Rose is a software engineer and technology writer at tech-reader.blog. 

Catch up on the latest explainer videos, podcasts, and industry discussions below.