What is Late-Arriving Data?

How it works

Every event has an event time (when it happened) and an arrival time (when the pipeline received it). If you partition by arrival date, late data is never lost but lands in the wrong day. If you partition by event date, a late event changes a day that consumers have already seen.

The usual batch pattern for event-date tables:

1. Select new rows by ingestion time for this run’s window, which is deterministic and never misses a row.
2. Find the distinct event dates those rows belong to.
3. Rebuild just those partitions of the aggregate from all their rows, idempotently, in one transaction.

Late data also affects dimensions. A fact can arrive before the customer or product row it references. Rather than dropping the fact or giving it a null key, insert a placeholder (“inferred”) dimension member with the natural key, point the fact at it, and fill in its attributes when the real row arrives.

Streaming engines solve the same problem with an event-time watermark: windows stay open for an allowed lateness, and events that arrive after the window closes are dropped or sent to a side output.

The technical part is only half the answer. Agree a lateness horizon with consumers (“daily figures can change for three days, then they are final”), what happens to events beyond it, and how completeness is shown.

Example

Today’s load contains one event from 4 October. Only the dates present in the new batch are rebuilt, so 4 October moves from 10 to 15 and rerunning gives the same result.

CREATE TABLE events (event_id int, event_date date, ingested date, amount int);
INSERT INTO events VALUES
  (1, '2026-10-04', '2026-10-04', 10),
  (2, '2026-10-05', '2026-10-05', 20),
  (3, '2026-10-04', '2026-10-05', 5);   -- arrived a day late

CREATE TABLE daily_totals (event_date date PRIMARY KEY, total int);
INSERT INTO daily_totals VALUES ('2026-10-04', 10);

-- Run for ingestion date 2026-10-05: rebuild every event date it touched.
BEGIN;
CREATE TEMP TABLE touched AS
  SELECT DISTINCT event_date FROM events WHERE ingested = DATE '2026-10-05';
DELETE FROM daily_totals WHERE event_date IN (SELECT event_date FROM touched);
INSERT INTO daily_totals
SELECT event_date, sum(amount) FROM events
WHERE event_date IN (SELECT event_date FROM touched)
GROUP BY event_date;
COMMIT;

SELECT * FROM daily_totals ORDER BY event_date;

 event_date | total
------------+-------
 2026-10-04 |    15
 2026-10-05 |    20

Common mistakes

• Selecting each run’s input by event time, so an event that arrives late is never picked up.
• Adding late events onto an existing aggregate instead of rebuilding the partition, which double-counts on a rerun.
• No agreed horizon, so last month’s numbers change without warning.
• Dropping facts whose dimension row has not arrived yet.