You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "amogh-jahagirdar (via GitHub)" <gi...@apache.org> on 2026/10/15 21:11:21 UTC

[PR] Spark 4.2: Add repro test for replayed streaming epoch overwriting committed data files [iceberg]

amogh-jahagirdar opened a new pull request, #18531:
URL: https://github.com/apache/iceberg/pull/18531

   Repro for a case where a Spark Streaming data file ends up with different contents than the manifest says (record count, bounds).
   
   Sequence of events:
   
   1. Epoch N runs. Tasks write files named {partitionId}-{taskId}-{queryId}-{epochId}-{fileCount}.parquet.
   2. Iceberg commits the snapshot for epoch N, with the files and their stats in the manifest.
   3. The driver dies before Spark writes the checkpoint commit log entry for N.
   4. Driver restarts on the same checkpoint. Offset log has N, commit log doesn't, so Spark replays N.
   5. taskAttemptId is a per-SparkContext counter that restarts from 0 on the new driver. If a replay task lands on the same counter value as the original (e.g. the first batch of a scheduled AvailableNow run), it gets the same partitionId and taskId, so the same file name.
   6. The replay can assign different rows to that partition (shuffle, AQE coalescing), and data writers use overwrite, so the committed file at that path is replaced.
   7. commit(N) sees epochId <= lastCommittedEpochId and skips. Nothing is cleaned up or flagged.
   8. Table now points at a file whose rows don't match its manifest entry.
   
   The test does 1, 2 and 4 to 6 directly against the streaming writer, then checks the committed file's row count against the manifest. It fails on current code. Fix not included.
   
   ---
   **AI Disclosure**
   - Model: Claude Sonnet 5.5
   - Human Oversight: Iterated a bunch, had it search for cases where spark streamign can still clobber files because that's the only way this could possibly be SS specific (if it was how stats were actually collected from footer and put into metadata this issue would be way more prevalent)
   - Prompt Summary: Investigate Spark Streaming data file / manifest stats mismatch and write a minimal repro test for the replayed epoch overwrite


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Add repro test for replayed streaming epoch overwriting committed data files [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar closed pull request #18531: Spark 4.2: Add repro test for replayed streaming epoch overwriting committed data files
URL: https://github.com/apache/iceberg/pull/18531


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Spark 4.2: Add repro test for replayed streaming epoch overwriting committed data files [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #18531:
URL: https://github.com/apache/iceberg/pull/18531#issuecomment-6202611405

   Closed this, meant to put this on my own fork and discuss with people and confirm before adding ai noise to the main repo.....


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org