You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 11:42:54 UTC
[I] Revisit TestData methods [hudi]
hudi-bot opened a new issue, #16860:
URL: https://github.com/apache/hudi/issues/16860
`TestData::checkWrittenData` doesn't expect data with buckets, when we have multiple base files in one partition. So `ITTestBucketStreamWrite::testBucketStreamWriteAfterRollbackFirstFileGroupCreation` works only because `bucketNum` is set to 1.
We need to fix it with revisiting of `TestData::checkWrittenData`, `TestData::checkWrittenDataCOW`, `TestData::checkWrittenDataMOR` to prevent such mistakes in future.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9056
- Type: Task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Revisit TestData methods [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #16860:
URL: https://github.com/apache/hudi/issues/16860#issuecomment-5493297632
Closing as fixed: the reported defect no longer reproduces on `master` (checked at `1e59c9eb6d9e`).
### Only one of the three methods was ever affected
At the time this was filed (`358ecc77a74c`, 2025-02-21), `TestData::checkWrittenData` selected a single base file per partition and read only that one:
```java
File latestDataFile = Arrays.stream(dataFiles)
.max(Comparator.comparing(f -> FSUtils.getCommitTime(f.getName())))
.orElse(dataFiles[0]);
```
So with more than one bucket in a partition, every other base file went unchecked. That matches the report exactly.
On current `master` it collects every latest file slice that has a base file, reads all of them into one buffer, sorts, and compares:
https://github.com/apache/hudi/blob/1e59c9eb6d9e341d5fd10e1418c0c3deb34b6579/hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/utils/TestData.java#L837-L883
`TestData::checkWrittenDataCOW` and `TestData::checkWrittenDataMOR` were never affected. Both already iterated over all latest base files / all latest merged file slices per partition when this ticket was opened, so no change was needed there.
### What fixed it
Commit `8a653836db55c1490cd3e3964d22f3e7abc780fe` (#19118, "feat(writer): Switch default to native log format for table version >= 10") replaced the single-file selection with the file-slice loop. It was a side effect of the log-format work rather than a deliberate fix here, which is why no commit references HUDI-9056.
Multi-bucket coverage also landed independently: `ITTestBucketStreamWrite::testBucketWriteIntoMultipleBuckets` (`bucketNum = 2`, both COW and MOR) was added in `f5de6628b88c927ef2784d916014891c2feaa468` (#12796, HUDI-8920), five days after this ticket was created.
I also swept `TestData` for any remaining single-latest-file selection and found none.
### One small leftover, not worth keeping this open
`ITTestBucketStreamWrite::testBucketStreamWriteAfterRollbackFirstFileGroupCreation` still calls `doWrite(tablePath, isCow, 1)`. The reason that mattered is gone, since `checkWrittenData` now aggregates across base files, so raising the bucket count there is safe and would give the rollback path genuine multi-bucket coverage. That is a good-first-issue-sized follow-up rather than remaining scope on this task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Revisit TestData methods [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #16860:
URL: https://github.com/apache/hudi/issues/16860#issuecomment-5391420424
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9056).
**Findings: still open.**
No commit referencing HUDI-9056 exists in the tree, so nothing has landed under this ticket's key.
The description here is substantive and self-describing, so this is workable as written - it is the absence of an owner rather than of a specification.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Revisit TestData methods [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy closed issue #16860: Revisit TestData methods
URL: https://github.com/apache/hudi/issues/16860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org