You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "gaborkaszab (via GitHub)" <gi...@apache.org> on 2026/01/08 13:21:25 UTC
[PR] Core, Data, Spark: Use partition stats scan API in tests [iceberg]
gaborkaszab opened a new pull request, #14996:
URL: https://github.com/apache/iceberg/pull/14996
This is the follow-up PR after #14640 and #14989 to replace the usage of legacy partition stat scan with the new partition Scan API in the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Use partition stats scan API in tests [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on code in PR #14996:
URL: https://github.com/apache/iceberg/pull/14996#discussion_r2685359940
##########
core/src/test/java/org/apache/iceberg/PartitionStatsHandlerTestBase.java:
##########
@@ -182,13 +182,29 @@ public void testAllDatatypePartitionWriting() throws Exception {
PartitionStatistics partitionStats = randomStats(partitionData);
List<PartitionStatistics> expected = Collections.singletonList(partitionStats);
- PartitionStatisticsFile statisticsFile =
- PartitionStatsHandler.writePartitionStatsFile(testTable, 42L, dataSchema, expected);
- List<PartitionStats> written;
- try (CloseableIterable<PartitionStats> recordIterator =
- PartitionStatsHandler.readPartitionStatsFile(
- dataSchema, testTable.io().newInputFile(statisticsFile.path()))) {
+ // Add a dummy file to the table to have a snapshot
+ DataFile dataFile =
+ DataFiles.builder(spec)
+ .withPath("some_path")
+ .withPartition(partitionData)
+ .withFileSizeInBytes(15)
+ .withFormat(FileFormat.PARQUET)
+ .withRecordCount(1)
+ .build();
+ testTable.newAppend().appendFile(dataFile).commit();
+ long snapshotId = testTable.currentSnapshot().snapshotId();
Review Comment:
The old and the new way of reading partition stats have different inputs: The old function accepts a stat file, while the new one accepts a snapshot ID. With this some tests using the old way "cheated" a bit while they created a stat file, and didn't register it with a snapshot to the table, simply passed the file to the read function.
With the new API we can't do this "cheat" because we query by snapshot ID, hence you need a snapshot in the table, so that you can register the stat file with that snapshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Use partition stats scan API in tests [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14996:
URL: https://github.com/apache/iceberg/pull/14996#discussion_r2682752926
##########
core/src/test/java/org/apache/iceberg/PartitionStatsHandlerTestBase.java:
##########
@@ -182,13 +182,29 @@ public void testAllDatatypePartitionWriting() throws Exception {
PartitionStatistics partitionStats = randomStats(partitionData);
List<PartitionStatistics> expected = Collections.singletonList(partitionStats);
- PartitionStatisticsFile statisticsFile =
- PartitionStatsHandler.writePartitionStatsFile(testTable, 42L, dataSchema, expected);
- List<PartitionStats> written;
- try (CloseableIterable<PartitionStats> recordIterator =
- PartitionStatsHandler.readPartitionStatsFile(
- dataSchema, testTable.io().newInputFile(statisticsFile.path()))) {
+ // Add a dummy file to the table to have a snapshot
+ DataFile dataFile =
+ DataFiles.builder(spec)
+ .withPath("some_path")
+ .withPartition(partitionData)
+ .withFileSizeInBytes(15)
+ .withFormat(FileFormat.PARQUET)
+ .withRecordCount(1)
+ .build();
+ testTable.newAppend().appendFile(dataFile).commit();
+ long snapshotId = testTable.currentSnapshot().snapshotId();
Review Comment:
Is this a behavioral change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Use partition stats scan API in tests [iceberg]
Posted by "gaborkaszab (via GitHub)" <gi...@apache.org>.
gaborkaszab commented on PR #14996:
URL: https://github.com/apache/iceberg/pull/14996#issuecomment-3742752040
> Are there any differences between the different Spark versions?
Spark changes are identical across versions, except 3.4 had only 1 test suite out of the affected 3 (there are no tests for rewrite data files and rewrite manifests)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Use partition stats scan API in tests [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #14996:
URL: https://github.com/apache/iceberg/pull/14996#issuecomment-3739137692
Are there any differences between the different Spark versions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Use partition stats scan API in tests [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #14996:
URL: https://github.com/apache/iceberg/pull/14996#issuecomment-3744214967
Merged to main.
Thanks @gaborkaszab for the PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, Data, Spark: Use partition stats scan API in tests [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #14996:
URL: https://github.com/apache/iceberg/pull/14996
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org