You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "pvary (via GitHub)" <gi...@apache.org> on 2025/12/03 13:30:51 UTC
[PR] Spark: ORC vectorized reader to use the delete filter [iceberg]
pvary opened a new pull request, #14746:
URL: https://github.com/apache/iceberg/pull/14746
Refactored out from #14652
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: ORC vectorized reader to use the delete filter [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on code in PR #14746:
URL: https://github.com/apache/iceberg/pull/14746#discussion_r2591816073
##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkOrcReadMetadataColumns.java:
##########
@@ -128,13 +135,43 @@ public void writeFile() throws IOException {
@TestTemplate
public void testReadRowNumbers() throws IOException {
- readAndValidate(null, null, null, EXPECTED_ROWS);
+ readAndValidate(null, null, null, EXPECTED_ROWS, NO_DELETES_FILTER);
+ }
+
+ @TestTemplate
+ public void testReadRowNumbersWithDelete() throws IOException {
+ assumeThat(vectorized).isTrue();
+
+ List<InternalRow> expectedRowsAfterDelete = Lists.newArrayList();
+ EXPECTED_ROWS.forEach(row -> expectedRowsAfterDelete.add(row.copy()));
+ // remove row at position 98, 99, 100, 101, 102, this crosses two row groups [0, 100) and [100,
+ // 200)
+ for (int i = 98; i <= 102; i++) {
+ expectedRowsAfterDelete.get(i).update(3, true);
+ }
+
+ ORC.ReadBuilder builder = ORC.read(Files.localInput(testFile)).project(PROJECTION_SCHEMA);
Review Comment:
Good catch!
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: ORC vectorized reader to use the delete filter [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #14746:
URL: https://github.com/apache/iceberg/pull/14746#discussion_r2591023226
##########
spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkOrcReadMetadataColumns.java:
##########
@@ -128,13 +135,43 @@ public void writeFile() throws IOException {
@TestTemplate
public void testReadRowNumbers() throws IOException {
- readAndValidate(null, null, null, EXPECTED_ROWS);
+ readAndValidate(null, null, null, EXPECTED_ROWS, NO_DELETES_FILTER);
+ }
+
+ @TestTemplate
+ public void testReadRowNumbersWithDelete() throws IOException {
+ assumeThat(vectorized).isTrue();
+
+ List<InternalRow> expectedRowsAfterDelete = Lists.newArrayList();
+ EXPECTED_ROWS.forEach(row -> expectedRowsAfterDelete.add(row.copy()));
+ // remove row at position 98, 99, 100, 101, 102, this crosses two row groups [0, 100) and [100,
+ // 200)
+ for (int i = 98; i <= 102; i++) {
+ expectedRowsAfterDelete.get(i).update(3, true);
+ }
+
+ ORC.ReadBuilder builder = ORC.read(Files.localInput(testFile)).project(PROJECTION_SCHEMA);
Review Comment:
This `builder` is not used. `readAndValidate` creates its own `ORC.ReadBuilder`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: ORC vectorized reader to use the delete filter [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on PR #14746:
URL: https://github.com/apache/iceberg/pull/14746#issuecomment-3618056756
Merged to main.
Thanks for the review @huaxingao!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Spark: ORC vectorized reader to use the delete filter [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary merged PR #14746:
URL: https://github.com/apache/iceberg/pull/14746
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org