You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "parthchandra (via GitHub)" <gi...@apache.org> on 2025/09/12 16:24:59 UTC
[PR] feat: support deleted rows in Comet [iceberg]
parthchandra opened a new pull request, #14062:
URL: https://github.com/apache/iceberg/pull/14062
This PR requires comet 0.10.0 and also https://github.com/apache/iceberg/pull/13786.
The PR removes the restriction on handling deleted rows in Comet. The basic change is to replace the use of `ColumnVectorWithFilter` with `CometSelectionVector` which provides the same functionality. `CometSelectionVector` represents the values vector and the row mapping id as arrow vectors allowing fast zero-copy transfer via Arrow FFI. All processing is done in Comet native code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14062:
URL: https://github.com/apache/iceberg/pull/14062#issuecomment-3535134860
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "hsiang-c (via GitHub)" <gi...@apache.org>.
hsiang-c commented on code in PR #14062:
URL: https://github.com/apache/iceberg/pull/14062#discussion_r2345077197
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:
##########
@@ -189,6 +189,21 @@ private boolean supportsCometBatchReads(Types.NestedField field) {
&& field.fieldId() != MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
}
+ private boolean supportsCometBatchReads(ScanTask task) {
Review Comment:
Ditto, can reuse `supportsParquetBatchReads` now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #14062:
URL: https://github.com/apache/iceberg/pull/14062#issuecomment-3286777335
Looks good overall. I will also do another pass after Comet 0.10.0 is released. Thanks @parthchandra for working on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #14062:
URL: https://github.com/apache/iceberg/pull/14062#issuecomment-3285963787
@huaxingao @anuragmantri @hsiang-c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "hsiang-c (via GitHub)" <gi...@apache.org>.
hsiang-c commented on code in PR #14062:
URL: https://github.com/apache/iceberg/pull/14062#discussion_r2345076415
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:
##########
@@ -189,6 +189,21 @@ private boolean supportsCometBatchReads(Types.NestedField field) {
&& field.fieldId() != MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
}
+ private boolean supportsCometBatchReads(ScanTask task) {
Review Comment:
nit: I created this method to filter scans w/ deletions at the beginning. With your deletion support, we can remove this method and use the existing `supportsParquetBatchReads` b/c `supportsCometBatchReads` and `supportsParquetBatchReads` are identical now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "hsiang-c (via GitHub)" <gi...@apache.org>.
hsiang-c commented on code in PR #14062:
URL: https://github.com/apache/iceberg/pull/14062#discussion_r2345423437
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:
##########
@@ -189,6 +189,21 @@ private boolean supportsCometBatchReads(Types.NestedField field) {
&& field.fieldId() != MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
}
+ private boolean supportsCometBatchReads(ScanTask task) {
Review Comment:
@huaxingao Sorry about that. I added this in a different draft PR to fallback to Spark when delete files are present. It is no longer needed with Parth's work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on code in PR #14062:
URL: https://github.com/apache/iceberg/pull/14062#discussion_r2345397470
##########
spark/v3.4/spark/src/main/java/org/apache/iceberg/spark/source/SparkBatch.java:
##########
@@ -189,6 +189,21 @@ private boolean supportsCometBatchReads(Types.NestedField field) {
&& field.fieldId() != MetadataColumns.LAST_UPDATED_SEQUENCE_NUMBER.fieldId();
}
+ private boolean supportsCometBatchReads(ScanTask task) {
Review Comment:
I am a bit confused with this method too. I think @hsiang-c is right: we should use `supportsParquetBatchReads`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #14062:
URL: https://github.com/apache/iceberg/pull/14062#issuecomment-3286761041
@hsiang-c I will revise this once Comet 0.10.0 is released and your changes are merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #14062:
URL: https://github.com/apache/iceberg/pull/14062#issuecomment-3286875600
Thanks @huaxingao. For reference, this is consumed on the Comet side here: https://github.com/apache/datafusion-comet/commit/bb318a51ad355bdb13ae42336f100ff6c95b0cdb#diff-00a84cd42c8ea9cf336eedc24279f079148064522404461b3d4c88d9b0e36259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #14062: feat: support deleted rows in Comet
URL: https://github.com/apache/iceberg/pull/14062
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: support deleted rows in Comet [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14062:
URL: https://github.com/apache/iceberg/pull/14062#issuecomment-3505454145
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org