You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "parthchandra (via GitHub)" <gi...@apache.org> on 2026/01/27 18:35:08 UTC

[PR] perf: [iceberg] per partition file scan task serialization [datafusion-comet]

parthchandra opened a new pull request, #3301:
URL: https://github.com/apache/datafusion-comet/pull/3301

   Experimental to see if this approach to do per partition file scan task serialization works.
   
   Note: Could not have been done without Claude Code.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: [iceberg] per partition file scan task serialization [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra closed pull request #3301: perf: [iceberg] per partition file scan task serialization
URL: https://github.com/apache/datafusion-comet/pull/3301


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: [iceberg] per partition file scan task serialization [datafusion-comet]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3301:
URL: https://github.com/apache/datafusion-comet/pull/3301#issuecomment-3807818693

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `0%` with `94 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 9.67%. Comparing base ([`f09f8af`](https://app.codecov.io/gh/apache/datafusion-comet/commit/f09f8af64c6599255e116a376f4f008f2fd63b43?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`859d3a8`](https://app.codecov.io/gh/apache/datafusion-comet/commit/859d3a8862b3547d9930b162af4da407afa7026a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 901 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...e/spark/sql/comet/CometIcebergNativeScanExec.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fcomet%2FCometIcebergNativeScanExec.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvY29tZXQvQ29tZXRJY2ViZXJnTmF0aXZlU2NhbkV4ZWMuc2NhbGE=) | 0.00% | [44 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../comet/serde/operator/CometIcebergNativeScan.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Fserde%2Foperator%2FCometIcebergNativeScan.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9zZXJkZS9vcGVyYXRvci9Db21ldEljZWJlcmdOYXRpdmVTY2FuLnNjYWxh) | 0.00% | [22 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...la/org/apache/spark/sql/comet/IcebergScanRDD.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fcomet%2FIcebergScanRDD.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvY29tZXQvSWNlYmVyZ1NjYW5SREQuc2NhbGE=) | 0.00% | [19 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [spark/src/main/scala/org/apache/comet/Native.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2FNative.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9OYXRpdmUuc2NhbGE=) | 0.00% | [5 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...rc/main/java/org/apache/comet/NativeJNIBridge.java](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fcomet%2FNativeJNIBridge.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2NvbWV0L05hdGl2ZUpOSUJyaWRnZS5qYXZh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...a/org/apache/comet/iceberg/IcebergReflection.scala](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&filepath=spark%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fcomet%2Ficeberg%2FIcebergReflection.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c3Bhcmsvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9jb21ldC9pY2ViZXJnL0ljZWJlcmdSZWZsZWN0aW9uLnNjYWxh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##               main   #3301       +/-   ##
   ============================================
   - Coverage     56.12%   9.67%   -46.45%     
   + Complexity      976     234      -742     
   ============================================
     Files           119     177       +58     
     Lines         11743   16242     +4499     
     Branches       2251    2659      +408     
   ============================================
   - Hits           6591    1572     -5019     
   - Misses         4012   14492    +10480     
   + Partials       1140     178      -962     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/datafusion-comet/pull/3301?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] perf: [iceberg] per partition file scan task serialization [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on PR #3301:
URL: https://github.com/apache/datafusion-comet/pull/3301#issuecomment-3819608351

   Closing as this did not address the problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org