You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2026/04/13 19:44:02 UTC

[I] bug: HDFS `get_ranges` performance 2x degradation [opendal]

comphead opened a new issue, #7383:
URL: https://github.com/apache/opendal/issues/7383

   ### Describe the bug
   
   In https://github.com/apache/datafusion-comet/issues/3926 we found that scan tasks performance degraded significantly, presumably after #7192 PR 
   
   HDFS scan task took ~2mins on testing pipeline, now it takes 4-5mins. I also checked artifact with default `get_ranges` and no perf degradation in this case. 
   
   @Xuanwo @kszucs are you guys considering to rollback or improve the `get_ranges`? 
   
   Some extra context: Played with implementation in #7380, specifically with reading ranges concurrently, however another problem came up with huge memory utilization
   
   ### Steps to Reproduce
   
   Personally I reproduced this on cluster, but even on local HDFS cluster the problem can be reproduced.
   
   ### Expected Behavior
   
   Performance should not be affected 
   
   ### Additional Context
   
   _No response_
   
   ### Are you willing to submit a PR to fix this bug?
   
   - [ ] Yes, I would like to submit a PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] bug: HDFS `get_ranges` performance 2x degradation [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #7383: bug: HDFS `get_ranges` performance 2x degradation
URL: https://github.com/apache/opendal/issues/7383


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org