You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/07/18 02:14:11 UTC

[GitHub] [hudi] vinothchandar commented on issue #1837: [SUPPORT]S3 file listing causing compaction to get eventually slow

vinothchandar commented on issue #1837:
URL: https://github.com/apache/hudi/issues/1837#issuecomment-660408124


   > This means that we need to move out these listing logic our of hudi-common if we want to parallelize it with spark context.
   I will be landing a PR over the weekend, that avoids listings for rollbacks.. consequently, I moved the place you are changing into hudi-client already.. So it should be simple to redo on top of that. 
   
   Overall, We already have a `StorageSchemes` class that does different things for S3/GCS etc and HDFS/Ignite.. based on append support.. As a more elegant fix, I feel if we can take a pass at listing usages and do different forms of listing based on storage schemes.. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org