You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/07/15 15:02:02 UTC

[GitHub] [hudi] vinothchandar commented on issue #1829: [SUPPORT] S3 slow file listing causes Hudi read performance.

vinothchandar commented on issue #1829:
URL: https://github.com/apache/hudi/issues/1829#issuecomment-658821557


   @zuyanton this seems like a general issue with `FileInputFormat` 
   
   ```
    int numThreads = job
           .getInt(
               org.apache.hadoop.mapreduce.lib.input.FileInputFormat.LIST_STATUS_NUM_THREADS,
               org.apache.hadoop.mapreduce.lib.input.FileInputFormat.DEFAULT_LIST_STATUS_NUM_THREADS);
   ```
   
   can you try adding `mapreduce.input.fileinputformat.list-status.num-threads=8` or something? (default inside Hadoop is 1) 
   
   cc @n3nash IIRC you mentioned a similar approach done at uber?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org