[GitHub] [hive] pgaref edited a comment on pull request #1547: HIVE-24225: Revert HIVE-23158 Optimize S3A recordReader policy for Random IO formats

[GitBox <gi...@apache.org>]

pgaref edited a comment on pull request #1547:
URL: https://github.com/apache/hive/pull/1547#issuecomment-704856290


   Hey @steveloughran  --- the approach of the above patch was a bit off, one problem was that the Fs objects were lazility initalized and could end up throwing exceptions when setting the option eagerly.
   The most important issue was that the LLAP IO creates its own FS object (and the above where only used for output) so the option itself was not properly propagated.
   
   A solution for all this could be the S3A **openFileWithOptions**  call that adds file options for the open File call instead of on the FS (still needs to add support for **fadvise**  though)
   https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L4828


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org