You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@hive.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2022/11/12 13:47:00 UTC

[jira] [Commented] (HIVE-26699) Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX

    [ https://issues.apache.org/jira/browse/HIVE-26699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17632676#comment-17632676 ] 

Steve Loughran commented on HIVE-26699:
---------------------------------------

you should be using the openFile() api call and set the read policy option to whole-file (assuming that is the intent), and ideally pass in the file status...or at least file length, which is enough for s3a to skip the HEAD, though not abfs.
see org.apache.hadoop.util.JsonSerialization for its max-performance json load, which the s3a and manifest committers both use

> Iceberg: S3 fadvise can hurt JSON parsing significantly in DWX
> --------------------------------------------------------------
>
>                 Key: HIVE-26699
>                 URL: https://issues.apache.org/jira/browse/HIVE-26699
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Priority: Major
>
> Hive reads JSON metadata information (TableMetadataParser::read()) multiple times; E.g during query compilation, AM split computation, stats computation, during commits  etc.
>  
> With large JSON files (due to multiple inserts), it takes a lot longer time with S3 FS with "fs.s3a.experimental.input.fadvise" set to "random". (e.g in the order of 10x).To be on safer side, it will be good to set this to "normal" mode in configs, when reading iceberg tables.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)