You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/04/08 18:44:00 UTC
[jira] [Work logged] (HIVE-23158) Optimize S3A recordReader policy
for Random IO formats
[ https://issues.apache.org/jira/browse/HIVE-23158?focusedWorklogId=418791&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-418791 ]
ASF GitHub Bot logged work on HIVE-23158:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 08/Apr/20 18:43
Start Date: 08/Apr/20 18:43
Worklog Time Spent: 10m
Work Description: pgaref commented on pull request #972: HIVE-23158 initial patch
URL: https://github.com/apache/hive/pull/972
Change-Id: I156397ce2a64485f125c0e6923da4a8012a8d53c
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 418791)
Remaining Estimate: 0h
Time Spent: 10m
> Optimize S3A recordReader policy for Random IO formats
> ------------------------------------------------------
>
> Key: HIVE-23158
> URL: https://issues.apache.org/jira/browse/HIVE-23158
> Project: Hive
> Issue Type: Bug
> Reporter: Panagiotis Garefalakis
> Assignee: Panagiotis Garefalakis
> Priority: Trivial
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> S3A filesystem client (inherited by Hadoop) supports the notion of input policies.
> These policies tune the behaviour of HTTP requests that are used for reading different filetypes such as TEXT or ORC.
> For formats such as ORC and Parquet do a lot of seek operations, thus there is an optimized RANDOM mode that reads files only partially instead of fully (default).
> I am suggesting to add some extra logic as part of HiveInputFormat to make sure we optimize for random IO when data is stored on S3A using formats such as ORC or Parquet.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)