You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Phabricator (JIRA)" <ji...@apache.org> on 2013/08/28 23:47:52 UTC
[jira] [Updated] (HIVE-5102) ORC getSplits should create splits
based the stripes
[ https://issues.apache.org/jira/browse/HIVE-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-5102:
------------------------------
Attachment: HIVE-5102.D12579.1.patch
omalley requested code review of "HIVE-5102 [jira] ORC getSplits should create splits based the stripes".
Reviewers: JIRA
working on orcinputformat
Currently ORC inherits getSplits from FileFormat, which basically makes a split per an HDFS block. This can create too little parallelism and would be better done by having getSplits look at the file footer and create splits based on the stripes.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D12579
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java
ql/src/java/org/apache/hadoop/hive/ql/io/orc/ReaderImpl.java
ql/src/java/org/apache/hadoop/hive/ql/io/orc/StripeInformation.java
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestInputOutputFormat.java
ql/src/test/org/apache/hadoop/hive/ql/io/orc/TestOrcFile.java
shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java
shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
MANAGE HERALD RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/30219/
To: JIRA, omalley
> ORC getSplits should create splits based the stripes
> -----------------------------------------------------
>
> Key: HIVE-5102
> URL: https://issues.apache.org/jira/browse/HIVE-5102
> Project: Hive
> Issue Type: Bug
> Components: File Formats
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Attachments: HIVE-5102.D12579.1.patch
>
>
> Currently ORC inherits getSplits from FileFormat, which basically makes a split per an HDFS block. This can create too little parallelism and would be better done by having getSplits look at the file footer and create splits based on the stripes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira