You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Sergey Shelukhin (JIRA)" <ji...@apache.org> on 2016/11/10 22:18:59 UTC
[jira] [Assigned] (HIVE-15178) ORC stripe merge may produce many MR
jobs and no merge if split size is small
[ https://issues.apache.org/jira/browse/HIVE-15178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sergey Shelukhin reassigned HIVE-15178:
---------------------------------------
Assignee: Sergey Shelukhin
> ORC stripe merge may produce many MR jobs and no merge if split size is small
> -----------------------------------------------------------------------------
>
> Key: HIVE-15178
> URL: https://issues.apache.org/jira/browse/HIVE-15178
> Project: Hive
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Assignee: Sergey Shelukhin
>
> orc_createas1
> logs the following:
> {noformat}
> 2016-11-10T13:38:54,366 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing split: Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2400+100InputFormatClass: org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> 2016-11-10T13:38:54,373 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing split: Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2500+100InputFormatClass: org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> 2016-11-10T13:38:54,380 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing split: Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2600+100InputFormatClass: org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> 2016-11-10T13:38:54,387 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing split: Paths:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0:2700+100InputFormatClass: org.apache.hadoop.hive.ql.io.orc.OrcFileStripeMergeInputFormat
> ...
> {noformat}
> It tries to merge 2 files, but instead ends up running tons of MR tasks for every 100 bytes and produces 2 files again (I assume most tasks don't produce the files because the stripes are invalid).
> {noformat}
> 2016-11-10T13:38:53,985 INFO [LocalJobRunner Map Task Executor #0] OrcFileMergeOperator: Merged stripe from file pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000000_0 [ offset : 3 length: 2770 row: 500 ]
> 2016-11-10T13:38:53,995 INFO [LocalJobRunner Map Task Executor #0] exec.AbstractFileMergeOperator: renamed path pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_task_tmp.-ext-10002/_tmp.000002_0 to pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_tmp.-ext-10002/000002_0 . File size is 2986
> 2016-11-10T13:38:54,206 INFO [LocalJobRunner Map Task Executor #0] OrcFileMergeOperator: Merged stripe from file pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/-ext-10004/000001_0 [ offset : 3 length: 2770 row: 500 ]
> 2016-11-10T13:38:54,215 INFO [LocalJobRunner Map Task Executor #0] exec.AbstractFileMergeOperator: renamed path pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_task_tmp.-ext-10002/_tmp.000030_0 to pfile:/Users/sergey/git/hivegit2/itests/qtest/target/warehouse/.hive-staging_hive_2016-11-10_13-38-52_334_1323113125332102866-1/_tmp.-ext-10002/000030_0 . File size is 2986
> {noformat}
> This is because the test sets the max split size to 100. Merge jobs is supposed to override that, but that doesn't happen somehow.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)