You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Hive QA (JIRA)" <ji...@apache.org> on 2016/01/14 10:00:47 UTC
[jira] [Commented] (HIVE-12847) ORC file footer cache should be
memory sensitive
[ https://issues.apache.org/jira/browse/HIVE-12847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097852#comment-15097852 ]
Hive QA commented on HIVE-12847:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12782173/HIVE-12847.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10016 tests executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testMultiSessionMultipleUse
org.apache.hadoop.hive.ql.exec.spark.session.TestSparkSessionManagerImpl.testSingleSessionMultipleUse
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6620/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6620/console
Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6620/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12782173 - PreCommit-HIVE-TRUNK-Build
> ORC file footer cache should be memory sensitive
> ------------------------------------------------
>
> Key: HIVE-12847
> URL: https://issues.apache.org/jira/browse/HIVE-12847
> Project: Hive
> Issue Type: Improvement
> Components: File Formats, ORC
> Affects Versions: 1.2.1
> Reporter: Nemon Lou
> Assignee: Nemon Lou
> Attachments: HIVE-12847.patch
>
>
> The size based footer cache can not control memory usage properly.
> Having seen a HiveServer2 hang due to ORC file footer cache taking up too much heap memory.
> A simple query like "select * from orc_table limit 1" can make HiveServer2 hang.
> The input table has about 1000 ORC files and each ORC file owns about 2500 stripes.
> {noformat}
> num #instances #bytes class name
> ----------------------------------------------
> 1: 214653601 25758432120 org.apache.hadoop.hive.ql.io.orc.OrcProto$ColumnStatistics
> 3: 122233301 8800797672 org.apache.hadoop.hive.ql.io.orc.OrcProto$StringStatistics
> 5: 89439001 6439608072 org.apache.hadoop.hive.ql.io.orc.OrcProto$IntegerStatistics
> 7: 2981300 262354400 org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeInformation
> 9: 2981300 143102400 org.apache.hadoop.hive.ql.io.orc.OrcProto$StripeStatistics
> 12: 2983691 71608584 org.apache.hadoop.hive.ql.io.orc.ReaderImpl$StripeInformationImpl
> 15: 80929 7121752 org.apache.hadoop.hive.ql.io.orc.OrcProto$Type
> 17: 103282 5783792 org.apache.hadoop.mapreduce.lib.input.FileSplit
> 20: 51641 3305024 org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit
> 21: 51641 3305024 org.apache.hadoop.hive.ql.io.orc.OrcSplit
> 31: 1 413152 [Lorg.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit;
> 100: 1122 26928 org.apache.hadoop.hive.ql.io.orc.Metadata
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)