You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "dima machlin (JIRA)" <ji...@apache.org> on 2014/02/17 14:22:22 UTC
[jira] [Updated] (HIVE-6448) Java heap space pasring a query with
many union all
[ https://issues.apache.org/jira/browse/HIVE-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dima machlin updated HIVE-6448:
-------------------------------
Description:
too many union all statements in a single query require massive amount of memory to process.
Under default HiveServer Xmx (256) we can run ~50 union alls.
It seems to be rather linear as increasing the Xmx to 1024 we can run ~200.
The error is :
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453)
at org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598)
at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609)
at org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
Query example :
select *
from (
select count(*) from default.dual union all
select count(*) from default.dual union all
......
select count(*) from default.dual) z
was:
too many union all statements in a single query require massive amount of memory to process.
Under default HiveServer Xmx (256) we can run ~50 union alls.
It seems to be rather linear as increasing the Xmx to 1024 we can run ~200.
The error is :
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453)
at org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598)
at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609)
at org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
> Java heap space pasring a query with many union all
> ---------------------------------------------------
>
> Key: HIVE-6448
> URL: https://issues.apache.org/jira/browse/HIVE-6448
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Processor
> Affects Versions: 0.11.0
> Environment:
> Reporter: dima machlin
>
> too many union all statements in a single query require massive amount of memory to process.
> Under default HiveServer Xmx (256) we can run ~50 union alls.
> It seems to be rather linear as increasing the Xmx to 1024 we can run ~200.
> The error is :
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453)
> at org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598)
> at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609)
> at org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
> Query example :
> select *
> from (
> select count(*) from default.dual union all
> select count(*) from default.dual union all
> ......
> select count(*) from default.dual) z
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)