You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Premal Shah <pr...@gmail.com> on 2016/10/29 04:39:58 UTC
Hive/Tez local mode running out of memory
Hive 2.0.1
Hadoop 2.7.2
Tex 0.8.4
We have a UDF in hive which take in some values and outputs a score. When
running a query on a table which calls the score function on every row,
looks like tez is not running the query on YARN, but trying to run it in
local mode. It then runs out of memory trying to insert that data into a
table.
Here's the query
*ADD JAR score.jar;*
*CREATE TEMPORARY FUNCTION score AS 'hive.udf.ScoreUDF';*
*CREATE TABLE abc AS*
*SELECT*
* id,*
* score(col1, col2) as score*
* , '2016-10-11' AS dt*
*FROM input_table*
*;*
Here's the output of the shell
Query ID = hadoop_20161028232841_5a06db96-ffaa-4e75-a657-c7cb46ccb3f5
Total jobs = 1
Launching Job 1 out of 1
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622)
at java.lang.StringBuilder.append(StringBuilder.java:202)
at com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1283)
at
com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:394)
at
com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at
com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at
com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
at
com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at
com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at
com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
at
com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at
com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at
com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
at
com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at
com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:283)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at
com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
at
com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at
com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:283)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at
com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
at
com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at
com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at
com.google.protobuf.TextFormat$Printer.access$400(TextFormat.java:248)
at
com.google.protobuf.TextFormat.shortDebugString(TextFormat.java:88)
FAILED: Execution Error, return code -101 from
org.apache.hadoop.hive.ql.exec.tez.TezTask. Java heap space
It looks like the job is not getting submitted to the cluster, but running
locally. We can't get tez to run the query on the cluster.
The hive shell starts with an Xmx of 4G.
If I set hive.execution.engine = mr, then the query works, because it runs
on the hadoop cluster.
What should we change to avoid this problem?
Thanx
--
Regards,
Premal Shah.
Re: Hive/Tez local mode running out of memory
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
Hi Premal
This is the problematic line
https://gist.github.com/premal/d054d4cc0ed00efdf60351ca2517db3d#file-hive-log4j2-properties-L23
Can you trying changing ALL to INFO or WARN and see if that helps?
Thanks
Prasanth
On Nov 2, 2016, at 8:23 PM, Premal Shah <pr...@gmail.com>> wrote:
Sorry,
here's the link - https://gist.github.com/premal/d054d4cc0ed00efdf60351ca2517db3d
On Wed, Nov 2, 2016 at 8:11 PM, Premal Shah <pr...@gmail.com>> wrote:
Hi Prasanth,
Here's a link to the hive log4js properties file.
We are on Hive 2.0.1.
We cannot use 2.1.0 because MSCK REPAIR TABLE is broken on it.
Thanx.
On Wed, Nov 2, 2016 at 5:44 PM, Prasanth Jayachandran <pj...@hortonworks.com>> wrote:
This looks related to https://issues.apache.org/jira/browse/HIVE-11751
The debug strings gets generated but gets filtered. Can you share your log4j2 properties file?
What version of hive are you using?
Thanks
Prasanth
On Nov 2, 2016, at 5:06 PM, Premal Shah <pr...@gmail.com>> wrote:
Hi Gopal,
The first thing I did was to look in the logs. Could not find anything even after setting the log level to DEBUG.
Any ideas will be appreciated.
On Tue, Nov 1, 2016 at 3:34 PM, Gopal Vijayaraghavan <go...@apache.org>> wrote:
> I am not sure what is going on here.
You can check the /tmp/$USER/hive.log and see what's happening in detail.
Cheers,
Gopal
--
Regards,
Premal Shah.
--
Regards,
Premal Shah.
--
Regards,
Premal Shah.
Re: Hive/Tez local mode running out of memory
Posted by Premal Shah <pr...@gmail.com>.
Sorry,
here's the link -
https://gist.github.com/premal/d054d4cc0ed00efdf60351ca2517db3d
On Wed, Nov 2, 2016 at 8:11 PM, Premal Shah <pr...@gmail.com> wrote:
> Hi Prasanth,
> Here's a link to the hive log4js properties file.
>
> We are on Hive 2.0.1.
>
> We cannot use 2.1.0 because MSCK REPAIR TABLE is broken on it.
>
> Thanx.
>
>
> On Wed, Nov 2, 2016 at 5:44 PM, Prasanth Jayachandran <
> pjayachandran@hortonworks.com> wrote:
>
>> This looks related to https://issues.apache.org/jira/browse/HIVE-11751
>>
>> The debug strings gets generated but gets filtered. Can you share your
>> log4j2 properties file?
>>
>> What version of hive are you using?
>>
>> Thanks
>> Prasanth
>>
>> On Nov 2, 2016, at 5:06 PM, Premal Shah <pr...@gmail.com> wrote:
>>
>> Hi Gopal,
>> The first thing I did was to look in the logs. Could not find anything
>> even after setting the log level to DEBUG.
>> Any ideas will be appreciated.
>>
>>
>>
>> On Tue, Nov 1, 2016 at 3:34 PM, Gopal Vijayaraghavan <go...@apache.org>
>> wrote:
>>
>>> > I am not sure what is going on here.
>>>
>>> You can check the /tmp/$USER/hive.log and see what's happening in detail.
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Gopal
>>>
>>
>>
>>
>> --
>> Regards,
>> Premal Shah.
>>
>>
>>
>
>
> --
> Regards,
> Premal Shah.
>
--
Regards,
Premal Shah.
Re: Hive/Tez local mode running out of memory
Posted by Premal Shah <pr...@gmail.com>.
Hi Prasanth,
Here's a link to the hive log4js properties file.
We are on Hive 2.0.1.
We cannot use 2.1.0 because MSCK REPAIR TABLE is broken on it.
Thanx.
On Wed, Nov 2, 2016 at 5:44 PM, Prasanth Jayachandran <
pjayachandran@hortonworks.com> wrote:
> This looks related to https://issues.apache.org/jira/browse/HIVE-11751
>
> The debug strings gets generated but gets filtered. Can you share your
> log4j2 properties file?
>
> What version of hive are you using?
>
> Thanks
> Prasanth
>
> On Nov 2, 2016, at 5:06 PM, Premal Shah <pr...@gmail.com> wrote:
>
> Hi Gopal,
> The first thing I did was to look in the logs. Could not find anything
> even after setting the log level to DEBUG.
> Any ideas will be appreciated.
>
>
>
> On Tue, Nov 1, 2016 at 3:34 PM, Gopal Vijayaraghavan <go...@apache.org>
> wrote:
>
>> > I am not sure what is going on here.
>>
>> You can check the /tmp/$USER/hive.log and see what's happening in detail.
>>
>>
>>
>> Cheers,
>>
>> Gopal
>>
>
>
>
> --
> Regards,
> Premal Shah.
>
>
>
--
Regards,
Premal Shah.
Re: Hive/Tez local mode running out of memory
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
This looks related to https://issues.apache.org/jira/browse/HIVE-11751
The debug strings gets generated but gets filtered. Can you share your log4j2 properties file?
What version of hive are you using?
Thanks
Prasanth
On Nov 2, 2016, at 5:06 PM, Premal Shah <pr...@gmail.com>> wrote:
Hi Gopal,
The first thing I did was to look in the logs. Could not find anything even after setting the log level to DEBUG.
Any ideas will be appreciated.
On Tue, Nov 1, 2016 at 3:34 PM, Gopal Vijayaraghavan <go...@apache.org>> wrote:
> I am not sure what is going on here.
You can check the /tmp/$USER/hive.log and see what's happening in detail.
Cheers,
Gopal
--
Regards,
Premal Shah.
Re: Hive/Tez local mode running out of memory
Posted by Premal Shah <pr...@gmail.com>.
Hi Gopal,
The first thing I did was to look in the logs. Could not find anything even
after setting the log level to DEBUG.
Any ideas will be appreciated.
On Tue, Nov 1, 2016 at 3:34 PM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
> > I am not sure what is going on here.
>
> You can check the /tmp/$USER/hive.log and see what's happening in detail.
>
>
>
> Cheers,
>
> Gopal
>
--
Regards,
Premal Shah.
Re: Hive/Tez local mode running out of memory
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> I am not sure what is going on here.
You can check the /tmp/$USER/hive.log and see what's happening in detail.
Cheers,
Gopal
Re: Hive/Tez local mode running out of memory
Posted by Premal Shah <pr...@gmail.com>.
Hi Gopal,
Thanx for chiming in.
This is running on the cluster, but it feels like it's not submitting the
job to the cluster, instead using the local mode (not to be confused with
the local mode for testing). However, I don't think Tez has a local mode
like Hive, so I am not sure what is going on here.
We are on hadoop 2.7.2, hive 2.0.1 and tez 0.8.4
We are migrating from hadoop1, so we just went with all the latest versions
in the test environment, except hive because they broke some stuff in
2.1.0.
On Tue, Nov 1, 2016 at 11:16 AM, Gopal Vijayaraghavan <go...@apache.org>
wrote:
>
>
> > Thanx for the reply. We don't override the log level. According to the
> docs, looks like the default level is INFO.
> > Any other ideas?
>
> That at a first glance looks like a broken install. A good approach would
> be to use a Tez cluster install instead of messing with a local mode runner
> (that's used by the QE tests and defaults to DEBUG).
>
> If you are stuck on a Hadoop distro which does not have Tez & you need it,
> you can get it to work with something like the following
>
> https://gist.github.com/epiphani/dd37e87acfb2f8c4cbb0
>
> If you are more adventurous and want to run LLAP on an unsupported
> platform, I maintain scripts which will configure and install it
>
> https://github.com/t3rmin4t0r/tez-autobuild/blob/llap/README.md
>
> Cheers,
> Gopal
>
>
>
--
Regards,
Premal Shah.
Re: Hive/Tez local mode running out of memory
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> Thanx for the reply. We don't override the log level. According to the docs, looks like the default level is INFO.
> Any other ideas?
That at a first glance looks like a broken install. A good approach would be to use a Tez cluster install instead of messing with a local mode runner (that's used by the QE tests and defaults to DEBUG).
If you are stuck on a Hadoop distro which does not have Tez & you need it, you can get it to work with something like the following
https://gist.github.com/epiphani/dd37e87acfb2f8c4cbb0
If you are more adventurous and want to run LLAP on an unsupported platform, I maintain scripts which will configure and install it
https://github.com/t3rmin4t0r/tez-autobuild/blob/llap/README.md
Cheers,
Gopal
Re: Hive/Tez local mode running out of memory
Posted by Premal Shah <pr...@gmail.com>.
Hi Prasanth,
Thanx for the reply. We don't override the log level. According to the
docs, looks like the default level is INFO.
Any other ideas?
On Sat, Oct 29, 2016 at 3:25 PM, Prasanth Jayachandran <
pjayachandran@hortonworks.com> wrote:
> What is the value of hive.tez.log.level? My guess is this happens only in
> DEBUG log level.
>
> Thanks
> Prasanth
>
>
>
>
> On Fri, Oct 28, 2016 at 9:40 PM -0700, "Premal Shah" <
> premal.j.shah@gmail.com> wrote:
>
> Hive 2.0.1
> Hadoop 2.7.2
> Tex 0.8.4
>
> We have a UDF in hive which take in some values and outputs a score. When
> running a query on a table which calls the score function on every row,
> looks like tez is not running the query on YARN, but trying to run it in
> local mode. It then runs out of memory trying to insert that data into a
> table.
>
> Here's the query
>
>
> *ADD JAR score.jar; *
> *CREATE TEMPORARY FUNCTION score AS 'hive.udf.ScoreUDF';*
>
> *CREATE TABLE abc AS*
> *SELECT*
> * id,*
> * score(col1, col2) as score*
> * , '2016-10-11' AS dt*
> *FROM input_table*
> *;*
>
> Here's the output of the shell
>
> Query ID = hadoop_20161028232841_5a06db96-ffaa-4e75-a657-c7cb46ccb3f5
> Total jobs = 1
> Launching Job 1 out of 1
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3332)
> at java.lang.AbstractStringBuilder.expandCapacity(
> AbstractStringBuilder.java:137)
> at java.lang.AbstractStringBuilder.ensureCapacityInternal(
> AbstractStringBuilder.java:121)
> at java.lang.AbstractStringBuilder.append(
> AbstractStringBuilder.java:622)
> at java.lang.StringBuilder.append(StringBuilder.java:202)
> at com.google.protobuf.TextFormat.escapeBytes(
> TextFormat.java:1283)
> at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:394)
> at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
> at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:286)
> at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
> at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:404)
> at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
> at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:286)
> at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
> at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:404)
> at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
> at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:286)
> at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
> at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:404)
> at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
> at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:283)
> at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
> at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:404)
> at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
> at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:283)
> at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
> at com.google.protobuf.TextFormat$Printer.
> printFieldValue(TextFormat.java:404)
> at com.google.protobuf.TextFormat$Printer.
> printSingleField(TextFormat.java:327)
> at com.google.protobuf.TextFormat$Printer.printField(
> TextFormat.java:286)
> at com.google.protobuf.TextFormat$Printer.print(
> TextFormat.java:273)
> at com.google.protobuf.TextFormat$Printer.access$400(
> TextFormat.java:248)
> at com.google.protobuf.TextFormat.shortDebugString(
> TextFormat.java:88)
> FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask.
> Java heap space
>
>
> It looks like the job is not getting submitted to the cluster, but running
> locally. We can't get tez to run the query on the cluster.
> The hive shell starts with an Xmx of 4G.
>
> If I set hive.execution.engine = mr, then the query works, because it runs
> on the hadoop cluster.
>
> What should we change to avoid this problem?
>
> Thanx
>
> --
> Regards,
> Premal Shah.
>
--
Regards,
Premal Shah.
Re: Hive/Tez local mode running out of memory
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
What is the value of hive.tez.log.level? My guess is this happens only in DEBUG log level.
Thanks
Prasanth
On Fri, Oct 28, 2016 at 9:40 PM -0700, "Premal Shah" <pr...@gmail.com>> wrote:
Hive 2.0.1
Hadoop 2.7.2
Tex 0.8.4
We have a UDF in hive which take in some values and outputs a score. When running a query on a table which calls the score function on every row, looks like tez is not running the query on YARN, but trying to run it in local mode. It then runs out of memory trying to insert that data into a table.
Here's the query
ADD JAR score.jar;
CREATE TEMPORARY FUNCTION score AS 'hive.udf.ScoreUDF';
CREATE TABLE abc AS
SELECT
id,
score(col1, col2) as score
, '2016-10-11' AS dt
FROM input_table
;
Here's the output of the shell
Query ID = hadoop_20161028232841_5a06db96-ffaa-4e75-a657-c7cb46ccb3f5
Total jobs = 1
Launching Job 1 out of 1
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622)
at java.lang.StringBuilder.append(StringBuilder.java:202)
at com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1283)
at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:394)
at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:283)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:283)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
at com.google.protobuf.TextFormat$Printer.access$400(TextFormat.java:248)
at com.google.protobuf.TextFormat.shortDebugString(TextFormat.java:88)
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Java heap space
It looks like the job is not getting submitted to the cluster, but running locally. We can't get tez to run the query on the cluster.
The hive shell starts with an Xmx of 4G.
If I set hive.execution.engine = mr, then the query works, because it runs on the hadoop cluster.
What should we change to avoid this problem?
Thanx
--
Regards,
Premal Shah.