[jira] [Resolved] (TEZ-3499) OOM in local mode

["Hitesh Shah (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TEZ-3499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hitesh Shah resolved TEZ-3499.
------------------------------
    Resolution: Invalid

Questions like this are best resolved by posting to the appropriate user mailing list. Given that this is a hive use-case, you should start by asking on the user@hive mailing list. Based on discussions, there may be a need to file a bug against either Hive or Tez but general questions should not be posted as a new JIRA. 

> OOM in local mode
> -----------------
>
>                 Key: TEZ-3499
>                 URL: https://issues.apache.org/jira/browse/TEZ-3499
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.8.4
>            Reporter: Premal Shah
>
> We have a UDF in hive which take in some values and outputs a score. When running a query on a table which calls the score function on every row, looks like tez is not running the query on YARN, but trying to run it in local mode. It then runs out of memory trying to insert that data into a table.
> Here's the query
> {noformat}
> ADD JAR score.jar;
> CREATE TEMPORARY FUNCTION score AS 'hive.udf.ScoreUDF';
> CREATE TABLE abc AS
> SELECT
>     id,
>     score(col1, col2) as score
>     , '2016-10-11' AS dt
> FROM input_table
> ;
> {noformat}
> Here's the output of the shell
> {noformat}
> Query ID = hadoop_20161028232841_5a06db96-ffaa-4e75-a657-c7cb46ccb3f5
> Total jobs = 1
> Launching Job 1 out of 1
> java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Arrays.java:3332)
>         at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
>         at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
>         at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622)
>         at java.lang.StringBuilder.append(StringBuilder.java:202)
>         at com.google.protobuf.TextFormat.escapeBytes(TextFormat.java:1283)
>         at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:394)
>         at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
>         at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
>         at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
>         at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
>         at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
>         at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
>         at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:283)
>         at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
>         at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:283)
>         at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.printFieldValue(TextFormat.java:404)
>         at com.google.protobuf.TextFormat$Printer.printSingleField(TextFormat.java:327)
>         at com.google.protobuf.TextFormat$Printer.printField(TextFormat.java:286)
>         at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:273)
>         at com.google.protobuf.TextFormat$Printer.access$400(TextFormat.java:248)
>         at com.google.protobuf.TextFormat.shortDebugString(TextFormat.java:88)
> FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Java heap space
> {noformat}
> It looks like the job is not getting submitted to the cluster, but running locally.
> We can't get tez to run the code on the cluster. The hive shell starts with an Xmx of 4G. 
> What should we change to avoid this problem?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)