You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by "nitinpathakala ." <ni...@gmail.com> on 2015/04/23 18:08:21 UTC

Fwd: java.lang.NullPointerException when using INSERT INTO ORC table with ACID property

Hi ,

We have a scenario to update a table from different set of tables for which
we are using hive 0.14.Here are the steps that we are following

1.Parent table is ORC table with ACID property.
2.Delta table has updated records that need to be inserted to parent
table(Delta table is also an ORC with ACID property).
3.Insert into table parent select * from delta table.

Throws below error :

java.lang.NullPointerException
        at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:265)
        at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getCombineSplits(CombineHiveInputFormat.java:272)
        at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:509)
        at
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
        at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
        at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
        at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
        at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:429)
        at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
        at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
        at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
        at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
        at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception 'java.lang.NullPointerException(null)'
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:

Could you please suggest.
Please let us know if further details are needed.

Regards,
Nitin

Re: java.lang.NullPointerException when using INSERT INTO ORC table with ACID property

Posted by Eugene Koifman <ek...@hortonworks.com>.

Can you provide precise table definitions (DDL, describe formatted, etc and statements you are running (with explain plan)?  (the simplest version that reproduces the issue)
Also your hive-site.xml.  Would be even better if you file a JIRA with the same.

You are clearly hitting a bug but it's impossible to be sure what the exact issue is based on the info provided.

From: "nitinpathakala ." <ni...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Friday, May 8, 2015 at 3:41 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: java.lang.NullPointerException when using INSERT INTO ORC table with ACID property

Hi Ivneet , you are correct if we set to false it works but we cannot update or delete as it does not conform ACID property.still waiting for some solution.

Regards,
Nitin

On May 8, 2015 4:00 PM, "ivneet" <ka...@gmail.com>> wrote:
Hi Nitin,

I am facing exact same issue on hive 0.14.0 (hortonworks 2.2 cluster).

Please let me know if you have found the solution.

I observed that if hive.enforce.bucketing is set to false in hive-
site.xml, then this issue is not faced, but hive configuration for ACID
requires this property to be set to true.

Thanks,
Ivneet

Re: java.lang.NullPointerException when using INSERT INTO ORC table with ACID property

Posted by "nitinpathakala ." <ni...@gmail.com>.

Hi Ivneet , you are correct if we set to false it works but we cannot
update or delete as it does not conform ACID property.still waiting for
some solution.

Regards,
Nitin
On May 8, 2015 4:00 PM, "ivneet" <ka...@gmail.com> wrote:

> Hi Nitin,
>
> I am facing exact same issue on hive 0.14.0 (hortonworks 2.2 cluster).
>
> Please let me know if you have found the solution.
>
> I observed that if hive.enforce.bucketing is set to false in hive-
> site.xml, then this issue is not faced, but hive configuration for ACID
> requires this property to be set to true.
>
> Thanks,
> Ivneet
>
>
>
>

Re: java.lang.NullPointerException when using INSERT INTO ORC table with ACID property

Posted by ivneet <ka...@gmail.com>.

Hi Nitin,

I am facing exact same issue on hive 0.14.0 (hortonworks 2.2 cluster).

Please let me know if you have found the solution.

I observed that if hive.enforce.bucketing is set to false in hive-
site.xml, then this issue is not faced, but hive configuration for ACID 
requires this property to be set to true.

Thanks,
Ivneet

Re: java.lang.NullPointerException when using INSERT INTO ORC table with ACID property

Posted by "nitinpathakala ." <ni...@gmail.com>.

Hi,

Any one encountered the below error while updating to ORC table with ACID
property.

java.lang.NullPointerException
        at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:265)
        at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getCombineSplits(CombineHiveInputFormat.java:272)
        at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:509)
        at
org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
        at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
        at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
        at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
        at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
        at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:429)
        at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
        at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
        at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
        at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
        at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
        at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Job Submission failed with exception 'java.lang.NullPointerException(null)'
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:

Regards,
Nitin

On Thu, Apr 23, 2015 at 9:38 PM, nitinpathakala . <ni...@gmail.com>
wrote:

> Hi ,
>
> We have a scenario to update a table from different set of tables for
> which we are using hive 0.14.Here are the steps that we are following
>
> 1.Parent table is ORC table with ACID property.
> 2.Delta table has updated records that need to be inserted to parent
> table(Delta table is also an ORC with ACID property).
> 3.Insert into table parent select * from delta table.
>
> Throws below error :
>
> java.lang.NullPointerException
>         at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:265)
>         at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getCombineSplits(CombineHiveInputFormat.java:272)
>         at
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:509)
>         at
> org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
>         at
> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
>         at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
>         at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
>         at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
>         at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
>         at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:429)
>         at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:137)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
>         at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1604)
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1364)
>         at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1177)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:247)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:199)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:410)
>         at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:783)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:483)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Job Submission failed with exception 'java.lang.NullPointerException(null)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched:
>
> Could you please suggest.
> Please let us know if further details are needed.
>
> Regards,
> Nitin
>
>