You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Sarath <sa...@algofusiontech.com> on 2012/09/25 12:17:32 UTC

Hive query failing

Hi,

When I run the query "select count(1) from table1;" it fails with the 
exception message as below.

/java.io.FileNotFoundException: File does not exist: 
/usr/lib/hive/lib/hive-builtins-0.8.1-cdh4.0.1.jar
     at 
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:736)
     at 
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
     at 
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
     at 
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:245)
     at 
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:283)
     at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:354)
     at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
     at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:396)
     at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
     at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:609)
     at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:604)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:396)
     at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
     at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:604)
     at 
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:452)
     at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:710)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Job Submission failed with exception 'java.io.FileNotFoundException(File 
does not exist: /usr/lib/hive/lib/hive-builtins-0.8.1-cdh4.0.1.jar)'
Execution failed with exit status: 2/

But the JAR mentioned in the message exists and appropriate R/W 
permissions have been set on the folder /usr/lib/hive for the user.
What is going wrong?

Regards,
Sarath.

Re: Hive query failing

Posted by Sarath <sa...@algofusiontech.com>.
@Kulkarni,

As suggested copied the JAR into the specified directory, now it is 
complaining as -
/java.io.FileNotFoundException: File does not exist: 
/tmp/hduser/hive_2012-09-27_11-24-32_966_3932937489091919630/-mr-10000/1/emptyFile/
where hduser is the unix login ID from where i'm executing the hive 
shell and /tmp/hduser folder is present both on local and hdfs file 
systems with write permissions to hduser.

@Manish
The data is present on the HDFS file system as file and I have issued 
the below query to create the table -
/create external table table1 (<field definitions>) row format delimited 
fields terminated by ',' location '/user/hduser/dumps/table_data.csv';/
Hope it is correct way.


On Tuesday 25 September 2012 05:39 PM, kulkarni.swarnim@gmail.com wrote:
> The jar is being looked on HDFS as the exception suggests. Run the 
> following commands:
>
> $ hadoop fs -mkdir //usr/lib/hive/lib/
> /$ hadoop fs -put $HIVE_HOME/lib/hive-builtins-0.8.1-cdh4.0.1.jar 
> ///usr/lib/hive/lib/
>
> Your queries should work now.
>
> On Sep 25, 2012, at 6:46 AM, Manish.Bhoge <Manish.Bhoge@target.com 
> <ma...@target.com>> wrote:
>
>> Sarath,
>>
>> Is this the external table where you have ran the query? How did you 
>> loaded the table?  Because it looks like the error is about the file 
>> related to table than CDH Jar.
>>
>> Thank You,
>>
>> Manish
>>
>> *From:*Sarath [mailto:sarathchandra.josyam@algofusiontech.com]
>> *Sent:* Tuesday, September 25, 2012 3:48 PM
>> *To:* user@hive.apache.org <ma...@hive.apache.org>
>> *Subject:* Hive query failing
>>
>> Hi,
>>
>> When I run the query "select count(1) from table1;" it fails with the 
>> exception message as below.
>>
>> /java.io.FileNotFoundException: File does not exist: 
>> /usr/lib/hive/lib/hive-builtins-0.8.1-cdh4.0.1.jar
>>     at 
>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:736)
>>     at 
>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
>>     at 
>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
>>     at 
>> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:245)
>>     at 
>> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:283)
>>     at 
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:354)
>>     at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
>>     at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>>     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
>>     at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:609)
>>     at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:604)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>>     at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:604)
>>     at 
>> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:452)
>>     at 
>> org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:710)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>> Job Submission failed with exception 
>> 'java.io.FileNotFoundException(File does not exist: 
>> /usr/lib/hive/lib/hive-builtins-0.8.1-cdh4.0.1.jar)'
>> Execution failed with exit status: 2/
>>
>> But the JAR mentioned in the message exists and appropriate R/W 
>> permissions have been set on the folder /usr/lib/hive for the user.
>> What is going wrong?
>>
>> Regards,
>> Sarath.
>>

Re: Hive query failing

Posted by Sarath <sa...@algofusiontech.com>.
Somehow (by resetting folder permissions) I got rid of the below error.
But now I'm getting a new error as below. Looks like I'm missing some 
configuration, but not sure what and where.

/hive> select count(1) from table1;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
   set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
   set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
   set mapred.reduce.tasks=<number>
java.io.IOException: Cannot initialize Cluster. Please check your 
configuration for mapreduce.framework.name and the correspond server 
addresses.
     at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:121)
     at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:83)
     at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:76)
     at org.apache.hadoop.mapred.JobClient.init(JobClient.java:487)
     at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:466)
     at 
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
     at 
org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
     at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133)
     at 
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
     at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332)
     at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123)
     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931)
     at 
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255)
     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212)
     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
     at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671)
     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
     at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Job Submission failed with exception 'java.io.IOException(Cannot 
initialize Cluster. Please check your configuration for 
mapreduce.framework.name and the correspond server addresses.)'
FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.MapRedTask/

On Tuesday 25 September 2012 05:39 PM, kulkarni.swarnim@gmail.com wrote:
> The jar is being looked on HDFS as the exception suggests. Run the 
> following commands:
>
> $ hadoop fs -mkdir //usr/lib/hive/lib/
> /$ hadoop fs -put $HIVE_HOME/lib/hive-builtins-0.8.1-cdh4.0.1.jar 
> ///usr/lib/hive/lib/
>
> Your queries should work now.
>
> On Sep 25, 2012, at 6:46 AM, Manish.Bhoge <Manish.Bhoge@target.com 
> <ma...@target.com>> wrote:
>
>> Sarath,
>>
>> Is this the external table where you have ran the query? How did you 
>> loaded the table?  Because it looks like the error is about the file 
>> related to table than CDH Jar.
>>
>> Thank You,
>>
>> Manish
>>
>> *From:*Sarath [mailto:sarathchandra.josyam@algofusiontech.com]
>> *Sent:* Tuesday, September 25, 2012 3:48 PM
>> *To:* user@hive.apache.org <ma...@hive.apache.org>
>> *Subject:* Hive query failing
>>
>> Hi,
>>
>> When I run the query "select count(1) from table1;" it fails with the 
>> exception message as below.
>>
>> /java.io.FileNotFoundException: File does not exist: 
>> /usr/lib/hive/lib/hive-builtins-0.8.1-cdh4.0.1.jar
>>     at 
>> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:736)
>>     at 
>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
>>     at 
>> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
>>     at 
>> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:245)
>>     at 
>> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:283)
>>     at 
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:354)
>>     at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
>>     at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>>     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
>>     at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:609)
>>     at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:604)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at 
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>>     at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:604)
>>     at 
>> org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:452)
>>     at 
>> org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:710)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>> Job Submission failed with exception 
>> 'java.io.FileNotFoundException(File does not exist: 
>> /usr/lib/hive/lib/hive-builtins-0.8.1-cdh4.0.1.jar)'
>> Execution failed with exit status: 2/
>>
>> But the JAR mentioned in the message exists and appropriate R/W 
>> permissions have been set on the folder /usr/lib/hive for the user.
>> What is going wrong?
>>
>> Regards,
>> Sarath.
>>

Re: Hive query failing

Posted by ku...@gmail.com.
The jar is being looked on HDFS as the exception suggests. Run the following commands:

$ hadoop fs -mkdir /usr/lib/hive/lib
$ hadoop fs -put $HIVE_HOME/lib/hive-builtins-0.8.1-cdh4.0.1.jar /usr/lib/hive/lib

Your queries should work now.

On Sep 25, 2012, at 6:46 AM, Manish.Bhoge <Ma...@target.com> wrote:

> Sarath,
>  
> Is this the external table where you have ran the query? How did you loaded the table?  Because it looks like the error is about the file related to table than CDH Jar.
>  
> Thank You,
> Manish
>  
> From: Sarath [mailto:sarathchandra.josyam@algofusiontech.com] 
> Sent: Tuesday, September 25, 2012 3:48 PM
> To: user@hive.apache.org
> Subject: Hive query failing
>  
> Hi,
> 
> When I run the query "select count(1) from table1;" it fails with the exception message as below.
> 
> java.io.FileNotFoundException: File does not exist: /usr/lib/hive/lib/hive-builtins-0.8.1-cdh4.0.1.jar
>     at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:736)
>     at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
>     at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
>     at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:245)
>     at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:283)
>     at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:354)
>     at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
>     at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
>     at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:609)
>     at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:604)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>     at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:604)
>     at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:452)
>     at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:710)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: /usr/lib/hive/lib/hive-builtins-0.8.1-cdh4.0.1.jar)'
> Execution failed with exit status: 2
> 
> But the JAR mentioned in the message exists and appropriate R/W permissions have been set on the folder /usr/lib/hive for the user.
> What is going wrong?
> 
> Regards,
> Sarath.

RE: Hive query failing

Posted by "Manish.Bhoge" <Ma...@target.com>.
Sarath,

Is this the external table where you have ran the query? How did you loaded the table?  Because it looks like the error is about the file related to table than CDH Jar.

Thank You,
Manish

From: Sarath [mailto:sarathchandra.josyam@algofusiontech.com]
Sent: Tuesday, September 25, 2012 3:48 PM
To: user@hive.apache.org
Subject: Hive query failing

Hi,

When I run the query "select count(1) from table1;" it fails with the exception message as below.

java.io.FileNotFoundException: File does not exist: /usr/lib/hive/lib/hive-builtins-0.8.1-cdh4.0.1.jar
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:736)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:208)
    at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:71)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:245)
    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:283)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:354)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:609)
    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:604)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:604)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:452)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:710)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Job Submission failed with exception 'java.io.FileNotFoundException(File does not exist: /usr/lib/hive/lib/hive-builtins-0.8.1-cdh4.0.1.jar)'
Execution failed with exit status: 2

But the JAR mentioned in the message exists and appropriate R/W permissions have been set on the folder /usr/lib/hive for the user.
What is going wrong?

Regards,
Sarath.