You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by gautier vivard <ga...@gmail.com> on 2013/05/13 09:40:38 UTC

select failed in Hive. java.io.FileNotFoundException

Hey,

I'm beginner in the hadoop world, and more specifically in Hive (I work on
hive-0.9.0 on Ubuntu). I work on MapR M3 in standalone mode. I'm training
on hive and have some issues with one query :

> SELECT year FROM records;
Automatically selecting local only mode for query
Total MapReduce jobs = 1
Launching Job 1 out of 1
Num 'java.io.FileNotFoundException"ber of reduce tasks is set to 0
since there's no reduce operator
Execution log at:
/tmp/mapr/mapr_20130506044545_e2ba972e-84a5-4e68-a33b-580be4a8d866.log
java.io.FileNotFoundException:
/var/mapr/cluster/mapred/jobTracker/staging/mapr406767829/.staging ()
    at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:518)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:330)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
    at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:875)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:843)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
    at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:693)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Job Submission failed with exception
'java.io.FileNotFoundException(/var/mapr/cluster/mapred/jobTracker/staging/mapr406767829/.staging
())'
Execution failed with exit status: 2
Obtaining error information

Task failed!
Task ID:
  Stage-1

Logs:

/tmp/mapr/hive.log
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.MapRedTask

But when I tried :

    > select * from records;

It works fine. I read that's because there is no reduce tasks in this query.
Any idea?

Thank's

Gautier

Re: HADOOP and Query Capabilities

Posted by Mohammad Tariq <do...@gmail.com>.
Sorry for the typo in the 3rd answer. I meant, question 1 covers this.

Warm Regards,
Tariq
cloudfront.blogspot.com


On Mon, May 13, 2013 at 10:13 PM, Mohammad Tariq <do...@gmail.com> wrote:

> Hello Nalin,
>
>      Please find my comments embedded below :
>
> 1. What tools are available to Query Hadoop data in real time vs batch?
> >> Real time : Cloudera Impala, Batch : Hive and Pig..You might wanna have
> a look at Apache Drill <http://incubator.apache.org/projects/drill.html>as well.
>
> 2. I believe HIVE provides a batch interface, not sure on what tools
> within HIVE support the query capabilities against HADOOP ?
>  >>I'm sorry, I didn't quite get this one.
>
> 3. Besides HIVE, are there any other Query tools to query HADOOP data
> (ad-hoc queries) ?
> >>I think question covers this one.
>
> 4. Finally, what skill set is required to use HIVE or other alternate
> tools ? Can business users uses these tools?
> >>I think it should be pretty easy for business folks. It would be helpful
> if you have some SQL knowledge.
>
> Warm Regards,
> Tariq
> cloudfront.blogspot.com
>
>
> On Mon, May 13, 2013 at 10:04 PM, Nalin Khosla <na...@rogers.com>wrote:
>
>>  Had a quick question wrt to querying HADOOP data;
>>
>> 1. What tools are available to Query Hadoop data in real time vs batch?
>>
>> 2. I believe HIVE provides a batch interface, not sure on what tools
>> within HIVE support the query capabilities against HADOOP ?
>>
>> 3. Besides HIVE, are there any other Query tools to query HADOOP data
>> (ad-hoc queries) ?
>>
>> 4. Finally, what skill set is required to use HIVE or other alternate
>> tools ? Can business users uses these tools?
>>
>> Thanks for your insight.
>>
>> Regards,
>> -nalin
>>
>>
>>
>>
>
>

Re: HADOOP and Query Capabilities

Posted by Mohammad Tariq <do...@gmail.com>.
Hello Nalin,

     Please find my comments embedded below :

1. What tools are available to Query Hadoop data in real time vs batch?
>> Real time : Cloudera Impala, Batch : Hive and Pig..You might wanna have
a look at Apache Drill <http://incubator.apache.org/projects/drill.html> as
well.

2. I believe HIVE provides a batch interface, not sure on what tools within
HIVE support the query capabilities against HADOOP ?
 >>I'm sorry, I didn't quite get this one.

3. Besides HIVE, are there any other Query tools to query HADOOP data
(ad-hoc queries) ?
>>I think question covers this one.

4. Finally, what skill set is required to use HIVE or other alternate tools
? Can business users uses these tools?
>>I think it should be pretty easy for business folks. It would be helpful
if you have some SQL knowledge.

Warm Regards,
Tariq
cloudfront.blogspot.com


On Mon, May 13, 2013 at 10:04 PM, Nalin Khosla <na...@rogers.com>wrote:

>  Had a quick question wrt to querying HADOOP data;
>
> 1. What tools are available to Query Hadoop data in real time vs batch?
>
> 2. I believe HIVE provides a batch interface, not sure on what tools
> within HIVE support the query capabilities against HADOOP ?
>
> 3. Besides HIVE, are there any other Query tools to query HADOOP data
> (ad-hoc queries) ?
>
> 4. Finally, what skill set is required to use HIVE or other alternate
> tools ? Can business users uses these tools?
>
> Thanks for your insight.
>
> Regards,
> -nalin
>
>
>
>

Re: HADOOP and Query Capabilities

Posted by Owen O'Malley <om...@apache.org>.
On Mon, May 13, 2013 at 9:34 AM, Nalin Khosla <na...@rogers.com>wrote:

> Had a quick question wrt to querying HADOOP data;
>
> 1. What tools are available to Query Hadoop data in real time vs batch?
>

The line between real time and batch isn't that clear. We are working on
substantially speeding up the performance of Hive (
http://www.slideshare.net/Hadoop_Summit/innovations-in-apache-hadoop-mapreduce-pig-hive-for-improving-query-performance).
The better question is whether you have small enough data so that it can
fit in RAM on your cluster. If so, you should look at Shark (
https://amplab.cs.berkeley.edu/2012/11/26/low-latency-sql-queries-at-massive-scale-a-performance-analysis-of-shark/)
or a proprietary MPP database such as Teradata or Impala.


>
> 2. I believe HIVE provides a batch interface, not sure on what tools
> within HIVE support the query capabilities against HADOOP ?
>

Hive currently uses MapReduce to run the queries. We plan on extending to
use Tez, which is a new Apache project that provides a richer framework for
queries.


>
> 3. Besides HIVE, are there any other Query tools to query HADOOP data
> (ad-hoc queries) ?
>

Pig and Cascading are the main open source ones for large data. Shark does
the smaller ad-hoc queries. Drill plans to fit into the ad-hoc space, but
hasn't made a release yet.

4. Finally, what skill set is required to use HIVE or other alternate tools
> ? Can business users uses these tools?
>

 Using Hive requires a learning curve. Business users will be able to run
queries against the data, but it will require someone with more engineering
background to design the table layouts and updating scheme.

-- Owen

HADOOP and Query Capabilities

Posted by Nalin Khosla <na...@rogers.com>.
Had a quick question wrt to querying HADOOP data;

1. What tools are available to Query Hadoop data in real time vs batch? 

2. I believe HIVE provides a batch interface, not sure on what tools within HIVE support the query capabilities against HADOOP ?

3. Besides HIVE, are there any other Query tools to query HADOOP data (ad-hoc queries) ?

4. Finally, what skill set is required to use HIVE or other alternate tools ? Can business users uses these tools?

Thanks for your insight.

Regards,
-nalin

Re: select failed in Hive. java.io.FileNotFoundException

Posted by Ted Xu <tx...@gopivotal.com>.
Hi Gautier,

Looks you're running Hive on top of local filesystem. The exception stack
said you haven't created staging directory. You can try creating the
directory manually, or changing the config parameter
"mapreduce.jobtracker.staging.root.dir" to a valid directory.


On Mon, May 13, 2013 at 4:38 PM, gautier vivard <ga...@gmail.com>wrote:

> Thank's for your response.
> I'm really a beginner, so I don't really understand what you mean by :
> "correctly"
>
>
> 2013/5/13 shashwat shriparv <dw...@gmail.com>
>
>> Set the hive class path correctly.
>>
>> *Thanks & Regards    *
>>
>> ∞
>> Shashwat Shriparv
>>
>>
>>
>> On Mon, May 13, 2013 at 1:10 PM, gautier vivard <ga...@gmail.com>wrote:
>>
>>> Hey,
>>>
>>> I'm beginner in the hadoop world, and more specifically in Hive (I work
>>> on hive-0.9.0 on Ubuntu). I work on MapR M3 in standalone mode. I'm
>>> training on hive and have some issues with one query :
>>>
>>> > SELECT year FROM records;
>>> Automatically selecting local only mode for query
>>> Total MapReduce jobs = 1
>>> Launching Job 1 out of 1
>>> Num 'java.io.FileNotFoundException"ber of reduce tasks is set to 0 since there's no reduce operator
>>> Execution log at: /tmp/mapr/mapr_20130506044545_e2ba972e-84a5-4e68-a33b-580be4a8d866.log
>>> java.io.FileNotFoundException: /var/mapr/cluster/mapred/jobTracker/staging/mapr406767829/.staging ()
>>>     at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>>     at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:518)
>>>     at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:330)
>>>     at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>>>     at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
>>>     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:875)
>>>     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
>>>     at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
>>>     at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:843)
>>>     at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
>>>     at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:693)
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>>> Job Submission failed with exception 'java.io.FileNotFoundException(/var/mapr/cluster/mapred/jobTracker/staging/mapr406767829/.staging ())'
>>> Execution failed with exit status: 2
>>> Obtaining error information
>>>
>>> Task failed!
>>> Task ID:
>>>   Stage-1
>>>
>>> Logs:
>>>
>>> /tmp/mapr/hive.log
>>> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>>>
>>> But when I tried :
>>>
>>>     > select * from records;
>>>
>>> It works fine. I read that's because there is no reduce tasks in this
>>> query.
>>> Any idea?
>>>
>>> Thank's
>>>
>>> Gautier
>>>
>>
>>
>


-- 
Regards,
Ted Xu

Re: select failed in Hive. java.io.FileNotFoundException

Posted by gautier vivard <ga...@gmail.com>.
Thank's for your response.
I'm really a beginner, so I don't really understand what you mean by :
"correctly"


2013/5/13 shashwat shriparv <dw...@gmail.com>

> Set the hive class path correctly.
>
> *Thanks & Regards    *
>
> ∞
> Shashwat Shriparv
>
>
>
> On Mon, May 13, 2013 at 1:10 PM, gautier vivard <ga...@gmail.com>wrote:
>
>> Hey,
>>
>> I'm beginner in the hadoop world, and more specifically in Hive (I work
>> on hive-0.9.0 on Ubuntu). I work on MapR M3 in standalone mode. I'm
>> training on hive and have some issues with one query :
>>
>> > SELECT year FROM records;
>> Automatically selecting local only mode for query
>> Total MapReduce jobs = 1
>> Launching Job 1 out of 1
>> Num 'java.io.FileNotFoundException"ber of reduce tasks is set to 0 since there's no reduce operator
>> Execution log at: /tmp/mapr/mapr_20130506044545_e2ba972e-84a5-4e68-a33b-580be4a8d866.log
>> java.io.FileNotFoundException: /var/mapr/cluster/mapred/jobTracker/staging/mapr406767829/.staging ()
>>     at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>>     at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:518)
>>     at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:330)
>>     at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>>     at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
>>     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:875)
>>     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
>>     at java.security.AccessController.doPrivileged(Native Method)
>>     at javax.security.auth.Subject.doAs(Subject.java:396)
>>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
>>     at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
>>     at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:843)
>>     at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
>>     at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:693)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
>> Job Submission failed with exception 'java.io.FileNotFoundException(/var/mapr/cluster/mapred/jobTracker/staging/mapr406767829/.staging ())'
>> Execution failed with exit status: 2
>> Obtaining error information
>>
>> Task failed!
>> Task ID:
>>   Stage-1
>>
>> Logs:
>>
>> /tmp/mapr/hive.log
>> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>>
>> But when I tried :
>>
>>     > select * from records;
>>
>> It works fine. I read that's because there is no reduce tasks in this
>> query.
>> Any idea?
>>
>> Thank's
>>
>> Gautier
>>
>
>

Re: select failed in Hive. java.io.FileNotFoundException

Posted by shashwat shriparv <dw...@gmail.com>.
Set the hive class path correctly.

*Thanks & Regards    *

∞
Shashwat Shriparv



On Mon, May 13, 2013 at 1:10 PM, gautier vivard <ga...@gmail.com>wrote:

> Hey,
>
> I'm beginner in the hadoop world, and more specifically in Hive (I work on
> hive-0.9.0 on Ubuntu). I work on MapR M3 in standalone mode. I'm training
> on hive and have some issues with one query :
>
> > SELECT year FROM records;
> Automatically selecting local only mode for query
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Num 'java.io.FileNotFoundException"ber of reduce tasks is set to 0 since there's no reduce operator
> Execution log at: /tmp/mapr/mapr_20130506044545_e2ba972e-84a5-4e68-a33b-580be4a8d866.log
> java.io.FileNotFoundException: /var/mapr/cluster/mapred/jobTracker/staging/mapr406767829/.staging ()
>     at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>     at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:518)
>     at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:330)
>     at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>     at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
>     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:875)
>     at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:869)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1126)
>     at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:869)
>     at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:843)
>     at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:435)
>     at org.apache.hadoop.hive.ql.exec.ExecDriver.main(ExecDriver.java:693)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> Job Submission failed with exception 'java.io.FileNotFoundException(/var/mapr/cluster/mapred/jobTracker/staging/mapr406767829/.staging ())'
> Execution failed with exit status: 2
> Obtaining error information
>
> Task failed!
> Task ID:
>   Stage-1
>
> Logs:
>
> /tmp/mapr/hive.log
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>
> But when I tried :
>
>     > select * from records;
>
> It works fine. I read that's because there is no reduce tasks in this
> query.
> Any idea?
>
> Thank's
>
> Gautier
>