You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Aryeh Berkowitz <ar...@iswcorp.com> on 2010/02/26 14:33:30 UTC

Distinct errors

When I try to do a SELECT DISTINCT, I get "No such file" errors.

hive> SELECT DISTINCT URL FROM URLS;
Total MapReduce jobs = 1
Launching Job 1 out of 1
java.io.IOException: No such file or directory
        at java.io.UnixFileSystem.createFileExclusively(Native Method)
        at java.io.File.checkAndCreate(File.java:1704)
        at java.io.File.createTempFile(File.java:1792)
        at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:87)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask

Re: Distinct errors

Posted by Arvind Prabhakar <ar...@cloudera.com>.
The patch available for
HIVE-1277<https://issues.apache.org/jira/browse/HIVE-1277>should
address this issue. Can you please try that out and let us know if it
worked?

Thanks,
Arvind


On Fri, Feb 26, 2010 at 10:13 AM, Peter Sankauskas <pe...@motally.com>wrote:

> Looks like the same error as before... the temp directory is missing.
>
> Kind regards,
> Peter Sankauskas
>
> Motally, Inc
> Office: +1 (415) 932-6898
>
>
>
> On Fri, Feb 26, 2010 at 5:33 AM, Aryeh Berkowitz <ar...@iswcorp.com>wrote:
>
>>  When I try to do a SELECT DISTINCT, I get “No such file” errors.
>>
>>
>>
>> hive> SELECT DISTINCT URL FROM URLS;
>>
>> Total MapReduce jobs = 1
>>
>> Launching Job 1 out of 1
>>
>> java.io.IOException: No such file or directory
>>
>>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>>
>>         at java.io.File.checkAndCreate(File.java:1704)
>>
>>         at java.io.File.createTempFile(File.java:1792)
>>
>>         at
>> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:87)
>>
>>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>>
>>         at
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>>
>>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
>>
>>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>>
>>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>>
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>>
>>         at
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>>
>>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
>>
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.MapRedTask
>>
>
>

Re: Distinct errors

Posted by Arvind Prabhakar <ar...@cloudera.com>.
I have found that this error also occurs when running in pseudo-distributed
mode. The reason most of the time is the default value of the scratch
directory configuration which is /tmp/hive-${user.name}. Since /tmp gets
automatically cleared up on some systems it leads to this problem.

The patch provided with this
Jira<https://issues.apache.org/jira/browse/HIVE-1277>addresses this by
attempting to automatically create the scratch directory
if it does not yet exist.

Arvind

On Thu, Mar 25, 2010 at 9:58 AM, Yongqiang He <
heyongqiang@software.ict.ac.cn> wrote:

>  This error often occurs when you are running job in local mode.
> If you see this error, you can first do a “set
> hive.exec.scratchdir=/tmp/create_this_dir_first;”.
>
> Thanks
> Yongqiang
>
> On 2/26/10 10:13 AM, "Peter Sankauskas" <pe...@motally.com> wrote:
>
> Looks like the same error as before... the temp directory is missing.
>
> Kind regards,
> Peter Sankauskas
>
> Motally, Inc
> Office: +1 (415) 932-6898
>
>
> On Fri, Feb 26, 2010 at 5:33 AM, Aryeh Berkowitz <ar...@iswcorp.com>
> wrote:
>
> When I try to do a SELECT DISTINCT, I get “No such file” errors.
>
> hive> SELECT DISTINCT URL FROM URLS;
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> java.io.IOException: No such file or directory
>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>         at java.io.File.checkAndCreate(File.java:1704)
>         at java.io.File.createTempFile(File.java:1792)
>         at
> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:87)
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>         at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
>
>
>
>

Re: Distinct errors

Posted by Yongqiang He <he...@software.ict.ac.cn>.
This error often occurs when you are running job in local mode.
If you see this error, you can first do a ³set
hive.exec.scratchdir=/tmp/create_this_dir_first;².

Thanks
Yongqiang
On 2/26/10 10:13 AM, "Peter Sankauskas" <pe...@motally.com> wrote:

> Looks like the same error as before... the temp directory is missing.
> 
> Kind regards,
> Peter Sankauskas
> 
> Motally, Inc
> Office: +1 (415) 932-6898
> 
> 
> On Fri, Feb 26, 2010 at 5:33 AM, Aryeh Berkowitz <ar...@iswcorp.com> wrote:
>> When I try to do a SELECT DISTINCT, I get ³No such file² errors.
>>  
>> hive> SELECT DISTINCT URL FROM URLS;
>> Total MapReduce jobs = 1
>> Launching Job 1 out of 1
>> java.io.IOException: No such file or directory
>>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>>         at java.io.File.checkAndCreate(File.java:1704)
>>         at java.io.File.createTempFile(File.java:1792)
>>         at 
>> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:87)
>>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>>         at 
>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
>>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>>         at 
>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>>         at 
>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at 
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at 
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.
>> java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> FAILED: Execution Error, return code 1 from
>> org.apache.hadoop.hive.ql.exec.MapRedTask
> 
>

Re: Distinct errors

Posted by Peter Sankauskas <pe...@motally.com>.
Looks like the same error as before... the temp directory is missing.

Kind regards,
Peter Sankauskas

Motally, Inc
Office: +1 (415) 932-6898


On Fri, Feb 26, 2010 at 5:33 AM, Aryeh Berkowitz <ar...@iswcorp.com> wrote:

>  When I try to do a SELECT DISTINCT, I get “No such file” errors.
>
>
>
> hive> SELECT DISTINCT URL FROM URLS;
>
> Total MapReduce jobs = 1
>
> Launching Job 1 out of 1
>
> java.io.IOException: No such file or directory
>
>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>
>         at java.io.File.checkAndCreate(File.java:1704)
>
>         at java.io.File.createTempFile(File.java:1792)
>
>         at
> org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:87)
>
>         at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
>
>         at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:55)
>
>         at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:630)
>
>         at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:504)
>
>         at org.apache.hadoop.hive.ql.Driver.run(Driver.java:382)
>
>         at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:138)
>
>         at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:197)
>
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:303)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>         at java.lang.reflect.Method.invoke(Method.java:597)
>
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
>