You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by kiranprasad <ki...@imimobile.com> on 2011/09/16 17:04:02 UTC
ERROR 2118: Input path does not exist
Hi
I am getting the below mentioned exception after I load a file and do Filter on it.
The file(test.txt) is saved inside PIG home/data/ folder.
grunt> A= LOAD 'data/test.txt' USING PigStorage();
grunt> B= FOREACH A GENERATE $0;
grunt> DUMP B;
2011-09-17 01:17:43,408 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2011-09-17 01:17:43,409 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
2011-09-17 01:17:43,652 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B: Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig.impl.io.InterStorage) - scope-4 Operator Key: scope-4)
2011-09-17 01:17:43,662 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2011-09-17 01:17:43,688 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2011-09-17 01:17:43,689 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2011-09-17 01:17:43,742 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2011-09-17 01:17:43,754 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2011-09-17 01:17:46,447 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2011-09-17 01:17:46,609 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2011-09-17 01:17:47,525 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2011-09-17 01:17:48,158 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job null has failed! Stop running all dependent jobs
2011-09-17 01:17:48,162 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2011-09-17 01:17:48,169 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
2011-09-17 01:17:48,173 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2011-09-17 01:17:48,174 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17 01:17:48 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
N/A A,B MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
... 7 more
hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
Input(s):
Failed to read data from "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Output(s):
Failed to produce result in "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
null
2011-09-17 01:17:48,174 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias B
Details at logfile: /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
Any idea where am I making the mistake ?
Regards
Kiran.G
Re: ERROR 2118: Input path does not exist
Posted by Thejas Nair <th...@hortonworks.com>.
Do you see any relevant warning messages ?
You might want to check if the format of your data is compatible with
what PigStorage expects.
Thanks,
Thejas
On 9/22/11 3:15 AM, kiranprasad wrote:
> Even when I try to Store directly the loaded file it is generating only
> 5 output files.
> The size of test.txt is 1 GB where as the putput folder size is 178 MB.
>
> A= LOAD 'data/test.txt' USING PigStorage();
> STORE A INTO 'output';
>
>
> -----Original Message----- From: kiranprasad
> Sent: Thursday, September 22, 2011 10:27 AM
> To: Thejas Nair
> Cc: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
>
> But when compared the number of records in output should be 12,600 but
> there
> are only 2 records in Linux VM output folder.
>
> Regards
> Kiran.G
>
> -----Original Message----- From: Thejas Nair
> Sent: Wednesday, September 21, 2011 10:55 PM
> To: kiranprasad
> Cc: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> This is unlikely to be a configuration issue.
> This query will result in a map-only job, and the number of part files
> depends on the number of map tasks spawned. In typical configuration, in
> pig mapreduce mode, it will be based on block size. Different number of
> map tasks or part files should not cause a difference in results.
>
> You might want to check for any difference in delimiters used in the
> query. Having a look at the actual lines that are different might help
> you figure out what is wrong.
>
> Thanks,
> Thejas
>
>
>
> On 9/21/11 4:50 AM, kiranprasad wrote:
>> Hi
>>
>> In windows system using Cygwin the out put I got were 35 files
>> (part-m-00001 - 00035) with the same log file xyz.txt (1 GB size) and
>> same filter
>>
>> using CYGWIN (Master)
>> -----------
>> grunt> A= LOAD 'data/xyz.txt' USING PigStorage();
>> grunt> B= FILTER A BY ($0 matches '9948.*');
>> grunt> STORE B INTO 'data/output2';
>>
>> using Linux VM (Master)
>> ---------
>> used the same script in this VM in local mode and mapred mode only 5
>> files ((part-m-00001 - 00005) ) were generated as output and number of
>> records also does nt match.
>>
>> grunt> A= LOAD 'data/DNDDB.txt' USING PigStorage();
>> grunt> B= FILTER A BY ($0 matches '9948.*');
>> grunt> STORE B INTO 'data/output2';
>>
>> I think I missed some configurations !
>>
>> Regards
>>
>> Kiran.G
>>
>> -----Original Message----- From: kiranprasad
>> Sent: Wednesday, September 21, 2011 4:58 PM
>> To: Thejas Nair ; user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> Now I am able to connect to HDFS and execute the PIG Latin scripts in
>> mapred
>> mode,
>> but when I compared the results with local mode and mapred mode they are
>> different.
>>
>> Regards
>> Kiran.G
>>
>> -----Original Message----- From: Thejas Nair
>> Sent: Wednesday, September 21, 2011 2:23 AM
>> To: user@pig.apache.org
>> Cc: kiranprasad
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> The put command that Marek described can do that.
>> http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#put
>>
>> You will need to have hadoop client on that machine or move data to a
>> machine that has it. Copying 10GB of data over a LAN (?) should not take
>> too long.
>>
>> -Thejas
>>
>>
>> On 9/20/11 12:22 AM, kiranprasad wrote:
>>> How can I LOAD a file which is in another machine, of 10 GB size.
>>>
>>> -----Original Message----- From: Marek Miglinski
>>> Sent: Tuesday, September 20, 2011 12:19 PM
>>> To: user@pig.apache.org
>>> Subject: RE: ERROR 2118: Input path does not exist
>>>
>>> Hey,
>>>
>>> '/data/test.txt' is supposed to be on hdfs (if your not executing with
>>> -x local), put it there from your local drive with command:
>>> hadoop fs -put
>>>
>>> for ex, create dir and the put:
>>> hadoop fs -mkdir /data
>>> hadoop fs -put /data/test.txt /data/
>>>
>>>
>>> Sincerely,
>>> Marek M.
>>> ________________________________________
>>> From: kiranprasad [kiranprasad.g@imimobile.com]
>>> Sent: Tuesday, September 20, 2011 7:47 AM
>>> To: user@pig.apache.org
>>> Subject: Re: ERROR 2118: Input path does not exist
>>>
>>> Hi Marek
>>>
>>> I got the response as below
>>>
>>> [kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
>>> Found 1 items
>>> drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
>>> but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
>>> I am getting the same exception.
>>>
>>> Message: org.apache.pig.backend.executionengine.
>>> ExecException: ERROR 2118: Input path does not exist:
>>> hdfs://10.0.0.61/data/msis
>>> dns.txt
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
>>> Format.getSplits(PigInputFormat.java:280)
>>> at
>>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>>> at
>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
>>> 79)
>>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>>> at
>>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
>>> rol.java:247)
>>> at
>>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
>>> 9)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>>> Input pa th does not
>>> exist: hdfs://10.0.0.61/data/msisdns.txt
>>> at
>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
>>> InputFormat.java:224)
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
>>> nputFormat.listStatus(PigTextInputFormat.java:36)
>>> at
>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
>>> nputFormat.java:241)
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
>>> Format.getSplits(PigInputFormat.java:268)
>>> ... 7 more
>>>
>>>
>>> where is the default path of the data.txt configured ?
>>>
>>> Regards
>>> Kiran.G
>>>
>>> -----Original Message-----
>>> From: Marek Miglinski
>>> Sent: Monday, September 19, 2011 3:14 PM
>>> To: user@pig.apache.org
>>> Subject: RE: ERROR 2118: Input path does not exist
>>>
>>> hadoop fs -ls /
>>>
>>> BACKSHASH at the end!
>>>
>>> And:
>>> A= LOAD '/data/test.txt' USING PigStorage();
>>>
>>> BACKSLASH before data!
>>>
>>>
>>> -----Original Message-----
>>> From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
>>> Sent: Monday, September 19, 2011 12:10 PM
>>> To: user@pig.apache.org
>>> Subject: Re: ERROR 2118: Input path does not exist
>>>
>>> Hi
>>>
>>> I am unable t run the below mentioned command : hadoop fs -ls getting
>>> the
>>> same output.
>>>
>>> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
>>> ls: Cannot access .: No such file or directory.
>>>
>>> Below is the exception.
>>> Message: org.apache.pig.backend.executionengine.ExecException: ERROR
>>> 2118:
>>> Input path does not exist:
>>> hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
>>>
>>> Regards
>>> Kiran.G
>>>
>>> -----Original Message-----
>>> From: Marek Miglinski
>>> Sent: Sunday, September 18, 2011 1:09 AM
>>> To: user@pig.apache.org
>>> Subject: RE: ERROR 2118: Input path does not exist
>>>
>>> I meant that you should use absolute path when you load HDFS path from
>>> PIG,
>>> so this is not correct:
>>> A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
>>> A= LOAD '/data/test.txt' USING PigStorage();
>>>
>>> If you want to display contents of HDFS, type from terminal:
>>> hadoop fs -ls /
>>> To display first level structure.
>>> hadoop fs -lsr /
>>> To display all levels.
>>>
>>>
>>> Sincerely,
>>> Marek M.
>>> ________________________________________
>>> From: kiranprasad [kiranprasad.g@imimobile.com]
>>> Sent: Saturday, September 17, 2011 8:46 AM
>>> To: user@pig.apache.org
>>> Subject: Re: ERROR 2118: Input path does not exist
>>>
>>> When I do hadoop fs -ls I am getting the below
>>>
>>>
>>> [kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
>>> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
>>> ls: Cannot access .: No such file or directory.
>>>
>>> Regards
>>> Kiran.G
>>>
>>> -----Original Message-----
>>> From: Damien Hardy
>>> Sent: Friday, September 16, 2011 8:34 PM
>>> To: user@pig.apache.org
>>> Subject: Re: ERROR 2118: Input path does not exist
>>>
>>> What is the result of "hadoop fs -ls
>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>>
>>> Regards,
>>>
>>> --
>>> Damien
>>>
>>> Le 16/09/2011 17:04, kiranprasad a écrit :
>>>> Hi
>>>>
>>>> I am getting the below mentioned exception after I load a file and do
>>>> Filter on it.
>>>> The file(test.txt) is saved inside PIG home/data/ folder.
>>>>
>>>>
>>>> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
>>>> grunt> GENERATE $0; DUMP B;
>>>> 2011-09-17 01:17:43,408 [main] INFO
>>>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>>>> script: UNKNOWN
>>>> 2011-09-17 01:17:43,409 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>>> pig.usenewlogicalplan is set to true. New logical plan will be used.
>>>> 2011-09-17 01:17:43,652 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>>> (Name: B:
>>>> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
>>>> .impl.io.InterStorage)
>>>> - scope-4 Operator Key: scope-4)
>>>> 2011-09-17 01:17:43,662 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
>>>> r - File concatenation threshold: 100 optimistic? false
>>>> 2011-09-17 01:17:43,688 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>>>> yOptimizer
>>>> - MR plan size before optimization: 1
>>>> 2011-09-17 01:17:43,689 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>>>> yOptimizer
>>>> - MR plan size after optimization: 1
>>>> 2011-09-17 01:17:43,742 [main] INFO
>>>> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
>>>> added to the job
>>>> 2011-09-17 01:17:43,754 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>>>> lCompiler
>>>> - mapred.job.reduce.markreset.buffer.percent is not set, set to
>>>> default
>>>> 0.3
>>>> 2011-09-17 01:17:46,447 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>>>> lCompiler
>>>> - Setting up single store job
>>>> 2011-09-17 01:17:46,609 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>>> Launcher
>>>> - 1 map-reduce job(s) waiting for submission.
>>>> 2011-09-17 01:17:47,525 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>>> Launcher
>>>> - 0% complete
>>>> 2011-09-17 01:17:48,158 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>>> Launcher
>>>> - job null has failed! Stop running all dependent jobs
>>>> 2011-09-17 01:17:48,162 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>>> Launcher
>>>> - 100% complete
>>>> 2011-09-17 01:17:48,169 [main] ERROR
>>>> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
>>>> recreate exception from backend error:
>>>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>>>> Input path does not exist:
>>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>>> 2011-09-17 01:17:48,173 [main] ERROR
>>>> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s)
>>>> failed!
>>>> 2011-09-17 01:17:48,174 [main] INFO
>>>> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>>>>
>>>> HadoopVersion PigVersion UserId StartedAt FinishedAt
>>>> Features
>>>> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
>>>> 01:17:48 UNKNOWN
>>>>
>>>> Failed!
>>>>
>>>> Failed Jobs:
>>>> JobId Alias Feature Message Outputs
>>>> N/A A,B MAP_ONLY Message:
>>>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>>>> Input path does not exist:
>>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>>> at
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
>>>>
>>>>
>>>>
>>>> at
>>>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>>>> at
>>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
>>>>
>>>> at
>>>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>>>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>>>> at
>>>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>>>>
>>>>
>>>>
>>>> at
>>>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>>>> at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>>>> Input path does not exist:
>>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>>> at
>>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
>>>>
>>>>
>>>>
>>>> at
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
>>>>
>>>>
>>>>
>>>> at
>>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
>>>>
>>>>
>>>>
>>>> at
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
>>>>
>>>>
>>>>
>>>> ... 7 more
>>>> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>>>>
>>>> Input(s):
>>>> Failed to read data from
>>>> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>>>
>>>> Output(s):
>>>> Failed to produce result in
>>>> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>>>>
>>>> Counters:
>>>> Total records written : 0
>>>> Total bytes written : 0
>>>> Spillable Memory Manager spill count : 0 Total bags proactively
>>>> spilled: 0 Total records proactively spilled: 0
>>>>
>>>> Job DAG:
>>>> null
>>>>
>>>>
>>>> 2011-09-17 01:17:48,174 [main] INFO
>>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>>> Launcher
>>>> - Failed!
>>>> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
>>>> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
>>>> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>>>>
>>>> Any idea where am I making the mistake ?
>>>>
>>>>
>>>> Regards
>>>> Kiran.G
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>
Re: ERROR 2118: Input path does not exist
Posted by kiranprasad <ki...@imimobile.com>.
Even when I try to Store directly the loaded file it is generating only 5
output files.
The size of test.txt is 1 GB where as the putput folder size is 178 MB.
A= LOAD 'data/test.txt' USING PigStorage();
STORE A INTO 'output';
-----Original Message-----
From: kiranprasad
Sent: Thursday, September 22, 2011 10:27 AM
To: Thejas Nair
Cc: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
But when compared the number of records in output should be 12,600 but there
are only 2 records in Linux VM output folder.
Regards
Kiran.G
-----Original Message-----
From: Thejas Nair
Sent: Wednesday, September 21, 2011 10:55 PM
To: kiranprasad
Cc: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
This is unlikely to be a configuration issue.
This query will result in a map-only job, and the number of part files
depends on the number of map tasks spawned. In typical configuration, in
pig mapreduce mode, it will be based on block size. Different number of
map tasks or part files should not cause a difference in results.
You might want to check for any difference in delimiters used in the
query. Having a look at the actual lines that are different might help
you figure out what is wrong.
Thanks,
Thejas
On 9/21/11 4:50 AM, kiranprasad wrote:
> Hi
>
> In windows system using Cygwin the out put I got were 35 files
> (part-m-00001 - 00035) with the same log file xyz.txt (1 GB size) and
> same filter
>
> using CYGWIN (Master)
> -----------
> grunt> A= LOAD 'data/xyz.txt' USING PigStorage();
> grunt> B= FILTER A BY ($0 matches '9948.*');
> grunt> STORE B INTO 'data/output2';
>
> using Linux VM (Master)
> ---------
> used the same script in this VM in local mode and mapred mode only 5
> files ((part-m-00001 - 00005) ) were generated as output and number of
> records also does nt match.
>
> grunt> A= LOAD 'data/DNDDB.txt' USING PigStorage();
> grunt> B= FILTER A BY ($0 matches '9948.*');
> grunt> STORE B INTO 'data/output2';
>
> I think I missed some configurations !
>
> Regards
>
> Kiran.G
>
> -----Original Message----- From: kiranprasad
> Sent: Wednesday, September 21, 2011 4:58 PM
> To: Thejas Nair ; user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> Now I am able to connect to HDFS and execute the PIG Latin scripts in
> mapred
> mode,
> but when I compared the results with local mode and mapred mode they are
> different.
>
> Regards
> Kiran.G
>
> -----Original Message----- From: Thejas Nair
> Sent: Wednesday, September 21, 2011 2:23 AM
> To: user@pig.apache.org
> Cc: kiranprasad
> Subject: Re: ERROR 2118: Input path does not exist
>
> The put command that Marek described can do that.
> http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#put
>
> You will need to have hadoop client on that machine or move data to a
> machine that has it. Copying 10GB of data over a LAN (?) should not take
> too long.
>
> -Thejas
>
>
> On 9/20/11 12:22 AM, kiranprasad wrote:
>> How can I LOAD a file which is in another machine, of 10 GB size.
>>
>> -----Original Message----- From: Marek Miglinski
>> Sent: Tuesday, September 20, 2011 12:19 PM
>> To: user@pig.apache.org
>> Subject: RE: ERROR 2118: Input path does not exist
>>
>> Hey,
>>
>> '/data/test.txt' is supposed to be on hdfs (if your not executing with
>> -x local), put it there from your local drive with command:
>> hadoop fs -put
>>
>> for ex, create dir and the put:
>> hadoop fs -mkdir /data
>> hadoop fs -put /data/test.txt /data/
>>
>>
>> Sincerely,
>> Marek M.
>> ________________________________________
>> From: kiranprasad [kiranprasad.g@imimobile.com]
>> Sent: Tuesday, September 20, 2011 7:47 AM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> Hi Marek
>>
>> I got the response as below
>>
>> [kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
>> Found 1 items
>> drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
>> but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
>> I am getting the same exception.
>>
>> Message: org.apache.pig.backend.executionengine.
>> ExecException: ERROR 2118: Input path does not exist:
>> hdfs://10.0.0.61/data/msis
>> dns.txt
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
>> Format.getSplits(PigInputFormat.java:280)
>> at
>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
>> 79)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
>> rol.java:247)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
>> 9)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>> Input pa th does not
>> exist: hdfs://10.0.0.61/data/msisdns.txt
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
>> InputFormat.java:224)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
>> nputFormat.listStatus(PigTextInputFormat.java:36)
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
>> nputFormat.java:241)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
>> Format.getSplits(PigInputFormat.java:268)
>> ... 7 more
>>
>>
>> where is the default path of the data.txt configured ?
>>
>> Regards
>> Kiran.G
>>
>> -----Original Message-----
>> From: Marek Miglinski
>> Sent: Monday, September 19, 2011 3:14 PM
>> To: user@pig.apache.org
>> Subject: RE: ERROR 2118: Input path does not exist
>>
>> hadoop fs -ls /
>>
>> BACKSHASH at the end!
>>
>> And:
>> A= LOAD '/data/test.txt' USING PigStorage();
>>
>> BACKSLASH before data!
>>
>>
>> -----Original Message-----
>> From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
>> Sent: Monday, September 19, 2011 12:10 PM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> Hi
>>
>> I am unable t run the below mentioned command : hadoop fs -ls getting the
>> same output.
>>
>> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
>> ls: Cannot access .: No such file or directory.
>>
>> Below is the exception.
>> Message: org.apache.pig.backend.executionengine.ExecException: ERROR
>> 2118:
>> Input path does not exist:
>> hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
>>
>> Regards
>> Kiran.G
>>
>> -----Original Message-----
>> From: Marek Miglinski
>> Sent: Sunday, September 18, 2011 1:09 AM
>> To: user@pig.apache.org
>> Subject: RE: ERROR 2118: Input path does not exist
>>
>> I meant that you should use absolute path when you load HDFS path from
>> PIG,
>> so this is not correct:
>> A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
>> A= LOAD '/data/test.txt' USING PigStorage();
>>
>> If you want to display contents of HDFS, type from terminal:
>> hadoop fs -ls /
>> To display first level structure.
>> hadoop fs -lsr /
>> To display all levels.
>>
>>
>> Sincerely,
>> Marek M.
>> ________________________________________
>> From: kiranprasad [kiranprasad.g@imimobile.com]
>> Sent: Saturday, September 17, 2011 8:46 AM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> When I do hadoop fs -ls I am getting the below
>>
>>
>> [kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
>> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
>> ls: Cannot access .: No such file or directory.
>>
>> Regards
>> Kiran.G
>>
>> -----Original Message-----
>> From: Damien Hardy
>> Sent: Friday, September 16, 2011 8:34 PM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> What is the result of "hadoop fs -ls
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>
>> Regards,
>>
>> --
>> Damien
>>
>> Le 16/09/2011 17:04, kiranprasad a écrit :
>>> Hi
>>>
>>> I am getting the below mentioned exception after I load a file and do
>>> Filter on it.
>>> The file(test.txt) is saved inside PIG home/data/ folder.
>>>
>>>
>>> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
>>> grunt> GENERATE $0; DUMP B;
>>> 2011-09-17 01:17:43,408 [main] INFO
>>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>>> script: UNKNOWN
>>> 2011-09-17 01:17:43,409 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>> pig.usenewlogicalplan is set to true. New logical plan will be used.
>>> 2011-09-17 01:17:43,652 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>> (Name: B:
>>> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
>>> .impl.io.InterStorage)
>>> - scope-4 Operator Key: scope-4)
>>> 2011-09-17 01:17:43,662 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
>>> r - File concatenation threshold: 100 optimistic? false
>>> 2011-09-17 01:17:43,688 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>>> yOptimizer
>>> - MR plan size before optimization: 1
>>> 2011-09-17 01:17:43,689 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>>> yOptimizer
>>> - MR plan size after optimization: 1
>>> 2011-09-17 01:17:43,742 [main] INFO
>>> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
>>> added to the job
>>> 2011-09-17 01:17:43,754 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>>> lCompiler
>>> - mapred.job.reduce.markreset.buffer.percent is not set, set to
>>> default
>>> 0.3
>>> 2011-09-17 01:17:46,447 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>>> lCompiler
>>> - Setting up single store job
>>> 2011-09-17 01:17:46,609 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - 1 map-reduce job(s) waiting for submission.
>>> 2011-09-17 01:17:47,525 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - 0% complete
>>> 2011-09-17 01:17:48,158 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - job null has failed! Stop running all dependent jobs
>>> 2011-09-17 01:17:48,162 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - 100% complete
>>> 2011-09-17 01:17:48,169 [main] ERROR
>>> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
>>> recreate exception from backend error:
>>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>>> Input path does not exist:
>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>> 2011-09-17 01:17:48,173 [main] ERROR
>>> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
>>> 2011-09-17 01:17:48,174 [main] INFO
>>> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>>>
>>> HadoopVersion PigVersion UserId StartedAt FinishedAt
>>> Features
>>> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
>>> 01:17:48 UNKNOWN
>>>
>>> Failed!
>>>
>>> Failed Jobs:
>>> JobId Alias Feature Message Outputs
>>> N/A A,B MAP_ONLY Message:
>>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>>> Input path does not exist:
>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
>>>
>>>
>>> at
>>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>>> at
>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
>>> at
>>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>>> at
>>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>>>
>>>
>>> at
>>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>>> Input path does not exist:
>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>> at
>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
>>>
>>>
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
>>>
>>>
>>> at
>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
>>>
>>>
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
>>>
>>>
>>> ... 7 more
>>> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>>>
>>> Input(s):
>>> Failed to read data from
>>> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>>
>>> Output(s):
>>> Failed to produce result in
>>> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>>>
>>> Counters:
>>> Total records written : 0
>>> Total bytes written : 0
>>> Spillable Memory Manager spill count : 0 Total bags proactively
>>> spilled: 0 Total records proactively spilled: 0
>>>
>>> Job DAG:
>>> null
>>>
>>>
>>> 2011-09-17 01:17:48,174 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - Failed!
>>> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
>>> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
>>> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>>>
>>> Any idea where am I making the mistake ?
>>>
>>>
>>> Regards
>>> Kiran.G
>>
>>
>>
>>
>>
>
>
>
Re: ERROR 2118: Input path does not exist
Posted by kiranprasad <ki...@imimobile.com>.
But when compared the number of records in output should be 12,600 but there
are only 2 records in Linux VM output folder.
Regards
Kiran.G
-----Original Message-----
From: Thejas Nair
Sent: Wednesday, September 21, 2011 10:55 PM
To: kiranprasad
Cc: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
This is unlikely to be a configuration issue.
This query will result in a map-only job, and the number of part files
depends on the number of map tasks spawned. In typical configuration, in
pig mapreduce mode, it will be based on block size. Different number of
map tasks or part files should not cause a difference in results.
You might want to check for any difference in delimiters used in the
query. Having a look at the actual lines that are different might help
you figure out what is wrong.
Thanks,
Thejas
On 9/21/11 4:50 AM, kiranprasad wrote:
> Hi
>
> In windows system using Cygwin the out put I got were 35 files
> (part-m-00001 - 00035) with the same log file xyz.txt (1 GB size) and
> same filter
>
> using CYGWIN (Master)
> -----------
> grunt> A= LOAD 'data/xyz.txt' USING PigStorage();
> grunt> B= FILTER A BY ($0 matches '9948.*');
> grunt> STORE B INTO 'data/output2';
>
> using Linux VM (Master)
> ---------
> used the same script in this VM in local mode and mapred mode only 5
> files ((part-m-00001 - 00005) ) were generated as output and number of
> records also does nt match.
>
> grunt> A= LOAD 'data/DNDDB.txt' USING PigStorage();
> grunt> B= FILTER A BY ($0 matches '9948.*');
> grunt> STORE B INTO 'data/output2';
>
> I think I missed some configurations !
>
> Regards
>
> Kiran.G
>
> -----Original Message----- From: kiranprasad
> Sent: Wednesday, September 21, 2011 4:58 PM
> To: Thejas Nair ; user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> Now I am able to connect to HDFS and execute the PIG Latin scripts in
> mapred
> mode,
> but when I compared the results with local mode and mapred mode they are
> different.
>
> Regards
> Kiran.G
>
> -----Original Message----- From: Thejas Nair
> Sent: Wednesday, September 21, 2011 2:23 AM
> To: user@pig.apache.org
> Cc: kiranprasad
> Subject: Re: ERROR 2118: Input path does not exist
>
> The put command that Marek described can do that.
> http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#put
>
> You will need to have hadoop client on that machine or move data to a
> machine that has it. Copying 10GB of data over a LAN (?) should not take
> too long.
>
> -Thejas
>
>
> On 9/20/11 12:22 AM, kiranprasad wrote:
>> How can I LOAD a file which is in another machine, of 10 GB size.
>>
>> -----Original Message----- From: Marek Miglinski
>> Sent: Tuesday, September 20, 2011 12:19 PM
>> To: user@pig.apache.org
>> Subject: RE: ERROR 2118: Input path does not exist
>>
>> Hey,
>>
>> '/data/test.txt' is supposed to be on hdfs (if your not executing with
>> -x local), put it there from your local drive with command:
>> hadoop fs -put
>>
>> for ex, create dir and the put:
>> hadoop fs -mkdir /data
>> hadoop fs -put /data/test.txt /data/
>>
>>
>> Sincerely,
>> Marek M.
>> ________________________________________
>> From: kiranprasad [kiranprasad.g@imimobile.com]
>> Sent: Tuesday, September 20, 2011 7:47 AM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> Hi Marek
>>
>> I got the response as below
>>
>> [kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
>> Found 1 items
>> drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
>> but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
>> I am getting the same exception.
>>
>> Message: org.apache.pig.backend.executionengine.
>> ExecException: ERROR 2118: Input path does not exist:
>> hdfs://10.0.0.61/data/msis
>> dns.txt
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
>> Format.getSplits(PigInputFormat.java:280)
>> at
>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
>> 79)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
>> rol.java:247)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
>> 9)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>> Input pa th does not
>> exist: hdfs://10.0.0.61/data/msisdns.txt
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
>> InputFormat.java:224)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
>> nputFormat.listStatus(PigTextInputFormat.java:36)
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
>> nputFormat.java:241)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
>> Format.getSplits(PigInputFormat.java:268)
>> ... 7 more
>>
>>
>> where is the default path of the data.txt configured ?
>>
>> Regards
>> Kiran.G
>>
>> -----Original Message-----
>> From: Marek Miglinski
>> Sent: Monday, September 19, 2011 3:14 PM
>> To: user@pig.apache.org
>> Subject: RE: ERROR 2118: Input path does not exist
>>
>> hadoop fs -ls /
>>
>> BACKSHASH at the end!
>>
>> And:
>> A= LOAD '/data/test.txt' USING PigStorage();
>>
>> BACKSLASH before data!
>>
>>
>> -----Original Message-----
>> From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
>> Sent: Monday, September 19, 2011 12:10 PM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> Hi
>>
>> I am unable t run the below mentioned command : hadoop fs -ls getting the
>> same output.
>>
>> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
>> ls: Cannot access .: No such file or directory.
>>
>> Below is the exception.
>> Message: org.apache.pig.backend.executionengine.ExecException: ERROR
>> 2118:
>> Input path does not exist:
>> hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
>>
>> Regards
>> Kiran.G
>>
>> -----Original Message-----
>> From: Marek Miglinski
>> Sent: Sunday, September 18, 2011 1:09 AM
>> To: user@pig.apache.org
>> Subject: RE: ERROR 2118: Input path does not exist
>>
>> I meant that you should use absolute path when you load HDFS path from
>> PIG,
>> so this is not correct:
>> A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
>> A= LOAD '/data/test.txt' USING PigStorage();
>>
>> If you want to display contents of HDFS, type from terminal:
>> hadoop fs -ls /
>> To display first level structure.
>> hadoop fs -lsr /
>> To display all levels.
>>
>>
>> Sincerely,
>> Marek M.
>> ________________________________________
>> From: kiranprasad [kiranprasad.g@imimobile.com]
>> Sent: Saturday, September 17, 2011 8:46 AM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> When I do hadoop fs -ls I am getting the below
>>
>>
>> [kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
>> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
>> ls: Cannot access .: No such file or directory.
>>
>> Regards
>> Kiran.G
>>
>> -----Original Message-----
>> From: Damien Hardy
>> Sent: Friday, September 16, 2011 8:34 PM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> What is the result of "hadoop fs -ls
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>
>> Regards,
>>
>> --
>> Damien
>>
>> Le 16/09/2011 17:04, kiranprasad a écrit :
>>> Hi
>>>
>>> I am getting the below mentioned exception after I load a file and do
>>> Filter on it.
>>> The file(test.txt) is saved inside PIG home/data/ folder.
>>>
>>>
>>> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
>>> grunt> GENERATE $0; DUMP B;
>>> 2011-09-17 01:17:43,408 [main] INFO
>>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>>> script: UNKNOWN
>>> 2011-09-17 01:17:43,409 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>> pig.usenewlogicalplan is set to true. New logical plan will be used.
>>> 2011-09-17 01:17:43,652 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>> (Name: B:
>>> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
>>> .impl.io.InterStorage)
>>> - scope-4 Operator Key: scope-4)
>>> 2011-09-17 01:17:43,662 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
>>> r - File concatenation threshold: 100 optimistic? false
>>> 2011-09-17 01:17:43,688 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>>> yOptimizer
>>> - MR plan size before optimization: 1
>>> 2011-09-17 01:17:43,689 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>>> yOptimizer
>>> - MR plan size after optimization: 1
>>> 2011-09-17 01:17:43,742 [main] INFO
>>> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
>>> added to the job
>>> 2011-09-17 01:17:43,754 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>>> lCompiler
>>> - mapred.job.reduce.markreset.buffer.percent is not set, set to
>>> default
>>> 0.3
>>> 2011-09-17 01:17:46,447 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>>> lCompiler
>>> - Setting up single store job
>>> 2011-09-17 01:17:46,609 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - 1 map-reduce job(s) waiting for submission.
>>> 2011-09-17 01:17:47,525 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - 0% complete
>>> 2011-09-17 01:17:48,158 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - job null has failed! Stop running all dependent jobs
>>> 2011-09-17 01:17:48,162 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - 100% complete
>>> 2011-09-17 01:17:48,169 [main] ERROR
>>> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
>>> recreate exception from backend error:
>>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>>> Input path does not exist:
>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>> 2011-09-17 01:17:48,173 [main] ERROR
>>> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
>>> 2011-09-17 01:17:48,174 [main] INFO
>>> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>>>
>>> HadoopVersion PigVersion UserId StartedAt FinishedAt
>>> Features
>>> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
>>> 01:17:48 UNKNOWN
>>>
>>> Failed!
>>>
>>> Failed Jobs:
>>> JobId Alias Feature Message Outputs
>>> N/A A,B MAP_ONLY Message:
>>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>>> Input path does not exist:
>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
>>>
>>>
>>> at
>>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>>> at
>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
>>> at
>>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>>> at
>>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>>>
>>>
>>> at
>>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>>> Input path does not exist:
>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>> at
>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
>>>
>>>
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
>>>
>>>
>>> at
>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
>>>
>>>
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
>>>
>>>
>>> ... 7 more
>>> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>>>
>>> Input(s):
>>> Failed to read data from
>>> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>>
>>> Output(s):
>>> Failed to produce result in
>>> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>>>
>>> Counters:
>>> Total records written : 0
>>> Total bytes written : 0
>>> Spillable Memory Manager spill count : 0 Total bags proactively
>>> spilled: 0 Total records proactively spilled: 0
>>>
>>> Job DAG:
>>> null
>>>
>>>
>>> 2011-09-17 01:17:48,174 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - Failed!
>>> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
>>> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
>>> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>>>
>>> Any idea where am I making the mistake ?
>>>
>>>
>>> Regards
>>> Kiran.G
>>
>>
>>
>>
>>
>
>
>
Re: ERROR 2118: Input path does not exist
Posted by Thejas Nair <th...@hortonworks.com>.
This is unlikely to be a configuration issue.
This query will result in a map-only job, and the number of part files
depends on the number of map tasks spawned. In typical configuration, in
pig mapreduce mode, it will be based on block size. Different number of
map tasks or part files should not cause a difference in results.
You might want to check for any difference in delimiters used in the
query. Having a look at the actual lines that are different might help
you figure out what is wrong.
Thanks,
Thejas
On 9/21/11 4:50 AM, kiranprasad wrote:
> Hi
>
> In windows system using Cygwin the out put I got were 35 files
> (part-m-00001 - 00035) with the same log file xyz.txt (1 GB size) and
> same filter
>
> using CYGWIN (Master)
> -----------
> grunt> A= LOAD 'data/xyz.txt' USING PigStorage();
> grunt> B= FILTER A BY ($0 matches '9948.*');
> grunt> STORE B INTO 'data/output2';
>
> using Linux VM (Master)
> ---------
> used the same script in this VM in local mode and mapred mode only 5
> files ((part-m-00001 - 00005) ) were generated as output and number of
> records also does nt match.
>
> grunt> A= LOAD 'data/DNDDB.txt' USING PigStorage();
> grunt> B= FILTER A BY ($0 matches '9948.*');
> grunt> STORE B INTO 'data/output2';
>
> I think I missed some configurations !
>
> Regards
>
> Kiran.G
>
> -----Original Message----- From: kiranprasad
> Sent: Wednesday, September 21, 2011 4:58 PM
> To: Thejas Nair ; user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> Now I am able to connect to HDFS and execute the PIG Latin scripts in
> mapred
> mode,
> but when I compared the results with local mode and mapred mode they are
> different.
>
> Regards
> Kiran.G
>
> -----Original Message----- From: Thejas Nair
> Sent: Wednesday, September 21, 2011 2:23 AM
> To: user@pig.apache.org
> Cc: kiranprasad
> Subject: Re: ERROR 2118: Input path does not exist
>
> The put command that Marek described can do that.
> http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#put
>
> You will need to have hadoop client on that machine or move data to a
> machine that has it. Copying 10GB of data over a LAN (?) should not take
> too long.
>
> -Thejas
>
>
> On 9/20/11 12:22 AM, kiranprasad wrote:
>> How can I LOAD a file which is in another machine, of 10 GB size.
>>
>> -----Original Message----- From: Marek Miglinski
>> Sent: Tuesday, September 20, 2011 12:19 PM
>> To: user@pig.apache.org
>> Subject: RE: ERROR 2118: Input path does not exist
>>
>> Hey,
>>
>> '/data/test.txt' is supposed to be on hdfs (if your not executing with
>> -x local), put it there from your local drive with command:
>> hadoop fs -put
>>
>> for ex, create dir and the put:
>> hadoop fs -mkdir /data
>> hadoop fs -put /data/test.txt /data/
>>
>>
>> Sincerely,
>> Marek M.
>> ________________________________________
>> From: kiranprasad [kiranprasad.g@imimobile.com]
>> Sent: Tuesday, September 20, 2011 7:47 AM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> Hi Marek
>>
>> I got the response as below
>>
>> [kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
>> Found 1 items
>> drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
>> but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
>> I am getting the same exception.
>>
>> Message: org.apache.pig.backend.executionengine.
>> ExecException: ERROR 2118: Input path does not exist:
>> hdfs://10.0.0.61/data/msis
>> dns.txt
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
>> Format.getSplits(PigInputFormat.java:280)
>> at
>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
>> 79)
>> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
>> rol.java:247)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
>> 9)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>> Input pa th does not
>> exist: hdfs://10.0.0.61/data/msisdns.txt
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
>> InputFormat.java:224)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
>> nputFormat.listStatus(PigTextInputFormat.java:36)
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
>> nputFormat.java:241)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
>> Format.getSplits(PigInputFormat.java:268)
>> ... 7 more
>>
>>
>> where is the default path of the data.txt configured ?
>>
>> Regards
>> Kiran.G
>>
>> -----Original Message-----
>> From: Marek Miglinski
>> Sent: Monday, September 19, 2011 3:14 PM
>> To: user@pig.apache.org
>> Subject: RE: ERROR 2118: Input path does not exist
>>
>> hadoop fs -ls /
>>
>> BACKSHASH at the end!
>>
>> And:
>> A= LOAD '/data/test.txt' USING PigStorage();
>>
>> BACKSLASH before data!
>>
>>
>> -----Original Message-----
>> From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
>> Sent: Monday, September 19, 2011 12:10 PM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> Hi
>>
>> I am unable t run the below mentioned command : hadoop fs -ls getting the
>> same output.
>>
>> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
>> ls: Cannot access .: No such file or directory.
>>
>> Below is the exception.
>> Message: org.apache.pig.backend.executionengine.ExecException: ERROR
>> 2118:
>> Input path does not exist:
>> hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
>>
>> Regards
>> Kiran.G
>>
>> -----Original Message-----
>> From: Marek Miglinski
>> Sent: Sunday, September 18, 2011 1:09 AM
>> To: user@pig.apache.org
>> Subject: RE: ERROR 2118: Input path does not exist
>>
>> I meant that you should use absolute path when you load HDFS path from
>> PIG,
>> so this is not correct:
>> A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
>> A= LOAD '/data/test.txt' USING PigStorage();
>>
>> If you want to display contents of HDFS, type from terminal:
>> hadoop fs -ls /
>> To display first level structure.
>> hadoop fs -lsr /
>> To display all levels.
>>
>>
>> Sincerely,
>> Marek M.
>> ________________________________________
>> From: kiranprasad [kiranprasad.g@imimobile.com]
>> Sent: Saturday, September 17, 2011 8:46 AM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> When I do hadoop fs -ls I am getting the below
>>
>>
>> [kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
>> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
>> ls: Cannot access .: No such file or directory.
>>
>> Regards
>> Kiran.G
>>
>> -----Original Message-----
>> From: Damien Hardy
>> Sent: Friday, September 16, 2011 8:34 PM
>> To: user@pig.apache.org
>> Subject: Re: ERROR 2118: Input path does not exist
>>
>> What is the result of "hadoop fs -ls
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>
>> Regards,
>>
>> --
>> Damien
>>
>> Le 16/09/2011 17:04, kiranprasad a écrit :
>>> Hi
>>>
>>> I am getting the below mentioned exception after I load a file and do
>>> Filter on it.
>>> The file(test.txt) is saved inside PIG home/data/ folder.
>>>
>>>
>>> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
>>> grunt> GENERATE $0; DUMP B;
>>> 2011-09-17 01:17:43,408 [main] INFO
>>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>>> script: UNKNOWN
>>> 2011-09-17 01:17:43,409 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>> pig.usenewlogicalplan is set to true. New logical plan will be used.
>>> 2011-09-17 01:17:43,652 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>>> (Name: B:
>>> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
>>> .impl.io.InterStorage)
>>> - scope-4 Operator Key: scope-4)
>>> 2011-09-17 01:17:43,662 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
>>> r - File concatenation threshold: 100 optimistic? false
>>> 2011-09-17 01:17:43,688 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>>> yOptimizer
>>> - MR plan size before optimization: 1
>>> 2011-09-17 01:17:43,689 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>>> yOptimizer
>>> - MR plan size after optimization: 1
>>> 2011-09-17 01:17:43,742 [main] INFO
>>> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
>>> added to the job
>>> 2011-09-17 01:17:43,754 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>>> lCompiler
>>> - mapred.job.reduce.markreset.buffer.percent is not set, set to
>>> default
>>> 0.3
>>> 2011-09-17 01:17:46,447 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>>> lCompiler
>>> - Setting up single store job
>>> 2011-09-17 01:17:46,609 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - 1 map-reduce job(s) waiting for submission.
>>> 2011-09-17 01:17:47,525 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - 0% complete
>>> 2011-09-17 01:17:48,158 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - job null has failed! Stop running all dependent jobs
>>> 2011-09-17 01:17:48,162 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - 100% complete
>>> 2011-09-17 01:17:48,169 [main] ERROR
>>> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
>>> recreate exception from backend error:
>>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>>> Input path does not exist:
>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>> 2011-09-17 01:17:48,173 [main] ERROR
>>> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
>>> 2011-09-17 01:17:48,174 [main] INFO
>>> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>>>
>>> HadoopVersion PigVersion UserId StartedAt FinishedAt
>>> Features
>>> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
>>> 01:17:48 UNKNOWN
>>>
>>> Failed!
>>>
>>> Failed Jobs:
>>> JobId Alias Feature Message Outputs
>>> N/A A,B MAP_ONLY Message:
>>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>>> Input path does not exist:
>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
>>>
>>>
>>> at
>>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>>> at
>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
>>> at
>>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>>> at
>>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>>>
>>>
>>> at
>>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>>> at java.lang.Thread.run(Thread.java:619)
>>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>>> Input path does not exist:
>>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>>> at
>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
>>>
>>>
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
>>>
>>>
>>> at
>>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
>>>
>>>
>>> at
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
>>>
>>>
>>> ... 7 more
>>> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>>>
>>> Input(s):
>>> Failed to read data from
>>> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>>
>>> Output(s):
>>> Failed to produce result in
>>> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>>>
>>> Counters:
>>> Total records written : 0
>>> Total bytes written : 0
>>> Spillable Memory Manager spill count : 0 Total bags proactively
>>> spilled: 0 Total records proactively spilled: 0
>>>
>>> Job DAG:
>>> null
>>>
>>>
>>> 2011-09-17 01:17:48,174 [main] INFO
>>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>>> Launcher
>>> - Failed!
>>> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
>>> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
>>> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>>>
>>> Any idea where am I making the mistake ?
>>>
>>>
>>> Regards
>>> Kiran.G
>>
>>
>>
>>
>>
>
>
>
Re: ERROR 2118: Input path does not exist
Posted by kiranprasad <ki...@imimobile.com>.
Hi
In windows system using Cygwin the out put I got were 35 files
(part-m-00001 - 00035) with the same log file xyz.txt (1 GB size) and same
filter
using CYGWIN (Master)
-----------
grunt> A= LOAD 'data/xyz.txt' USING PigStorage();
grunt> B= FILTER A BY ($0 matches '9948.*');
grunt> STORE B INTO 'data/output2';
using Linux VM (Master)
---------
used the same script in this VM in local mode and mapred mode only 5 files
((part-m-00001 - 00005) ) were generated as output and number of records
also does nt match.
grunt> A= LOAD 'data/DNDDB.txt' USING PigStorage();
grunt> B= FILTER A BY ($0 matches '9948.*');
grunt> STORE B INTO 'data/output2';
I think I missed some configurations !
Regards
Kiran.G
-----Original Message-----
From: kiranprasad
Sent: Wednesday, September 21, 2011 4:58 PM
To: Thejas Nair ; user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
Now I am able to connect to HDFS and execute the PIG Latin scripts in mapred
mode,
but when I compared the results with local mode and mapred mode they are
different.
Regards
Kiran.G
-----Original Message-----
From: Thejas Nair
Sent: Wednesday, September 21, 2011 2:23 AM
To: user@pig.apache.org
Cc: kiranprasad
Subject: Re: ERROR 2118: Input path does not exist
The put command that Marek described can do that.
http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#put
You will need to have hadoop client on that machine or move data to a
machine that has it. Copying 10GB of data over a LAN (?) should not take
too long.
-Thejas
On 9/20/11 12:22 AM, kiranprasad wrote:
> How can I LOAD a file which is in another machine, of 10 GB size.
>
> -----Original Message----- From: Marek Miglinski
> Sent: Tuesday, September 20, 2011 12:19 PM
> To: user@pig.apache.org
> Subject: RE: ERROR 2118: Input path does not exist
>
> Hey,
>
> '/data/test.txt' is supposed to be on hdfs (if your not executing with
> -x local), put it there from your local drive with command:
> hadoop fs -put
>
> for ex, create dir and the put:
> hadoop fs -mkdir /data
> hadoop fs -put /data/test.txt /data/
>
>
> Sincerely,
> Marek M.
> ________________________________________
> From: kiranprasad [kiranprasad.g@imimobile.com]
> Sent: Tuesday, September 20, 2011 7:47 AM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> Hi Marek
>
> I got the response as below
>
> [kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
> Found 1 items
> drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
> but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
> I am getting the same exception.
>
> Message: org.apache.pig.backend.executionengine.
> ExecException: ERROR 2118: Input path does not exist:
> hdfs://10.0.0.61/data/msis
> dns.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
> Format.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
> 79)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
> rol.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
> 9)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input pa th does not
> exist: hdfs://10.0.0.61/data/msisdns.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
> InputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
> nputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
> nputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
> Format.getSplits(PigInputFormat.java:268)
> ... 7 more
>
>
> where is the default path of the data.txt configured ?
>
> Regards
> Kiran.G
>
> -----Original Message-----
> From: Marek Miglinski
> Sent: Monday, September 19, 2011 3:14 PM
> To: user@pig.apache.org
> Subject: RE: ERROR 2118: Input path does not exist
>
> hadoop fs -ls /
>
> BACKSHASH at the end!
>
> And:
> A= LOAD '/data/test.txt' USING PigStorage();
>
> BACKSLASH before data!
>
>
> -----Original Message-----
> From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
> Sent: Monday, September 19, 2011 12:10 PM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> Hi
>
> I am unable t run the below mentioned command : hadoop fs -ls getting the
> same output.
>
> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
> ls: Cannot access .: No such file or directory.
>
> Below is the exception.
> Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
>
> Regards
> Kiran.G
>
> -----Original Message-----
> From: Marek Miglinski
> Sent: Sunday, September 18, 2011 1:09 AM
> To: user@pig.apache.org
> Subject: RE: ERROR 2118: Input path does not exist
>
> I meant that you should use absolute path when you load HDFS path from
> PIG,
> so this is not correct:
> A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
> A= LOAD '/data/test.txt' USING PigStorage();
>
> If you want to display contents of HDFS, type from terminal:
> hadoop fs -ls /
> To display first level structure.
> hadoop fs -lsr /
> To display all levels.
>
>
> Sincerely,
> Marek M.
> ________________________________________
> From: kiranprasad [kiranprasad.g@imimobile.com]
> Sent: Saturday, September 17, 2011 8:46 AM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> When I do hadoop fs -ls I am getting the below
>
>
> [kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
> ls: Cannot access .: No such file or directory.
>
> Regards
> Kiran.G
>
> -----Original Message-----
> From: Damien Hardy
> Sent: Friday, September 16, 2011 8:34 PM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> What is the result of "hadoop fs -ls
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Regards,
>
> --
> Damien
>
> Le 16/09/2011 17:04, kiranprasad a écrit :
>> Hi
>>
>> I am getting the below mentioned exception after I load a file and do
>> Filter on it.
>> The file(test.txt) is saved inside PIG home/data/ folder.
>>
>>
>> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
>> grunt> GENERATE $0; DUMP B;
>> 2011-09-17 01:17:43,408 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>> script: UNKNOWN
>> 2011-09-17 01:17:43,409 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> pig.usenewlogicalplan is set to true. New logical plan will be used.
>> 2011-09-17 01:17:43,652 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> (Name: B:
>> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
>> .impl.io.InterStorage)
>> - scope-4 Operator Key: scope-4)
>> 2011-09-17 01:17:43,662 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
>> r - File concatenation threshold: 100 optimistic? false
>> 2011-09-17 01:17:43,688 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>> yOptimizer
>> - MR plan size before optimization: 1
>> 2011-09-17 01:17:43,689 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>> yOptimizer
>> - MR plan size after optimization: 1
>> 2011-09-17 01:17:43,742 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
>> added to the job
>> 2011-09-17 01:17:43,754 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>> lCompiler
>> - mapred.job.reduce.markreset.buffer.percent is not set, set to
>> default
>> 0.3
>> 2011-09-17 01:17:46,447 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>> lCompiler
>> - Setting up single store job
>> 2011-09-17 01:17:46,609 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - 1 map-reduce job(s) waiting for submission.
>> 2011-09-17 01:17:47,525 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - 0% complete
>> 2011-09-17 01:17:48,158 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - job null has failed! Stop running all dependent jobs
>> 2011-09-17 01:17:48,162 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - 100% complete
>> 2011-09-17 01:17:48,169 [main] ERROR
>> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
>> recreate exception from backend error:
>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>> Input path does not exist:
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>> 2011-09-17 01:17:48,173 [main] ERROR
>> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
>> 2011-09-17 01:17:48,174 [main] INFO
>> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>>
>> HadoopVersion PigVersion UserId StartedAt FinishedAt
>> Features
>> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
>> 01:17:48 UNKNOWN
>>
>> Failed!
>>
>> Failed Jobs:
>> JobId Alias Feature Message Outputs
>> N/A A,B MAP_ONLY Message:
>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>> Input path does not exist:
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
>>
>> at
>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>>
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>> Input path does not exist:
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
>>
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
>>
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
>>
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
>>
>> ... 7 more
>> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>>
>> Input(s):
>> Failed to read data from
>> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>
>> Output(s):
>> Failed to produce result in
>> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>>
>> Counters:
>> Total records written : 0
>> Total bytes written : 0
>> Spillable Memory Manager spill count : 0 Total bags proactively
>> spilled: 0 Total records proactively spilled: 0
>>
>> Job DAG:
>> null
>>
>>
>> 2011-09-17 01:17:48,174 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - Failed!
>> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
>> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
>> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>>
>> Any idea where am I making the mistake ?
>>
>>
>> Regards
>> Kiran.G
>
>
>
>
>
Re: ERROR 2118: Input path does not exist
Posted by kiranprasad <ki...@imimobile.com>.
Now I am able to connect to HDFS and execute the PIG Latin scripts in mapred
mode,
but when I compared the results with local mode and mapred mode they are
different.
Regards
Kiran.G
-----Original Message-----
From: Thejas Nair
Sent: Wednesday, September 21, 2011 2:23 AM
To: user@pig.apache.org
Cc: kiranprasad
Subject: Re: ERROR 2118: Input path does not exist
The put command that Marek described can do that.
http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#put
You will need to have hadoop client on that machine or move data to a
machine that has it. Copying 10GB of data over a LAN (?) should not take
too long.
-Thejas
On 9/20/11 12:22 AM, kiranprasad wrote:
> How can I LOAD a file which is in another machine, of 10 GB size.
>
> -----Original Message----- From: Marek Miglinski
> Sent: Tuesday, September 20, 2011 12:19 PM
> To: user@pig.apache.org
> Subject: RE: ERROR 2118: Input path does not exist
>
> Hey,
>
> '/data/test.txt' is supposed to be on hdfs (if your not executing with
> -x local), put it there from your local drive with command:
> hadoop fs -put
>
> for ex, create dir and the put:
> hadoop fs -mkdir /data
> hadoop fs -put /data/test.txt /data/
>
>
> Sincerely,
> Marek M.
> ________________________________________
> From: kiranprasad [kiranprasad.g@imimobile.com]
> Sent: Tuesday, September 20, 2011 7:47 AM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> Hi Marek
>
> I got the response as below
>
> [kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
> Found 1 items
> drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
> but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
> I am getting the same exception.
>
> Message: org.apache.pig.backend.executionengine.
> ExecException: ERROR 2118: Input path does not exist:
> hdfs://10.0.0.61/data/msis
> dns.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
> Format.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
> 79)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
> rol.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
> 9)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input pa th does not
> exist: hdfs://10.0.0.61/data/msisdns.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
> InputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
> nputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
> nputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
> Format.getSplits(PigInputFormat.java:268)
> ... 7 more
>
>
> where is the default path of the data.txt configured ?
>
> Regards
> Kiran.G
>
> -----Original Message-----
> From: Marek Miglinski
> Sent: Monday, September 19, 2011 3:14 PM
> To: user@pig.apache.org
> Subject: RE: ERROR 2118: Input path does not exist
>
> hadoop fs -ls /
>
> BACKSHASH at the end!
>
> And:
> A= LOAD '/data/test.txt' USING PigStorage();
>
> BACKSLASH before data!
>
>
> -----Original Message-----
> From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
> Sent: Monday, September 19, 2011 12:10 PM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> Hi
>
> I am unable t run the below mentioned command : hadoop fs -ls getting the
> same output.
>
> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
> ls: Cannot access .: No such file or directory.
>
> Below is the exception.
> Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
>
> Regards
> Kiran.G
>
> -----Original Message-----
> From: Marek Miglinski
> Sent: Sunday, September 18, 2011 1:09 AM
> To: user@pig.apache.org
> Subject: RE: ERROR 2118: Input path does not exist
>
> I meant that you should use absolute path when you load HDFS path from
> PIG,
> so this is not correct:
> A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
> A= LOAD '/data/test.txt' USING PigStorage();
>
> If you want to display contents of HDFS, type from terminal:
> hadoop fs -ls /
> To display first level structure.
> hadoop fs -lsr /
> To display all levels.
>
>
> Sincerely,
> Marek M.
> ________________________________________
> From: kiranprasad [kiranprasad.g@imimobile.com]
> Sent: Saturday, September 17, 2011 8:46 AM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> When I do hadoop fs -ls I am getting the below
>
>
> [kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
> ls: Cannot access .: No such file or directory.
>
> Regards
> Kiran.G
>
> -----Original Message-----
> From: Damien Hardy
> Sent: Friday, September 16, 2011 8:34 PM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> What is the result of "hadoop fs -ls
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Regards,
>
> --
> Damien
>
> Le 16/09/2011 17:04, kiranprasad a écrit :
>> Hi
>>
>> I am getting the below mentioned exception after I load a file and do
>> Filter on it.
>> The file(test.txt) is saved inside PIG home/data/ folder.
>>
>>
>> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
>> grunt> GENERATE $0; DUMP B;
>> 2011-09-17 01:17:43,408 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>> script: UNKNOWN
>> 2011-09-17 01:17:43,409 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> pig.usenewlogicalplan is set to true. New logical plan will be used.
>> 2011-09-17 01:17:43,652 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> (Name: B:
>> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
>> .impl.io.InterStorage)
>> - scope-4 Operator Key: scope-4)
>> 2011-09-17 01:17:43,662 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
>> r - File concatenation threshold: 100 optimistic? false
>> 2011-09-17 01:17:43,688 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>> yOptimizer
>> - MR plan size before optimization: 1
>> 2011-09-17 01:17:43,689 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>> yOptimizer
>> - MR plan size after optimization: 1
>> 2011-09-17 01:17:43,742 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
>> added to the job
>> 2011-09-17 01:17:43,754 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>> lCompiler
>> - mapred.job.reduce.markreset.buffer.percent is not set, set to
>> default
>> 0.3
>> 2011-09-17 01:17:46,447 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>> lCompiler
>> - Setting up single store job
>> 2011-09-17 01:17:46,609 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - 1 map-reduce job(s) waiting for submission.
>> 2011-09-17 01:17:47,525 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - 0% complete
>> 2011-09-17 01:17:48,158 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - job null has failed! Stop running all dependent jobs
>> 2011-09-17 01:17:48,162 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - 100% complete
>> 2011-09-17 01:17:48,169 [main] ERROR
>> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
>> recreate exception from backend error:
>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>> Input path does not exist:
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>> 2011-09-17 01:17:48,173 [main] ERROR
>> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
>> 2011-09-17 01:17:48,174 [main] INFO
>> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>>
>> HadoopVersion PigVersion UserId StartedAt FinishedAt
>> Features
>> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
>> 01:17:48 UNKNOWN
>>
>> Failed!
>>
>> Failed Jobs:
>> JobId Alias Feature Message Outputs
>> N/A A,B MAP_ONLY Message:
>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>> Input path does not exist:
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
>>
>> at
>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>>
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>> Input path does not exist:
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
>>
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
>>
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
>>
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
>>
>> ... 7 more
>> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>>
>> Input(s):
>> Failed to read data from
>> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>
>> Output(s):
>> Failed to produce result in
>> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>>
>> Counters:
>> Total records written : 0
>> Total bytes written : 0
>> Spillable Memory Manager spill count : 0 Total bags proactively
>> spilled: 0 Total records proactively spilled: 0
>>
>> Job DAG:
>> null
>>
>>
>> 2011-09-17 01:17:48,174 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - Failed!
>> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
>> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
>> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>>
>> Any idea where am I making the mistake ?
>>
>>
>> Regards
>> Kiran.G
>
>
>
>
>
Re: ERROR 2118: Input path does not exist
Posted by Thejas Nair <th...@hortonworks.com>.
The put command that Marek described can do that.
http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#put
You will need to have hadoop client on that machine or move data to a
machine that has it. Copying 10GB of data over a LAN (?) should not take
too long.
-Thejas
On 9/20/11 12:22 AM, kiranprasad wrote:
> How can I LOAD a file which is in another machine, of 10 GB size.
>
> -----Original Message----- From: Marek Miglinski
> Sent: Tuesday, September 20, 2011 12:19 PM
> To: user@pig.apache.org
> Subject: RE: ERROR 2118: Input path does not exist
>
> Hey,
>
> '/data/test.txt' is supposed to be on hdfs (if your not executing with
> -x local), put it there from your local drive with command:
> hadoop fs -put
>
> for ex, create dir and the put:
> hadoop fs -mkdir /data
> hadoop fs -put /data/test.txt /data/
>
>
> Sincerely,
> Marek M.
> ________________________________________
> From: kiranprasad [kiranprasad.g@imimobile.com]
> Sent: Tuesday, September 20, 2011 7:47 AM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> Hi Marek
>
> I got the response as below
>
> [kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
> Found 1 items
> drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
> but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
> I am getting the same exception.
>
> Message: org.apache.pig.backend.executionengine.
> ExecException: ERROR 2118: Input path does not exist:
> hdfs://10.0.0.61/data/msis
> dns.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
> Format.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
> 79)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
> rol.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
> 9)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input pa th does not
> exist: hdfs://10.0.0.61/data/msisdns.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
> InputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
> nputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
> nputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
> Format.getSplits(PigInputFormat.java:268)
> ... 7 more
>
>
> where is the default path of the data.txt configured ?
>
> Regards
> Kiran.G
>
> -----Original Message-----
> From: Marek Miglinski
> Sent: Monday, September 19, 2011 3:14 PM
> To: user@pig.apache.org
> Subject: RE: ERROR 2118: Input path does not exist
>
> hadoop fs -ls /
>
> BACKSHASH at the end!
>
> And:
> A= LOAD '/data/test.txt' USING PigStorage();
>
> BACKSLASH before data!
>
>
> -----Original Message-----
> From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
> Sent: Monday, September 19, 2011 12:10 PM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> Hi
>
> I am unable t run the below mentioned command : hadoop fs -ls getting the
> same output.
>
> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
> ls: Cannot access .: No such file or directory.
>
> Below is the exception.
> Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
>
> Regards
> Kiran.G
>
> -----Original Message-----
> From: Marek Miglinski
> Sent: Sunday, September 18, 2011 1:09 AM
> To: user@pig.apache.org
> Subject: RE: ERROR 2118: Input path does not exist
>
> I meant that you should use absolute path when you load HDFS path from PIG,
> so this is not correct:
> A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
> A= LOAD '/data/test.txt' USING PigStorage();
>
> If you want to display contents of HDFS, type from terminal:
> hadoop fs -ls /
> To display first level structure.
> hadoop fs -lsr /
> To display all levels.
>
>
> Sincerely,
> Marek M.
> ________________________________________
> From: kiranprasad [kiranprasad.g@imimobile.com]
> Sent: Saturday, September 17, 2011 8:46 AM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> When I do hadoop fs -ls I am getting the below
>
>
> [kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
> [kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
> ls: Cannot access .: No such file or directory.
>
> Regards
> Kiran.G
>
> -----Original Message-----
> From: Damien Hardy
> Sent: Friday, September 16, 2011 8:34 PM
> To: user@pig.apache.org
> Subject: Re: ERROR 2118: Input path does not exist
>
> What is the result of "hadoop fs -ls
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Regards,
>
> --
> Damien
>
> Le 16/09/2011 17:04, kiranprasad a écrit :
>> Hi
>>
>> I am getting the below mentioned exception after I load a file and do
>> Filter on it.
>> The file(test.txt) is saved inside PIG home/data/ folder.
>>
>>
>> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
>> grunt> GENERATE $0; DUMP B;
>> 2011-09-17 01:17:43,408 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>> script: UNKNOWN
>> 2011-09-17 01:17:43,409 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> pig.usenewlogicalplan is set to true. New logical plan will be used.
>> 2011-09-17 01:17:43,652 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> (Name: B:
>> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
>> .impl.io.InterStorage)
>> - scope-4 Operator Key: scope-4)
>> 2011-09-17 01:17:43,662 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
>> r - File concatenation threshold: 100 optimistic? false
>> 2011-09-17 01:17:43,688 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>> yOptimizer
>> - MR plan size before optimization: 1
>> 2011-09-17 01:17:43,689 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
>> yOptimizer
>> - MR plan size after optimization: 1
>> 2011-09-17 01:17:43,742 [main] INFO
>> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
>> added to the job
>> 2011-09-17 01:17:43,754 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>> lCompiler
>> - mapred.job.reduce.markreset.buffer.percent is not set, set to
>> default
>> 0.3
>> 2011-09-17 01:17:46,447 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
>> lCompiler
>> - Setting up single store job
>> 2011-09-17 01:17:46,609 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - 1 map-reduce job(s) waiting for submission.
>> 2011-09-17 01:17:47,525 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - 0% complete
>> 2011-09-17 01:17:48,158 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - job null has failed! Stop running all dependent jobs
>> 2011-09-17 01:17:48,162 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - 100% complete
>> 2011-09-17 01:17:48,169 [main] ERROR
>> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
>> recreate exception from backend error:
>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>> Input path does not exist:
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>> 2011-09-17 01:17:48,173 [main] ERROR
>> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
>> 2011-09-17 01:17:48,174 [main] INFO
>> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>>
>> HadoopVersion PigVersion UserId StartedAt FinishedAt
>> Features
>> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
>> 01:17:48 UNKNOWN
>>
>> Failed!
>>
>> Failed Jobs:
>> JobId Alias Feature Message Outputs
>> N/A A,B MAP_ONLY Message:
>> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
>> Input path does not exist:
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
>>
>> at
>> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
>> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>>
>> at
>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>> at java.lang.Thread.run(Thread.java:619)
>> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
>> Input path does not exist:
>> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
>>
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
>>
>> at
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
>>
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
>>
>> ... 7 more
>> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>>
>> Input(s):
>> Failed to read data from
>> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>>
>> Output(s):
>> Failed to produce result in
>> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>>
>> Counters:
>> Total records written : 0
>> Total bytes written : 0
>> Spillable Memory Manager spill count : 0 Total bags proactively
>> spilled: 0 Total records proactively spilled: 0
>>
>> Job DAG:
>> null
>>
>>
>> 2011-09-17 01:17:48,174 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
>> Launcher
>> - Failed!
>> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
>> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
>> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>>
>> Any idea where am I making the mistake ?
>>
>>
>> Regards
>> Kiran.G
>
>
>
>
>
Re: ERROR 2118: Input path does not exist
Posted by kiranprasad <ki...@imimobile.com>.
Out of 35 files only 5 output files are generated.
The total number of records should be 10345 but in these 5 files only 2
records are there.
What could be the reason for generating only 5 output files.
-----Original Message-----
From: kiranprasad
Sent: Tuesday, September 20, 2011 12:52 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
How can I LOAD a file which is in another machine, of 10 GB size.
-----Original Message-----
From: Marek Miglinski
Sent: Tuesday, September 20, 2011 12:19 PM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
Hey,
'/data/test.txt' is supposed to be on hdfs (if your not executing with -x
local), put it there from your local drive with command:
hadoop fs -put
for ex, create dir and the put:
hadoop fs -mkdir /data
hadoop fs -put /data/test.txt /data/
Sincerely,
Marek M.
________________________________________
From: kiranprasad [kiranprasad.g@imimobile.com]
Sent: Tuesday, September 20, 2011 7:47 AM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
Hi Marek
I got the response as below
[kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
Found 1 items
drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
I am getting the same exception.
Message: org.apache.pig.backend.executionengine.
ExecException: ERROR 2118: Input path does not exist:
hdfs://10.0.0.61/data/msis
dns.txt
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
Format.getSplits(PigInputFormat.java:280)
at
org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
79)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
rol.java:247)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
9)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
Input pa th does not
exist: hdfs://10.0.0.61/data/msisdns.txt
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
InputFormat.java:224)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
nputFormat.listStatus(PigTextInputFormat.java:36)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
nputFormat.java:241)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
Format.getSplits(PigInputFormat.java:268)
... 7 more
where is the default path of the data.txt configured ?
Regards
Kiran.G
-----Original Message-----
From: Marek Miglinski
Sent: Monday, September 19, 2011 3:14 PM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
hadoop fs -ls /
BACKSHASH at the end!
And:
A= LOAD '/data/test.txt' USING PigStorage();
BACKSLASH before data!
-----Original Message-----
From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
Sent: Monday, September 19, 2011 12:10 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
Hi
I am unable t run the below mentioned command : hadoop fs -ls getting the
same output.
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Below is the exception.
Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
Input path does not exist:
hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
Regards
Kiran.G
-----Original Message-----
From: Marek Miglinski
Sent: Sunday, September 18, 2011 1:09 AM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
I meant that you should use absolute path when you load HDFS path from PIG,
so this is not correct:
A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
A= LOAD '/data/test.txt' USING PigStorage();
If you want to display contents of HDFS, type from terminal:
hadoop fs -ls /
To display first level structure.
hadoop fs -lsr /
To display all levels.
Sincerely,
Marek M.
________________________________________
From: kiranprasad [kiranprasad.g@imimobile.com]
Sent: Saturday, September 17, 2011 8:46 AM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
When I do hadoop fs -ls I am getting the below
[kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Regards
Kiran.G
-----Original Message-----
From: Damien Hardy
Sent: Friday, September 16, 2011 8:34 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
What is the result of "hadoop fs -ls
hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Regards,
--
Damien
Le 16/09/2011 17:04, kiranprasad a écrit :
> Hi
>
> I am getting the below mentioned exception after I load a file and do
> Filter on it.
> The file(test.txt) is saved inside PIG home/data/ folder.
>
>
> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
> grunt> GENERATE $0; DUMP B;
> 2011-09-17 01:17:43,408 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2011-09-17 01:17:43,409 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-09-17 01:17:43,652 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B:
> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
> .impl.io.InterStorage)
> - scope-4 Operator Key: scope-4)
> 2011-09-17 01:17:43,662 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
> r - File concatenation threshold: 100 optimistic? false
> 2011-09-17 01:17:43,688 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
> yOptimizer
> - MR plan size before optimization: 1
> 2011-09-17 01:17:43,689 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
> yOptimizer
> - MR plan size after optimization: 1
> 2011-09-17 01:17:43,742 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
> added to the job
> 2011-09-17 01:17:43,754 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
> lCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to
> default
> 0.3
> 2011-09-17 01:17:46,447 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
> lCompiler
> - Setting up single store job
> 2011-09-17 01:17:46,609 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 1 map-reduce job(s) waiting for submission.
> 2011-09-17 01:17:47,525 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 0% complete
> 2011-09-17 01:17:48,158 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - job null has failed! Stop running all dependent jobs
> 2011-09-17 01:17:48,162 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 100% complete
> 2011-09-17 01:17:48,169 [main] ERROR
> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
> recreate exception from backend error:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> 2011-09-17 01:17:48,173 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
> 01:17:48 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A A,B MAP_ONLY Message:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
> ... 7 more
> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>
> Input(s):
> Failed to read data from
> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Output(s):
> Failed to produce result in
> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0 Total bags proactively
> spilled: 0 Total records proactively spilled: 0
>
> Job DAG:
> null
>
>
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - Failed!
> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>
> Any idea where am I making the mistake ?
>
>
> Regards
> Kiran.G
Re: ERROR 2118: Input path does not exist
Posted by kiranprasad <ki...@imimobile.com>.
How can I LOAD a file which is in another machine, of 10 GB size.
-----Original Message-----
From: Marek Miglinski
Sent: Tuesday, September 20, 2011 12:19 PM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
Hey,
'/data/test.txt' is supposed to be on hdfs (if your not executing with -x
local), put it there from your local drive with command:
hadoop fs -put
for ex, create dir and the put:
hadoop fs -mkdir /data
hadoop fs -put /data/test.txt /data/
Sincerely,
Marek M.
________________________________________
From: kiranprasad [kiranprasad.g@imimobile.com]
Sent: Tuesday, September 20, 2011 7:47 AM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
Hi Marek
I got the response as below
[kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
Found 1 items
drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
I am getting the same exception.
Message: org.apache.pig.backend.executionengine.
ExecException: ERROR 2118: Input path does not exist:
hdfs://10.0.0.61/data/msis
dns.txt
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
Format.getSplits(PigInputFormat.java:280)
at
org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
79)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
rol.java:247)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
9)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
Input pa th does not
exist: hdfs://10.0.0.61/data/msisdns.txt
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
InputFormat.java:224)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
nputFormat.listStatus(PigTextInputFormat.java:36)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
nputFormat.java:241)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
Format.getSplits(PigInputFormat.java:268)
... 7 more
where is the default path of the data.txt configured ?
Regards
Kiran.G
-----Original Message-----
From: Marek Miglinski
Sent: Monday, September 19, 2011 3:14 PM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
hadoop fs -ls /
BACKSHASH at the end!
And:
A= LOAD '/data/test.txt' USING PigStorage();
BACKSLASH before data!
-----Original Message-----
From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
Sent: Monday, September 19, 2011 12:10 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
Hi
I am unable t run the below mentioned command : hadoop fs -ls getting the
same output.
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Below is the exception.
Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
Input path does not exist:
hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
Regards
Kiran.G
-----Original Message-----
From: Marek Miglinski
Sent: Sunday, September 18, 2011 1:09 AM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
I meant that you should use absolute path when you load HDFS path from PIG,
so this is not correct:
A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
A= LOAD '/data/test.txt' USING PigStorage();
If you want to display contents of HDFS, type from terminal:
hadoop fs -ls /
To display first level structure.
hadoop fs -lsr /
To display all levels.
Sincerely,
Marek M.
________________________________________
From: kiranprasad [kiranprasad.g@imimobile.com]
Sent: Saturday, September 17, 2011 8:46 AM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
When I do hadoop fs -ls I am getting the below
[kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Regards
Kiran.G
-----Original Message-----
From: Damien Hardy
Sent: Friday, September 16, 2011 8:34 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
What is the result of "hadoop fs -ls
hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Regards,
--
Damien
Le 16/09/2011 17:04, kiranprasad a écrit :
> Hi
>
> I am getting the below mentioned exception after I load a file and do
> Filter on it.
> The file(test.txt) is saved inside PIG home/data/ folder.
>
>
> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
> grunt> GENERATE $0; DUMP B;
> 2011-09-17 01:17:43,408 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2011-09-17 01:17:43,409 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-09-17 01:17:43,652 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B:
> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
> .impl.io.InterStorage)
> - scope-4 Operator Key: scope-4)
> 2011-09-17 01:17:43,662 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
> r - File concatenation threshold: 100 optimistic? false
> 2011-09-17 01:17:43,688 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
> yOptimizer
> - MR plan size before optimization: 1
> 2011-09-17 01:17:43,689 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
> yOptimizer
> - MR plan size after optimization: 1
> 2011-09-17 01:17:43,742 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
> added to the job
> 2011-09-17 01:17:43,754 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
> lCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to
> default
> 0.3
> 2011-09-17 01:17:46,447 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
> lCompiler
> - Setting up single store job
> 2011-09-17 01:17:46,609 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 1 map-reduce job(s) waiting for submission.
> 2011-09-17 01:17:47,525 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 0% complete
> 2011-09-17 01:17:48,158 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - job null has failed! Stop running all dependent jobs
> 2011-09-17 01:17:48,162 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 100% complete
> 2011-09-17 01:17:48,169 [main] ERROR
> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
> recreate exception from backend error:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> 2011-09-17 01:17:48,173 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
> 01:17:48 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A A,B MAP_ONLY Message:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
> ... 7 more
> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>
> Input(s):
> Failed to read data from
> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Output(s):
> Failed to produce result in
> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0 Total bags proactively
> spilled: 0 Total records proactively spilled: 0
>
> Job DAG:
> null
>
>
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - Failed!
> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>
> Any idea where am I making the mistake ?
>
>
> Regards
> Kiran.G
RE: ERROR 2118: Input path does not exist
Posted by Marek Miglinski <mm...@seven.com>.
Hey,
'/data/test.txt' is supposed to be on hdfs (if your not executing with -x local), put it there from your local drive with command:
hadoop fs -put
for ex, create dir and the put:
hadoop fs -mkdir /data
hadoop fs -put /data/test.txt /data/
Sincerely,
Marek M.
________________________________________
From: kiranprasad [kiranprasad.g@imimobile.com]
Sent: Tuesday, September 20, 2011 7:47 AM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
Hi Marek
I got the response as below
[kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
Found 1 items
drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
I am getting the same exception.
Message: org.apache.pig.backend.executionengine.
ExecException: ERROR 2118: Input path does not exist:
hdfs://10.0.0.61/data/msis
dns.txt
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
Format.getSplits(PigInputFormat.java:280)
at
org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
79)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
rol.java:247)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
9)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
Input pa th does not
exist: hdfs://10.0.0.61/data/msisdns.txt
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
InputFormat.java:224)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
nputFormat.listStatus(PigTextInputFormat.java:36)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
nputFormat.java:241)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
Format.getSplits(PigInputFormat.java:268)
... 7 more
where is the default path of the data.txt configured ?
Regards
Kiran.G
-----Original Message-----
From: Marek Miglinski
Sent: Monday, September 19, 2011 3:14 PM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
hadoop fs -ls /
BACKSHASH at the end!
And:
A= LOAD '/data/test.txt' USING PigStorage();
BACKSLASH before data!
-----Original Message-----
From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
Sent: Monday, September 19, 2011 12:10 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
Hi
I am unable t run the below mentioned command : hadoop fs -ls getting the
same output.
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Below is the exception.
Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
Input path does not exist:
hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
Regards
Kiran.G
-----Original Message-----
From: Marek Miglinski
Sent: Sunday, September 18, 2011 1:09 AM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
I meant that you should use absolute path when you load HDFS path from PIG,
so this is not correct:
A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
A= LOAD '/data/test.txt' USING PigStorage();
If you want to display contents of HDFS, type from terminal:
hadoop fs -ls /
To display first level structure.
hadoop fs -lsr /
To display all levels.
Sincerely,
Marek M.
________________________________________
From: kiranprasad [kiranprasad.g@imimobile.com]
Sent: Saturday, September 17, 2011 8:46 AM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
When I do hadoop fs -ls I am getting the below
[kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Regards
Kiran.G
-----Original Message-----
From: Damien Hardy
Sent: Friday, September 16, 2011 8:34 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
What is the result of "hadoop fs -ls
hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Regards,
--
Damien
Le 16/09/2011 17:04, kiranprasad a écrit :
> Hi
>
> I am getting the below mentioned exception after I load a file and do
> Filter on it.
> The file(test.txt) is saved inside PIG home/data/ folder.
>
>
> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
> grunt> GENERATE $0; DUMP B;
> 2011-09-17 01:17:43,408 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2011-09-17 01:17:43,409 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-09-17 01:17:43,652 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B:
> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
> .impl.io.InterStorage)
> - scope-4 Operator Key: scope-4)
> 2011-09-17 01:17:43,662 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
> r - File concatenation threshold: 100 optimistic? false
> 2011-09-17 01:17:43,688 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
> yOptimizer
> - MR plan size before optimization: 1
> 2011-09-17 01:17:43,689 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
> yOptimizer
> - MR plan size after optimization: 1
> 2011-09-17 01:17:43,742 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
> added to the job
> 2011-09-17 01:17:43,754 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
> lCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to
> default
> 0.3
> 2011-09-17 01:17:46,447 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
> lCompiler
> - Setting up single store job
> 2011-09-17 01:17:46,609 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 1 map-reduce job(s) waiting for submission.
> 2011-09-17 01:17:47,525 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 0% complete
> 2011-09-17 01:17:48,158 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - job null has failed! Stop running all dependent jobs
> 2011-09-17 01:17:48,162 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 100% complete
> 2011-09-17 01:17:48,169 [main] ERROR
> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
> recreate exception from backend error:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> 2011-09-17 01:17:48,173 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
> 01:17:48 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A A,B MAP_ONLY Message:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
> ... 7 more
> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>
> Input(s):
> Failed to read data from
> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Output(s):
> Failed to produce result in
> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0 Total bags proactively
> spilled: 0 Total records proactively spilled: 0
>
> Job DAG:
> null
>
>
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - Failed!
> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>
> Any idea where am I making the mistake ?
>
>
> Regards
> Kiran.G
Re: ERROR 2118: Input path does not exist
Posted by kiranprasad <ki...@imimobile.com>.
Hi Marek
I got the response as below
[kiranprasad.g@pig4 bin]$ ./hadoop fs -ls /
Found 1 items
drwxr-xr-x - kiranprasad.g supergroup 0 2011-09-19 19:23 /tmp
but after loading (A= LOAD '/data/test.txt' USING PigStorage();),
I am getting the same exception.
Message: org.apache.pig.backend.executionengine.
ExecException: ERROR 2118: Input path does not exist:
hdfs://10.0.0.61/data/msis
dns.txt
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
Format.getSplits(PigInputFormat.java:280)
at
org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:7
79)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobCont
rol.java:247)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:27
9)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
Input pa th does not
exist: hdfs://10.0.0.61/data/msisdns.txt
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(File
InputFormat.java:224)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextI
nputFormat.listStatus(PigTextInputFormat.java:36)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileI
nputFormat.java:241)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInput
Format.getSplits(PigInputFormat.java:268)
... 7 more
where is the default path of the data.txt configured ?
Regards
Kiran.G
-----Original Message-----
From: Marek Miglinski
Sent: Monday, September 19, 2011 3:14 PM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
hadoop fs -ls /
BACKSHASH at the end!
And:
A= LOAD '/data/test.txt' USING PigStorage();
BACKSLASH before data!
-----Original Message-----
From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
Sent: Monday, September 19, 2011 12:10 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
Hi
I am unable t run the below mentioned command : hadoop fs -ls getting the
same output.
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Below is the exception.
Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
Input path does not exist:
hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
Regards
Kiran.G
-----Original Message-----
From: Marek Miglinski
Sent: Sunday, September 18, 2011 1:09 AM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
I meant that you should use absolute path when you load HDFS path from PIG,
so this is not correct:
A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
A= LOAD '/data/test.txt' USING PigStorage();
If you want to display contents of HDFS, type from terminal:
hadoop fs -ls /
To display first level structure.
hadoop fs -lsr /
To display all levels.
Sincerely,
Marek M.
________________________________________
From: kiranprasad [kiranprasad.g@imimobile.com]
Sent: Saturday, September 17, 2011 8:46 AM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
When I do hadoop fs -ls I am getting the below
[kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Regards
Kiran.G
-----Original Message-----
From: Damien Hardy
Sent: Friday, September 16, 2011 8:34 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
What is the result of "hadoop fs -ls
hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Regards,
--
Damien
Le 16/09/2011 17:04, kiranprasad a écrit :
> Hi
>
> I am getting the below mentioned exception after I load a file and do
> Filter on it.
> The file(test.txt) is saved inside PIG home/data/ folder.
>
>
> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
> grunt> GENERATE $0; DUMP B;
> 2011-09-17 01:17:43,408 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2011-09-17 01:17:43,409 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-09-17 01:17:43,652 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B:
> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
> .impl.io.InterStorage)
> - scope-4 Operator Key: scope-4)
> 2011-09-17 01:17:43,662 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
> r - File concatenation threshold: 100 optimistic? false
> 2011-09-17 01:17:43,688 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
> yOptimizer
> - MR plan size before optimization: 1
> 2011-09-17 01:17:43,689 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
> yOptimizer
> - MR plan size after optimization: 1
> 2011-09-17 01:17:43,742 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
> added to the job
> 2011-09-17 01:17:43,754 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
> lCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to
> default
> 0.3
> 2011-09-17 01:17:46,447 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
> lCompiler
> - Setting up single store job
> 2011-09-17 01:17:46,609 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 1 map-reduce job(s) waiting for submission.
> 2011-09-17 01:17:47,525 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 0% complete
> 2011-09-17 01:17:48,158 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - job null has failed! Stop running all dependent jobs
> 2011-09-17 01:17:48,162 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 100% complete
> 2011-09-17 01:17:48,169 [main] ERROR
> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
> recreate exception from backend error:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> 2011-09-17 01:17:48,173 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
> 01:17:48 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A A,B MAP_ONLY Message:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
> ... 7 more
> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>
> Input(s):
> Failed to read data from
> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Output(s):
> Failed to produce result in
> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0 Total bags proactively
> spilled: 0 Total records proactively spilled: 0
>
> Job DAG:
> null
>
>
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - Failed!
> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>
> Any idea where am I making the mistake ?
>
>
> Regards
> Kiran.G
RE: ERROR 2118: Input path does not exist
Posted by Marek Miglinski <mm...@seven.com>.
hadoop fs -ls /
BACKSHASH at the end!
And:
A= LOAD '/data/test.txt' USING PigStorage();
BACKSLASH before data!
-----Original Message-----
From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
Sent: Monday, September 19, 2011 12:10 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
Hi
I am unable t run the below mentioned command : hadoop fs -ls getting the same output.
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Below is the exception.
Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
Input path does not exist:
hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
Regards
Kiran.G
-----Original Message-----
From: Marek Miglinski
Sent: Sunday, September 18, 2011 1:09 AM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
I meant that you should use absolute path when you load HDFS path from PIG, so this is not correct:
A = LOAD 'data/test.txt' USING PigStorage(); This is correct:
A= LOAD '/data/test.txt' USING PigStorage();
If you want to display contents of HDFS, type from terminal:
hadoop fs -ls /
To display first level structure.
hadoop fs -lsr /
To display all levels.
Sincerely,
Marek M.
________________________________________
From: kiranprasad [kiranprasad.g@imimobile.com]
Sent: Saturday, September 17, 2011 8:46 AM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
When I do hadoop fs -ls I am getting the below
[kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Regards
Kiran.G
-----Original Message-----
From: Damien Hardy
Sent: Friday, September 16, 2011 8:34 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
What is the result of "hadoop fs -ls
hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Regards,
--
Damien
Le 16/09/2011 17:04, kiranprasad a écrit :
> Hi
>
> I am getting the below mentioned exception after I load a file and do
> Filter on it.
> The file(test.txt) is saved inside PIG home/data/ folder.
>
>
> grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A
> grunt> GENERATE $0; DUMP B;
> 2011-09-17 01:17:43,408 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2011-09-17 01:17:43,409 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-09-17 01:17:43,652 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B:
> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig
> .impl.io.InterStorage)
> - scope-4 Operator Key: scope-4)
> 2011-09-17 01:17:43,662 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompile
> r - File concatenation threshold: 100 optimistic? false
> 2011-09-17 01:17:43,688 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
> yOptimizer
> - MR plan size before optimization: 1
> 2011-09-17 01:17:43,689 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQuer
> yOptimizer
> - MR plan size after optimization: 1
> 2011-09-17 01:17:43,742 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
> added to the job
> 2011-09-17 01:17:43,754 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
> lCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to
> default
> 0.3
> 2011-09-17 01:17:46,447 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobContro
> lCompiler
> - Setting up single store job
> 2011-09-17 01:17:46,609 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 1 map-reduce job(s) waiting for submission.
> 2011-09-17 01:17:47,525 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 0% complete
> 2011-09-17 01:17:48,158 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - job null has failed! Stop running all dependent jobs
> 2011-09-17 01:17:48,162 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - 100% complete
> 2011-09-17 01:17:48,169 [main] ERROR
> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to
> recreate exception from backend error:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> 2011-09-17 01:17:48,173 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
> 01:17:48 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A A,B MAP_ONLY Message:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
> Input path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
> ... 7 more
> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>
> Input(s):
> Failed to read data from
> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Output(s):
> Failed to produce result in
> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0 Total bags proactively
> spilled: 0 Total records proactively spilled: 0
>
> Job DAG:
> null
>
>
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduce
> Launcher
> - Failed!
> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt
> - ERROR 1066: Unable to open iterator for alias B Details at logfile:
> /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>
> Any idea where am I making the mistake ?
>
>
> Regards
> Kiran.G
Re: ERROR 2118: Input path does not exist
Posted by kiranprasad <ki...@imimobile.com>.
Hi
I am unable t run the below mentioned command : hadoop fs -ls
getting the same output.
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Below is the exception.
Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
Input path does not exist:
hdfs://10.0.0.61/home/kiranprasad.g/pig-0.8.1/data/msisdns.txt
Regards
Kiran.G
-----Original Message-----
From: Marek Miglinski
Sent: Sunday, September 18, 2011 1:09 AM
To: user@pig.apache.org
Subject: RE: ERROR 2118: Input path does not exist
I meant that you should use absolute path when you load HDFS path from PIG,
so this is not correct:
A = LOAD 'data/test.txt' USING PigStorage();
This is correct:
A= LOAD '/data/test.txt' USING PigStorage();
If you want to display contents of HDFS, type from terminal:
hadoop fs -ls /
To display first level structure.
hadoop fs -lsr /
To display all levels.
Sincerely,
Marek M.
________________________________________
From: kiranprasad [kiranprasad.g@imimobile.com]
Sent: Saturday, September 17, 2011 8:46 AM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
When I do hadoop fs -ls I am getting the below
[kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Regards
Kiran.G
-----Original Message-----
From: Damien Hardy
Sent: Friday, September 16, 2011 8:34 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
What is the result of "hadoop fs -ls
hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Regards,
--
Damien
Le 16/09/2011 17:04, kiranprasad a écrit :
> Hi
>
> I am getting the below mentioned exception after I load a file and do
> Filter on it.
> The file(test.txt) is saved inside PIG home/data/ folder.
>
>
> grunt> A= LOAD 'data/test.txt' USING PigStorage();
> grunt> B= FOREACH A GENERATE $0;
> grunt> DUMP B;
> 2011-09-17 01:17:43,408 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2011-09-17 01:17:43,409 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-09-17 01:17:43,652 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B:
> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig.impl.io.InterStorage)
> - scope-4 Operator Key: scope-4)
> 2011-09-17 01:17:43,662 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2011-09-17 01:17:43,688 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2011-09-17 01:17:43,689 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2011-09-17 01:17:43,742 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
> 2011-09-17 01:17:43,754 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default
> 0.3
> 2011-09-17 01:17:46,447 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2011-09-17 01:17:46,609 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2011-09-17 01:17:47,525 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2011-09-17 01:17:48,158 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job null has failed! Stop running all dependent jobs
> 2011-09-17 01:17:48,162 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> 2011-09-17 01:17:48,169 [main] ERROR
> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate
> exception from backend error:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input
> path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> 2011-09-17 01:17:48,173 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
> 01:17:48 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A A,B MAP_ONLY Message:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input
> path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
> ... 7 more
> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>
> Input(s):
> Failed to read data from
> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Output(s):
> Failed to produce result in
> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> null
>
>
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Failed!
> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 1066: Unable to open iterator for alias B
> Details at logfile: /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>
> Any idea where am I making the mistake ?
>
>
> Regards
> Kiran.G
RE: ERROR 2118: Input path does not exist
Posted by Marek Miglinski <mm...@seven.com>.
I meant that you should use absolute path when you load HDFS path from PIG, so this is not correct:
A = LOAD 'data/test.txt' USING PigStorage();
This is correct:
A= LOAD '/data/test.txt' USING PigStorage();
If you want to display contents of HDFS, type from terminal:
hadoop fs -ls /
To display first level structure.
hadoop fs -lsr /
To display all levels.
Sincerely,
Marek M.
________________________________________
From: kiranprasad [kiranprasad.g@imimobile.com]
Sent: Saturday, September 17, 2011 8:46 AM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
When I do hadoop fs -ls I am getting the below
[kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Regards
Kiran.G
-----Original Message-----
From: Damien Hardy
Sent: Friday, September 16, 2011 8:34 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
What is the result of "hadoop fs -ls
hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Regards,
--
Damien
Le 16/09/2011 17:04, kiranprasad a écrit :
> Hi
>
> I am getting the below mentioned exception after I load a file and do
> Filter on it.
> The file(test.txt) is saved inside PIG home/data/ folder.
>
>
> grunt> A= LOAD 'data/test.txt' USING PigStorage();
> grunt> B= FOREACH A GENERATE $0;
> grunt> DUMP B;
> 2011-09-17 01:17:43,408 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2011-09-17 01:17:43,409 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-09-17 01:17:43,652 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B:
> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig.impl.io.InterStorage)
> - scope-4 Operator Key: scope-4)
> 2011-09-17 01:17:43,662 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2011-09-17 01:17:43,688 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2011-09-17 01:17:43,689 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2011-09-17 01:17:43,742 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
> 2011-09-17 01:17:43,754 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default
> 0.3
> 2011-09-17 01:17:46,447 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2011-09-17 01:17:46,609 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2011-09-17 01:17:47,525 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2011-09-17 01:17:48,158 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job null has failed! Stop running all dependent jobs
> 2011-09-17 01:17:48,162 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> 2011-09-17 01:17:48,169 [main] ERROR
> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate
> exception from backend error:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input
> path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> 2011-09-17 01:17:48,173 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
> 01:17:48 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A A,B MAP_ONLY Message:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input
> path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
> ... 7 more
> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>
> Input(s):
> Failed to read data from
> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Output(s):
> Failed to produce result in
> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> null
>
>
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Failed!
> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 1066: Unable to open iterator for alias B
> Details at logfile: /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>
> Any idea where am I making the mistake ?
>
>
> Regards
> Kiran.G
Re: ERROR 2118: Input path does not exist
Posted by kiranprasad <ki...@imimobile.com>.
When I do hadoop fs -ls I am getting the below
[kiranprasad.g@pig4 ~]$ cd hadoop-0.20.2
[kiranprasad.g@pig4 hadoop-0.20.2]$ bin/hadoop fs -ls
ls: Cannot access .: No such file or directory.
Regards
Kiran.G
-----Original Message-----
From: Damien Hardy
Sent: Friday, September 16, 2011 8:34 PM
To: user@pig.apache.org
Subject: Re: ERROR 2118: Input path does not exist
What is the result of "hadoop fs -ls
hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Regards,
--
Damien
Le 16/09/2011 17:04, kiranprasad a écrit :
> Hi
>
> I am getting the below mentioned exception after I load a file and do
> Filter on it.
> The file(test.txt) is saved inside PIG home/data/ folder.
>
>
> grunt> A= LOAD 'data/test.txt' USING PigStorage();
> grunt> B= FOREACH A GENERATE $0;
> grunt> DUMP B;
> 2011-09-17 01:17:43,408 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2011-09-17 01:17:43,409 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-09-17 01:17:43,652 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B:
> Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig.impl.io.InterStorage)
> - scope-4 Operator Key: scope-4)
> 2011-09-17 01:17:43,662 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2011-09-17 01:17:43,688 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2011-09-17 01:17:43,689 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2011-09-17 01:17:43,742 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
> 2011-09-17 01:17:43,754 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default
> 0.3
> 2011-09-17 01:17:46,447 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2011-09-17 01:17:46,609 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2011-09-17 01:17:47,525 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2011-09-17 01:17:48,158 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job null has failed! Stop running all dependent jobs
> 2011-09-17 01:17:48,162 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> 2011-09-17 01:17:48,169 [main] ERROR
> org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate
> exception from backend error:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input
> path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> 2011-09-17 01:17:48,173 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17
> 01:17:48 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A A,B MAP_ONLY Message:
> org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input
> path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
> at
> org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
> Input path does not exist:
> hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
> at
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
> ... 7 more
> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>
> Input(s):
> Failed to read data from
> "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Output(s):
> Failed to produce result in
> "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> null
>
>
> 2011-09-17 01:17:48,174 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Failed!
> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 1066: Unable to open iterator for alias B
> Details at logfile: /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>
> Any idea where am I making the mistake ?
>
>
> Regards
> Kiran.G
Re: ERROR 2118: Input path does not exist
Posted by Damien Hardy <dh...@figarocms.fr>.
What is the result of "hadoop fs -ls
hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Regards,
--
Damien
Le 16/09/2011 17:04, kiranprasad a écrit :
> Hi
>
> I am getting the below mentioned exception after I load a file and do Filter on it.
> The file(test.txt) is saved inside PIG home/data/ folder.
>
>
> grunt> A= LOAD 'data/test.txt' USING PigStorage();
> grunt> B= FOREACH A GENERATE $0;
> grunt> DUMP B;
> 2011-09-17 01:17:43,408 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
> 2011-09-17 01:17:43,409 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-09-17 01:17:43,652 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B: Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig.impl.io.InterStorage) - scope-4 Operator Key: scope-4)
> 2011-09-17 01:17:43,662 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
> 2011-09-17 01:17:43,688 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
> 2011-09-17 01:17:43,689 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
> 2011-09-17 01:17:43,742 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
> 2011-09-17 01:17:43,754 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2011-09-17 01:17:46,447 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
> 2011-09-17 01:17:46,609 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
> 2011-09-17 01:17:47,525 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
> 2011-09-17 01:17:48,158 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job null has failed! Stop running all dependent jobs
> 2011-09-17 01:17:48,162 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
> 2011-09-17 01:17:48,169 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> 2011-09-17 01:17:48,173 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2011-09-17 01:17:48,174 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17 01:17:48 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A A,B MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
> at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
> ... 7 more
> hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
>
> Input(s):
> Failed to read data from "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
>
> Output(s):
> Failed to produce result in "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> null
>
>
> 2011-09-17 01:17:48,174 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
> 2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias B
> Details at logfile: /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
>
> Any idea where am I making the mistake ?
>
>
> Regards
> Kiran.G
RE: ERROR 2118: Input path does not exist
Posted by Marek Miglinski <mm...@seven.com>.
Yes,
Type in absolute path ('/data/test.txt'):
A= LOAD '/data/test.txt' USING PigStorage();
Hadoop is looking for file in hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt, not in hdfs://10.0.0.61 /data/msisdn.txt where it is stored.
Sincerely,
Marek M.
-----Original Message-----
From: kiranprasad [mailto:kiranprasad.g@imimobile.com]
Sent: Friday, September 16, 2011 6:04 PM
To: user@pig.apache.org
Subject: ERROR 2118: Input path does not exist
Hi
I am getting the below mentioned exception after I load a file and do Filter on it.
The file(test.txt) is saved inside PIG home/data/ folder.
grunt> A= LOAD 'data/test.txt' USING PigStorage(); B= FOREACH A GENERATE
grunt> $0; DUMP B;
2011-09-17 01:17:43,408 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2011-09-17 01:17:43,409 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
2011-09-17 01:17:43,652 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: B: Store(hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250:org.apache.pig.impl.io.InterStorage) - scope-4 Operator Key: scope-4)
2011-09-17 01:17:43,662 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2011-09-17 01:17:43,688 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2011-09-17 01:17:43,689 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2011-09-17 01:17:43,742 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2011-09-17 01:17:43,754 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2011-09-17 01:17:46,447 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2011-09-17 01:17:46,609 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2011-09-17 01:17:47,525 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2011-09-17 01:17:48,158 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job null has failed! Stop running all dependent jobs
2011-09-17 01:17:48,162 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2011-09-17 01:17:48,169 [main] ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate exception from backend error: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
2011-09-17 01:17:48,173 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2011-09-17 01:17:48,174 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2 0.8.1 kiranprasad.g 2011-09-17 01:17:43 2011-09-17 01:17:48 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
N/A A,B MAP_ONLY Message: org.apache.pig.backend.executionengine.ExecException: ERROR 2118: Input path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:280)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:224)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextInputFormat.listStatus(PigTextInputFormat.java:36)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
... 7 more
hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250,
Input(s):
Failed to read data from "hdfs://10.0.0.61/user/kiranprasad.g/data/msisdn.txt"
Output(s):
Failed to produce result in "hdfs://10.0.0.61/tmp/temp-754030090/tmp1617007250"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0 Total bags proactively spilled: 0 Total records proactively spilled: 0
Job DAG:
null
2011-09-17 01:17:48,174 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2011-09-17 01:17:48,184 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias B Details at logfile: /home/kiranprasad.g/pig-0.8.1/pig_1316202429844.log
Any idea where am I making the mistake ?
Regards
Kiran.G