You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by lei tang <fi...@gmail.com> on 2012/10/16 00:43:33 UTC
Help on running pig in local mode
Hi,
I'm using Pig for my daily job. Pig dose a good job in mapreduce mode in
our internal hadoop cluster. But it signals an error whenever I want to
run it in local mode. I believe this is due to some permission or config
issue. Does anybody know how I can address this problem? Any idea is
appreciated.
[ltang01@stg-trgt00 in]$ pig -x local
2012-10-11 17:29:25,707 [main] INFO org.apache.pig.Main - Apache Pig
version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
2012-10-11 17:29:25,708 [main] INFO org.apache.pig.Main - Logging error
messages to: /somewhere/in/pig_1350001765703.log
2012-10-11 17:29:26,068 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: file:///
grunt> A = load 'toy.txt';
grunt> dump A;
2012-10-11 17:29:36,056 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2012-10-11 17:29:36,218 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2012-10-11 17:29:36,249 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2012-10-11 17:29:36,250 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2012-10-11 17:29:36,382 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2012-10-11 17:29:36,403 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2012-10-11 17:29:36,445 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2012-10-11 17:29:36,486 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId=
2012-10-11 17:29:36,493 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2012-10-11 17:29:36,995 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2012-10-11 17:29:37,002 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job null has failed! Stop running all dependent jobs
2012-10-11 17:29:37,002 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2012-10-11 17:29:37,007 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
recreate exception from backend error: ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:679)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
2012-10-11 17:29:37,007 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2012-10-11 17:29:37,007 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats
reported below may be incomplete
2012-10-11 17:29:37,009 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt
Features
0.20.2-dev 0.10.0 ltang01 2012-10-11 17:29:36 2012-10-11
17:29:37 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
N/A A MAP_ONLY Message: ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:679)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
file:/tmp/temp-916111882/tmp1833113178,
Input(s):
Failed to read data from "file:///data1/home/ltang01/mf/prep/in/toy.txt"
Output(s):
Failed to produce result in "file:/tmp/temp-916111882/tmp1833113178"
Job DAG:
null
2012-10-11 17:29:37,009 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2012-10-11 17:29:37,015 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1066: Unable to open iterator for alias A
Details at logfile: /data1/home/ltang01/mf/prep/in/pig_1350001765703.log
Re: Help on running pig in local mode
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
What are you running this on?
This is really odd:
/proc/<pid>/status does not have information about swap space used(VmSwap).
D
On Fri, Oct 19, 2012 at 11:11 AM, lei tang <fi...@gmail.com> wrote:
> I checked the hadoop config and found that the staging dir specified does
> not exist in local path (only in hdfs) and neither do I have write access.
> So I changed the staging dir as below, but encounter some weird error
> message:
>
> [ltang01@stg-trgt00 ~]$ pig
> -Dmapreduce.jobtracker.staging.root.dir=~/tmp/staging -x local
> 2012-10-19 11:06:49,624 [main] INFO org.apache.pig.Main - Apache Pig
> version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
> 2012-10-19 11:06:49,625 [main] INFO org.apache.pig.Main - Logging error
> messages to: /data1/home/ltang01/pig_1350670009621.log
> 2012-10-19 11:06:50,009 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: file:///
> grunt> A = load 'toy.txt';
> grunt> dump A;
> 2012-10-19 11:06:53,485 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2012-10-19 11:06:53,660 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2012-10-19 11:06:53,693 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2012-10-19 11:06:53,693 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2012-10-19 11:06:53,886 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
> 2012-10-19 11:06:53,908 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-10-19 11:06:53,953 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2012-10-19 11:06:53,996 [main] INFO
> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 2012-10-19 11:06:54,000 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2012-10-19 11:06:54,125 [Thread-5] WARN org.apache.hadoop.mapred.JobClient
> - No job jar file set. User classes may not be found. See JobConf(Class)
> or JobConf#setJar(String).
> 2012-10-19 11:06:54,207 [Thread-5] INFO
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
> to process : 1
> 2012-10-19 11:06:54,207 [Thread-5] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths to process : 1
> 2012-10-19 11:06:54,222 [Thread-5] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths (combined) to process : 1
> 2012-10-19 11:06:54,501 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2012-10-19 11:06:55,005 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_local_0001
> 2012-10-19 11:06:55,268 [Thread-6] INFO
> org.apache.hadoop.mapreduce.util.ProcessTree - setsid exited with exit code
> 0
> 2012-10-19 11:06:55,824 [Thread-6] WARN
> org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree -
> /proc/<pid>/status does not have information about swap space used(VmSwap).
> Can not track swap usage of a task.
> 2012-10-19 11:06:55,825 [Thread-6] INFO org.apache.hadoop.mapred.Task -
> Using ResourceCalculatorPlugin :
> org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@52cab854
> 2012-10-19 11:06:55,853 [Thread-6] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader
> - Current split being processed file:/data1/home/ltang01/toy.txt:0+8
> 2012-10-19 11:06:55,914 [Thread-6] INFO org.apache.hadoop.mapred.Task -
> Task:attempt_local_0001_m_000000_0 is done. And is in the process of
> commiting
> 2012-10-19 11:06:55,921 [Thread-6] INFO
> org.apache.hadoop.mapred.LocalJobRunner -
> 2012-10-19 11:06:55,922 [Thread-6] INFO org.apache.hadoop.mapred.Task -
> Task attempt_local_0001_m_000000_0 is allowed to commit now
> 2012-10-19 11:06:55,927 [Thread-6] INFO
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output
> of task 'attempt_local_0001_m_000000_0' to
> file:/tmp/temp487011820/tmp-1248641571
> 2012-10-19 11:06:55,927 [Thread-6] INFO
> org.apache.hadoop.mapred.LocalJobRunner -
> 2012-10-19 11:06:55,928 [Thread-6] INFO org.apache.hadoop.mapred.Task -
> Task 'attempt_local_0001_m_000000_0' done.
> 2012-10-19 11:06:55,929 [Thread-6] WARN
> org.apache.hadoop.mapred.FileOutputCommitter - Output path is null in
> cleanup
> 2012-10-19 11:06:56,380 [main] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server: localhost/127.0.0.1:9001. Already tried 0
> time(s).
> 2012-10-19 11:06:57,382 [main] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server: localhost/127.0.0.1:9001. Already tried 1
> time(s).
> 2012-10-19 11:06:58,383 [main] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server: localhost/127.0.0.1:9001. Already tried 2
> time(s).
> 2012-10-19 11:06:59,385 [main] INFO org.apache.hadoop.ipc.Client -
> Retrying connect to server: localhost/127.0.0.1:9001. Already tried 3
> time(s).
>
>
> It keeps trying to connect to the local server. I have no idea what this is
> about. Can you help?
>
> - Lei
>
> On Wed, Oct 17, 2012 at 9:05 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
>> I think it's trying to find the staging directory set in your
>> configuration, not finding it, and isn't able to create it.
>> depending on your configs, that could be in different places, but
>> usually it's looking under /tmp/mapred . Check permissions there.
>>
>> D
>>
>> On Mon, Oct 15, 2012 at 4:35 PM, lei tang <fi...@gmail.com> wrote:
>> > Yes. it is there. Tried the full path as well, still not working.
>> >
>> > On Mon, Oct 15, 2012 at 3:45 PM, Prashant Kommireddi <
>> prash1784@gmail.com>wrote:
>> >
>> >> Is this file present on your local FS?
>> >> file:///data1/home/ltang01/mf/prep/in/toy.txt
>> >>
>> >> On Mon, Oct 15, 2012 at 3:43 PM, lei tang <fi...@gmail.com> wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > I'm using Pig for my daily job. Pig dose a good job in mapreduce mode
>> in
>> >> > our internal hadoop cluster. But it signals an error whenever I want
>> to
>> >> > run it in local mode. I believe this is due to some permission or
>> config
>> >> > issue. Does anybody know how I can address this problem? Any idea is
>> >> > appreciated.
>> >> >
>> >> > [ltang01@stg-trgt00 in]$ pig -x local
>> >> > 2012-10-11 17:29:25,707 [main] INFO org.apache.pig.Main - Apache Pig
>> >> > version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
>> >> > 2012-10-11 17:29:25,708 [main] INFO org.apache.pig.Main - Logging
>> error
>> >> > messages to: /somewhere/in/pig_1350001765703.log
>> >> > 2012-10-11 17:29:26,068 [main] INFO
>> >> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> >> Connecting
>> >> > to hadoop file system at: file:///
>> >> > grunt> A = load 'toy.txt';
>> >> > grunt> dump A;
>> >> > 2012-10-11 17:29:36,056 [main] INFO
>> >> > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>> >> > script: UNKNOWN
>> >> > 2012-10-11 17:29:36,218 [main] INFO
>> >> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
>> >> > File concatenation threshold: 100 optimistic? false
>> >> > 2012-10-11 17:29:36,249 [main] INFO
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> >> > - MR plan size before optimization: 1
>> >> > 2012-10-11 17:29:36,250 [main] INFO
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> >> > - MR plan size after optimization: 1
>> >> > 2012-10-11 17:29:36,382 [main] INFO
>> >> > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
>> added
>> >> > to the job
>> >> > 2012-10-11 17:29:36,403 [main] INFO
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> >> > - mapred.job.reduce.markreset.buffer.percent is not set, set to
>> default
>> >> 0.3
>> >> > 2012-10-11 17:29:36,445 [main] INFO
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> >> > - Setting up single store job
>> >> > 2012-10-11 17:29:36,486 [main] INFO
>> >> > org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics
>> with
>> >> > processName=JobTracker, sessionId=
>> >> > 2012-10-11 17:29:36,493 [main] INFO
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> >> > - 1 map-reduce job(s) waiting for submission.
>> >> > 2012-10-11 17:29:36,995 [main] INFO
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> >> > - 0% complete
>> >> > 2012-10-11 17:29:37,002 [main] INFO
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> >> > - job null has failed! Stop running all dependent jobs
>> >> > 2012-10-11 17:29:37,002 [main] INFO
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> >> > - 100% complete
>> >> > 2012-10-11 17:29:37,007 [main] ERROR
>> >> > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
>> >> > recreate exception from backend error: ENOENT: No such file or
>> directory
>> >> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
>> >> > at
>> >> >
>> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
>> >> > at
>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
>> >> > at
>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
>> >> > at java.security.AccessController.doPrivileged(Native Method)
>> >> > at javax.security.auth.Subject.doAs(Subject.java:416)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
>> >> > at
>> >> >
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
>> >> > at
>> >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
>> >> > at
>> org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>> >> > at
>> >> >
>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>> >> > at java.lang.Thread.run(Thread.java:679)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
>> >> >
>> >> > 2012-10-11 17:29:37,007 [main] ERROR
>> >> > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s)
>> failed!
>> >> > 2012-10-11 17:29:37,007 [main] INFO
>> >> > org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode.
>> Stats
>> >> > reported below may be incomplete
>> >> > 2012-10-11 17:29:37,009 [main] INFO
>> >> > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
>> >> >
>> >> > HadoopVersion PigVersion UserId StartedAt FinishedAt
>> >> > Features
>> >> > 0.20.2-dev 0.10.0 ltang01 2012-10-11 17:29:36 2012-10-11
>> >> > 17:29:37 UNKNOWN
>> >> >
>> >> > Failed!
>> >> >
>> >> > Failed Jobs:
>> >> > JobId Alias Feature Message Outputs
>> >> > N/A A MAP_ONLY Message: ENOENT: No such file or
>> >> directory
>> >> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
>> >> > at
>> >> >
>> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
>> >> > at
>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
>> >> > at
>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
>> >> > at java.security.AccessController.doPrivileged(Native Method)
>> >> > at javax.security.auth.Subject.doAs(Subject.java:416)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
>> >> > at
>> >> >
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
>> >> > at
>> >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
>> >> > at
>> org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>> >> > at
>> >> >
>> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>> >> > at java.lang.Thread.run(Thread.java:679)
>> >> > at
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
>> >> > file:/tmp/temp-916111882/tmp1833113178,
>> >> >
>> >> > Input(s):
>> >> > Failed to read data from
>> "file:///data1/home/ltang01/mf/prep/in/toy.txt"
>> >> >
>> >> > Output(s):
>> >> > Failed to produce result in "file:/tmp/temp-916111882/tmp1833113178"
>> >> >
>> >> > Job DAG:
>> >> > null
>> >> >
>> >> >
>> >> > 2012-10-11 17:29:37,009 [main] INFO
>> >> >
>> >> >
>> >>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> >> > - Failed!
>> >> > 2012-10-11 17:29:37,015 [main] ERROR org.apache.pig.tools.grunt.Grunt
>> -
>> >> > ERROR 1066: Unable to open iterator for alias A
>> >> > Details at logfile:
>> /data1/home/ltang01/mf/prep/in/pig_1350001765703.log
>> >> >
>> >>
>>
Re: Help on running pig in local mode
Posted by lei tang <fi...@gmail.com>.
I checked the hadoop config and found that the staging dir specified does
not exist in local path (only in hdfs) and neither do I have write access.
So I changed the staging dir as below, but encounter some weird error
message:
[ltang01@stg-trgt00 ~]$ pig
-Dmapreduce.jobtracker.staging.root.dir=~/tmp/staging -x local
2012-10-19 11:06:49,624 [main] INFO org.apache.pig.Main - Apache Pig
version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
2012-10-19 11:06:49,625 [main] INFO org.apache.pig.Main - Logging error
messages to: /data1/home/ltang01/pig_1350670009621.log
2012-10-19 11:06:50,009 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
to hadoop file system at: file:///
grunt> A = load 'toy.txt';
grunt> dump A;
2012-10-19 11:06:53,485 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2012-10-19 11:06:53,660 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2012-10-19 11:06:53,693 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2012-10-19 11:06:53,693 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2012-10-19 11:06:53,886 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2012-10-19 11:06:53,908 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2012-10-19 11:06:53,953 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2012-10-19 11:06:53,996 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
processName=JobTracker, sessionId=
2012-10-19 11:06:54,000 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2012-10-19 11:06:54,125 [Thread-5] WARN org.apache.hadoop.mapred.JobClient
- No job jar file set. User classes may not be found. See JobConf(Class)
or JobConf#setJar(String).
2012-10-19 11:06:54,207 [Thread-5] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2012-10-19 11:06:54,207 [Thread-5] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2012-10-19 11:06:54,222 [Thread-5] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths (combined) to process : 1
2012-10-19 11:06:54,501 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2012-10-19 11:06:55,005 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_local_0001
2012-10-19 11:06:55,268 [Thread-6] INFO
org.apache.hadoop.mapreduce.util.ProcessTree - setsid exited with exit code
0
2012-10-19 11:06:55,824 [Thread-6] WARN
org.apache.hadoop.mapreduce.util.ProcfsBasedProcessTree -
/proc/<pid>/status does not have information about swap space used(VmSwap).
Can not track swap usage of a task.
2012-10-19 11:06:55,825 [Thread-6] INFO org.apache.hadoop.mapred.Task -
Using ResourceCalculatorPlugin :
org.apache.hadoop.mapreduce.util.LinuxResourceCalculatorPlugin@52cab854
2012-10-19 11:06:55,853 [Thread-6] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader
- Current split being processed file:/data1/home/ltang01/toy.txt:0+8
2012-10-19 11:06:55,914 [Thread-6] INFO org.apache.hadoop.mapred.Task -
Task:attempt_local_0001_m_000000_0 is done. And is in the process of
commiting
2012-10-19 11:06:55,921 [Thread-6] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2012-10-19 11:06:55,922 [Thread-6] INFO org.apache.hadoop.mapred.Task -
Task attempt_local_0001_m_000000_0 is allowed to commit now
2012-10-19 11:06:55,927 [Thread-6] INFO
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Saved output
of task 'attempt_local_0001_m_000000_0' to
file:/tmp/temp487011820/tmp-1248641571
2012-10-19 11:06:55,927 [Thread-6] INFO
org.apache.hadoop.mapred.LocalJobRunner -
2012-10-19 11:06:55,928 [Thread-6] INFO org.apache.hadoop.mapred.Task -
Task 'attempt_local_0001_m_000000_0' done.
2012-10-19 11:06:55,929 [Thread-6] WARN
org.apache.hadoop.mapred.FileOutputCommitter - Output path is null in
cleanup
2012-10-19 11:06:56,380 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 0
time(s).
2012-10-19 11:06:57,382 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 1
time(s).
2012-10-19 11:06:58,383 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 2
time(s).
2012-10-19 11:06:59,385 [main] INFO org.apache.hadoop.ipc.Client -
Retrying connect to server: localhost/127.0.0.1:9001. Already tried 3
time(s).
It keeps trying to connect to the local server. I have no idea what this is
about. Can you help?
- Lei
On Wed, Oct 17, 2012 at 9:05 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> I think it's trying to find the staging directory set in your
> configuration, not finding it, and isn't able to create it.
> depending on your configs, that could be in different places, but
> usually it's looking under /tmp/mapred . Check permissions there.
>
> D
>
> On Mon, Oct 15, 2012 at 4:35 PM, lei tang <fi...@gmail.com> wrote:
> > Yes. it is there. Tried the full path as well, still not working.
> >
> > On Mon, Oct 15, 2012 at 3:45 PM, Prashant Kommireddi <
> prash1784@gmail.com>wrote:
> >
> >> Is this file present on your local FS?
> >> file:///data1/home/ltang01/mf/prep/in/toy.txt
> >>
> >> On Mon, Oct 15, 2012 at 3:43 PM, lei tang <fi...@gmail.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > I'm using Pig for my daily job. Pig dose a good job in mapreduce mode
> in
> >> > our internal hadoop cluster. But it signals an error whenever I want
> to
> >> > run it in local mode. I believe this is due to some permission or
> config
> >> > issue. Does anybody know how I can address this problem? Any idea is
> >> > appreciated.
> >> >
> >> > [ltang01@stg-trgt00 in]$ pig -x local
> >> > 2012-10-11 17:29:25,707 [main] INFO org.apache.pig.Main - Apache Pig
> >> > version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
> >> > 2012-10-11 17:29:25,708 [main] INFO org.apache.pig.Main - Logging
> error
> >> > messages to: /somewhere/in/pig_1350001765703.log
> >> > 2012-10-11 17:29:26,068 [main] INFO
> >> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> >> Connecting
> >> > to hadoop file system at: file:///
> >> > grunt> A = load 'toy.txt';
> >> > grunt> dump A;
> >> > 2012-10-11 17:29:36,056 [main] INFO
> >> > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> >> > script: UNKNOWN
> >> > 2012-10-11 17:29:36,218 [main] INFO
> >> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> >> > File concatenation threshold: 100 optimistic? false
> >> > 2012-10-11 17:29:36,249 [main] INFO
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> >> > - MR plan size before optimization: 1
> >> > 2012-10-11 17:29:36,250 [main] INFO
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> >> > - MR plan size after optimization: 1
> >> > 2012-10-11 17:29:36,382 [main] INFO
> >> > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
> added
> >> > to the job
> >> > 2012-10-11 17:29:36,403 [main] INFO
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> >> > - mapred.job.reduce.markreset.buffer.percent is not set, set to
> default
> >> 0.3
> >> > 2012-10-11 17:29:36,445 [main] INFO
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> >> > - Setting up single store job
> >> > 2012-10-11 17:29:36,486 [main] INFO
> >> > org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics
> with
> >> > processName=JobTracker, sessionId=
> >> > 2012-10-11 17:29:36,493 [main] INFO
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> >> > - 1 map-reduce job(s) waiting for submission.
> >> > 2012-10-11 17:29:36,995 [main] INFO
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> >> > - 0% complete
> >> > 2012-10-11 17:29:37,002 [main] INFO
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> >> > - job null has failed! Stop running all dependent jobs
> >> > 2012-10-11 17:29:37,002 [main] INFO
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> >> > - 100% complete
> >> > 2012-10-11 17:29:37,007 [main] ERROR
> >> > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> >> > recreate exception from backend error: ENOENT: No such file or
> directory
> >> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> >> > at
> >> >
> >> >
> >>
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
> >> > at
> >> >
> >>
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
> >> > at
> >> >
> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> >> > at
> >> >
> >> >
> >>
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
> >> > at
> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
> >> > at
> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> > at javax.security.auth.Subject.doAs(Subject.java:416)
> >> > at
> >> >
> >> >
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
> >> > at
> >> >
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
> >> > at
> >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
> >> > at
> org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> >> > at
> >> >
> >> >
> >>
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> >> > at
> >> >
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> >> > at java.lang.Thread.run(Thread.java:679)
> >> > at
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
> >> >
> >> > 2012-10-11 17:29:37,007 [main] ERROR
> >> > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s)
> failed!
> >> > 2012-10-11 17:29:37,007 [main] INFO
> >> > org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode.
> Stats
> >> > reported below may be incomplete
> >> > 2012-10-11 17:29:37,009 [main] INFO
> >> > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
> >> >
> >> > HadoopVersion PigVersion UserId StartedAt FinishedAt
> >> > Features
> >> > 0.20.2-dev 0.10.0 ltang01 2012-10-11 17:29:36 2012-10-11
> >> > 17:29:37 UNKNOWN
> >> >
> >> > Failed!
> >> >
> >> > Failed Jobs:
> >> > JobId Alias Feature Message Outputs
> >> > N/A A MAP_ONLY Message: ENOENT: No such file or
> >> directory
> >> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> >> > at
> >> >
> >> >
> >>
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
> >> > at
> >> >
> >>
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
> >> > at
> >> >
> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> >> > at
> >> >
> >> >
> >>
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
> >> > at
> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
> >> > at
> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> > at javax.security.auth.Subject.doAs(Subject.java:416)
> >> > at
> >> >
> >> >
> >>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
> >> > at
> >> >
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
> >> > at
> >> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
> >> > at
> org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> >> > at
> >> >
> >> >
> >>
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> >> > at
> >> >
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> >> > at java.lang.Thread.run(Thread.java:679)
> >> > at
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
> >> > file:/tmp/temp-916111882/tmp1833113178,
> >> >
> >> > Input(s):
> >> > Failed to read data from
> "file:///data1/home/ltang01/mf/prep/in/toy.txt"
> >> >
> >> > Output(s):
> >> > Failed to produce result in "file:/tmp/temp-916111882/tmp1833113178"
> >> >
> >> > Job DAG:
> >> > null
> >> >
> >> >
> >> > 2012-10-11 17:29:37,009 [main] INFO
> >> >
> >> >
> >>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> >> > - Failed!
> >> > 2012-10-11 17:29:37,015 [main] ERROR org.apache.pig.tools.grunt.Grunt
> -
> >> > ERROR 1066: Unable to open iterator for alias A
> >> > Details at logfile:
> /data1/home/ltang01/mf/prep/in/pig_1350001765703.log
> >> >
> >>
>
Re: Help on running pig in local mode
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
I think it's trying to find the staging directory set in your
configuration, not finding it, and isn't able to create it.
depending on your configs, that could be in different places, but
usually it's looking under /tmp/mapred . Check permissions there.
D
On Mon, Oct 15, 2012 at 4:35 PM, lei tang <fi...@gmail.com> wrote:
> Yes. it is there. Tried the full path as well, still not working.
>
> On Mon, Oct 15, 2012 at 3:45 PM, Prashant Kommireddi <pr...@gmail.com>wrote:
>
>> Is this file present on your local FS?
>> file:///data1/home/ltang01/mf/prep/in/toy.txt
>>
>> On Mon, Oct 15, 2012 at 3:43 PM, lei tang <fi...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I'm using Pig for my daily job. Pig dose a good job in mapreduce mode in
>> > our internal hadoop cluster. But it signals an error whenever I want to
>> > run it in local mode. I believe this is due to some permission or config
>> > issue. Does anybody know how I can address this problem? Any idea is
>> > appreciated.
>> >
>> > [ltang01@stg-trgt00 in]$ pig -x local
>> > 2012-10-11 17:29:25,707 [main] INFO org.apache.pig.Main - Apache Pig
>> > version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
>> > 2012-10-11 17:29:25,708 [main] INFO org.apache.pig.Main - Logging error
>> > messages to: /somewhere/in/pig_1350001765703.log
>> > 2012-10-11 17:29:26,068 [main] INFO
>> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> Connecting
>> > to hadoop file system at: file:///
>> > grunt> A = load 'toy.txt';
>> > grunt> dump A;
>> > 2012-10-11 17:29:36,056 [main] INFO
>> > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
>> > script: UNKNOWN
>> > 2012-10-11 17:29:36,218 [main] INFO
>> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
>> > File concatenation threshold: 100 optimistic? false
>> > 2012-10-11 17:29:36,249 [main] INFO
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> > - MR plan size before optimization: 1
>> > 2012-10-11 17:29:36,250 [main] INFO
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> > - MR plan size after optimization: 1
>> > 2012-10-11 17:29:36,382 [main] INFO
>> > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
>> > to the job
>> > 2012-10-11 17:29:36,403 [main] INFO
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> > - mapred.job.reduce.markreset.buffer.percent is not set, set to default
>> 0.3
>> > 2012-10-11 17:29:36,445 [main] INFO
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> > - Setting up single store job
>> > 2012-10-11 17:29:36,486 [main] INFO
>> > org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
>> > processName=JobTracker, sessionId=
>> > 2012-10-11 17:29:36,493 [main] INFO
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> > - 1 map-reduce job(s) waiting for submission.
>> > 2012-10-11 17:29:36,995 [main] INFO
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> > - 0% complete
>> > 2012-10-11 17:29:37,002 [main] INFO
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> > - job null has failed! Stop running all dependent jobs
>> > 2012-10-11 17:29:37,002 [main] INFO
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> > - 100% complete
>> > 2012-10-11 17:29:37,007 [main] ERROR
>> > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
>> > recreate exception from backend error: ENOENT: No such file or directory
>> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> > at
>> >
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
>> > at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
>> > at
>> > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>> > at
>> >
>> >
>> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:416)
>> > at
>> >
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
>> > at
>> > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
>> > at
>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
>> > at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> > at
>> >
>> >
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>> > at
>> > org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>> > at java.lang.Thread.run(Thread.java:679)
>> > at
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
>> >
>> > 2012-10-11 17:29:37,007 [main] ERROR
>> > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
>> > 2012-10-11 17:29:37,007 [main] INFO
>> > org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats
>> > reported below may be incomplete
>> > 2012-10-11 17:29:37,009 [main] INFO
>> > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
>> >
>> > HadoopVersion PigVersion UserId StartedAt FinishedAt
>> > Features
>> > 0.20.2-dev 0.10.0 ltang01 2012-10-11 17:29:36 2012-10-11
>> > 17:29:37 UNKNOWN
>> >
>> > Failed!
>> >
>> > Failed Jobs:
>> > JobId Alias Feature Message Outputs
>> > N/A A MAP_ONLY Message: ENOENT: No such file or
>> directory
>> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
>> > at
>> >
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
>> > at
>> >
>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
>> > at
>> > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>> > at
>> >
>> >
>> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:416)
>> > at
>> >
>> >
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
>> > at
>> > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
>> > at
>> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
>> > at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
>> > at
>> >
>> >
>> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
>> > at
>> > org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
>> > at java.lang.Thread.run(Thread.java:679)
>> > at
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
>> > file:/tmp/temp-916111882/tmp1833113178,
>> >
>> > Input(s):
>> > Failed to read data from "file:///data1/home/ltang01/mf/prep/in/toy.txt"
>> >
>> > Output(s):
>> > Failed to produce result in "file:/tmp/temp-916111882/tmp1833113178"
>> >
>> > Job DAG:
>> > null
>> >
>> >
>> > 2012-10-11 17:29:37,009 [main] INFO
>> >
>> >
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> > - Failed!
>> > 2012-10-11 17:29:37,015 [main] ERROR org.apache.pig.tools.grunt.Grunt -
>> > ERROR 1066: Unable to open iterator for alias A
>> > Details at logfile: /data1/home/ltang01/mf/prep/in/pig_1350001765703.log
>> >
>>
Re: Help on running pig in local mode
Posted by lei tang <fi...@gmail.com>.
Yes. it is there. Tried the full path as well, still not working.
On Mon, Oct 15, 2012 at 3:45 PM, Prashant Kommireddi <pr...@gmail.com>wrote:
> Is this file present on your local FS?
> file:///data1/home/ltang01/mf/prep/in/toy.txt
>
> On Mon, Oct 15, 2012 at 3:43 PM, lei tang <fi...@gmail.com> wrote:
>
> > Hi,
> >
> > I'm using Pig for my daily job. Pig dose a good job in mapreduce mode in
> > our internal hadoop cluster. But it signals an error whenever I want to
> > run it in local mode. I believe this is due to some permission or config
> > issue. Does anybody know how I can address this problem? Any idea is
> > appreciated.
> >
> > [ltang01@stg-trgt00 in]$ pig -x local
> > 2012-10-11 17:29:25,707 [main] INFO org.apache.pig.Main - Apache Pig
> > version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
> > 2012-10-11 17:29:25,708 [main] INFO org.apache.pig.Main - Logging error
> > messages to: /somewhere/in/pig_1350001765703.log
> > 2012-10-11 17:29:26,068 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting
> > to hadoop file system at: file:///
> > grunt> A = load 'toy.txt';
> > grunt> dump A;
> > 2012-10-11 17:29:36,056 [main] INFO
> > org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> > script: UNKNOWN
> > 2012-10-11 17:29:36,218 [main] INFO
> > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> > File concatenation threshold: 100 optimistic? false
> > 2012-10-11 17:29:36,249 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> > - MR plan size before optimization: 1
> > 2012-10-11 17:29:36,250 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> > - MR plan size after optimization: 1
> > 2012-10-11 17:29:36,382 [main] INFO
> > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> > to the job
> > 2012-10-11 17:29:36,403 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> > - mapred.job.reduce.markreset.buffer.percent is not set, set to default
> 0.3
> > 2012-10-11 17:29:36,445 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> > - Setting up single store job
> > 2012-10-11 17:29:36,486 [main] INFO
> > org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
> > processName=JobTracker, sessionId=
> > 2012-10-11 17:29:36,493 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 1 map-reduce job(s) waiting for submission.
> > 2012-10-11 17:29:36,995 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 0% complete
> > 2012-10-11 17:29:37,002 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - job null has failed! Stop running all dependent jobs
> > 2012-10-11 17:29:37,002 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - 100% complete
> > 2012-10-11 17:29:37,007 [main] ERROR
> > org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> > recreate exception from backend error: ENOENT: No such file or directory
> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> > at
> >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
> > at
> >
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
> > at
> > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> > at
> >
> >
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:416)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
> > at
> > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
> > at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
> > at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> > at
> >
> >
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> > at
> > org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> > at java.lang.Thread.run(Thread.java:679)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
> >
> > 2012-10-11 17:29:37,007 [main] ERROR
> > org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> > 2012-10-11 17:29:37,007 [main] INFO
> > org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats
> > reported below may be incomplete
> > 2012-10-11 17:29:37,009 [main] INFO
> > org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
> >
> > HadoopVersion PigVersion UserId StartedAt FinishedAt
> > Features
> > 0.20.2-dev 0.10.0 ltang01 2012-10-11 17:29:36 2012-10-11
> > 17:29:37 UNKNOWN
> >
> > Failed!
> >
> > Failed Jobs:
> > JobId Alias Feature Message Outputs
> > N/A A MAP_ONLY Message: ENOENT: No such file or
> directory
> > at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> > at
> >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
> > at
> >
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
> > at
> > org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> > at
> >
> >
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:416)
> > at
> >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
> > at
> > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
> > at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
> > at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> > at
> >
> >
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> > at
> > org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> > at java.lang.Thread.run(Thread.java:679)
> > at
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
> > file:/tmp/temp-916111882/tmp1833113178,
> >
> > Input(s):
> > Failed to read data from "file:///data1/home/ltang01/mf/prep/in/toy.txt"
> >
> > Output(s):
> > Failed to produce result in "file:/tmp/temp-916111882/tmp1833113178"
> >
> > Job DAG:
> > null
> >
> >
> > 2012-10-11 17:29:37,009 [main] INFO
> >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> > - Failed!
> > 2012-10-11 17:29:37,015 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> > ERROR 1066: Unable to open iterator for alias A
> > Details at logfile: /data1/home/ltang01/mf/prep/in/pig_1350001765703.log
> >
>
Re: Help on running pig in local mode
Posted by Prashant Kommireddi <pr...@gmail.com>.
Is this file present on your local FS?
file:///data1/home/ltang01/mf/prep/in/toy.txt
On Mon, Oct 15, 2012 at 3:43 PM, lei tang <fi...@gmail.com> wrote:
> Hi,
>
> I'm using Pig for my daily job. Pig dose a good job in mapreduce mode in
> our internal hadoop cluster. But it signals an error whenever I want to
> run it in local mode. I believe this is due to some permission or config
> issue. Does anybody know how I can address this problem? Any idea is
> appreciated.
>
> [ltang01@stg-trgt00 in]$ pig -x local
> 2012-10-11 17:29:25,707 [main] INFO org.apache.pig.Main - Apache Pig
> version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12
> 2012-10-11 17:29:25,708 [main] INFO org.apache.pig.Main - Logging error
> messages to: /somewhere/in/pig_1350001765703.log
> 2012-10-11 17:29:26,068 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting
> to hadoop file system at: file:///
> grunt> A = load 'toy.txt';
> grunt> dump A;
> 2012-10-11 17:29:36,056 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2012-10-11 17:29:36,218 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
> File concatenation threshold: 100 optimistic? false
> 2012-10-11 17:29:36,249 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2012-10-11 17:29:36,250 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2012-10-11 17:29:36,382 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
> to the job
> 2012-10-11 17:29:36,403 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2012-10-11 17:29:36,445 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2012-10-11 17:29:36,486 [main] INFO
> org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> 2012-10-11 17:29:36,493 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2012-10-11 17:29:36,995 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2012-10-11 17:29:37,002 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job null has failed! Stop running all dependent jobs
> 2012-10-11 17:29:37,002 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> 2012-10-11 17:29:37,007 [main] ERROR
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> recreate exception from backend error: ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at
>
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
> at
> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> at
>
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
>
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:679)
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
>
> 2012-10-11 17:29:37,007 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> 2012-10-11 17:29:37,007 [main] INFO
> org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats
> reported below may be incomplete
> 2012-10-11 17:29:37,009 [main] INFO
> org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt
> Features
> 0.20.2-dev 0.10.0 ltang01 2012-10-11 17:29:36 2012-10-11
> 17:29:37 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
> N/A A MAP_ONLY Message: ENOENT: No such file or directory
> at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
> at
>
> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:513)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:325)
> at
> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
> at
>
> org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:126)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:873)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:867)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1109)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:867)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:841)
> at org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378)
> at
>
> org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
> at
> org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
> at java.lang.Thread.run(Thread.java:679)
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
> file:/tmp/temp-916111882/tmp1833113178,
>
> Input(s):
> Failed to read data from "file:///data1/home/ltang01/mf/prep/in/toy.txt"
>
> Output(s):
> Failed to produce result in "file:/tmp/temp-916111882/tmp1833113178"
>
> Job DAG:
> null
>
>
> 2012-10-11 17:29:37,009 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Failed!
> 2012-10-11 17:29:37,015 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 1066: Unable to open iterator for alias A
> Details at logfile: /data1/home/ltang01/mf/prep/in/pig_1350001765703.log
>