[jira] [Created] (PIG-3651) Cannot Launch Pig Job on Hadoop (Infinite Loop?)

["Michael Knapp (JIRA)" <ji...@apache.org>]

Michael Knapp created PIG-3651:
----------------------------------

             Summary: Cannot Launch Pig Job on Hadoop (Infinite Loop?)
                 Key: PIG-3651
                 URL: https://issues.apache.org/jira/browse/PIG-3651
             Project: Pig
          Issue Type: Bug
    Affects Versions: 0.11
            Reporter: Michael Knapp


There must be an infinite loop in the pig code, I wrote a very simple pig script, all it has is one load statement, and the next line dumps it to the console.  When I run that pig gets to the MapReduceLauncher and just hangs, never making any progress.

I'm running CDH4.5 on a CentOS 6.4 VM, all installed from Cloudera's yum repo. It is configured to all run in pseudo-distributed mode. Everything is running as a service and appears to be configured correctly (thank heaven!)

When I run the pig script on the command line I get a whole bunch of log statements and it looks like it is running, but once it starts, it never makes any progress, no matter how long I wait. These are the last couple lines:

2014-01-05 15:10:41,113 [JobControl] INFO  org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1388936205793_0006
2014-01-05 15:10:41,511 [JobControl] INFO  org.apache.hadoop.yarn.client.YarnClientImpl - Submitted application application_1388936205793_0006 to ResourceManager at /0.0.0.0:8032
2014-01-05 15:10:41,564 [JobControl] INFO  org.apache.hadoop.mapreduce.Job - The url to track the job: http://localhost:8088/proxy/application_1388936205793_0006/
2014-01-05 15:10:41,653 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete

I modified the pig script to point to my local filesystem file, and ran the pig script in local mode, and the job finished successfully in seconds. The local copy of the file is identical to the one hdfs has. 

I tried adding all my hadoop jars to the classpath, both in my pig script (using REGISTER) and in the pig bash script, it made no difference.

I ran jstack on the process and saw this:

"main" prio=10 tid=0x00007ff468018000 nid=0x2bbf in Object.wait() [0x00007ff46f14c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000fba8c6e0> (a org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1)
	at java.lang.Thread.join(Thread.java:1288)
	- locked <0x00000000fba8c6e0> (a org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:282)
	at org.apache.pig.PigServer.launchPlan(PigServer.java:1266)
	at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1251)
	at org.apache.pig.PigServer.storeEx(PigServer.java:933)
	at org.apache.pig.PigServer.store(PigServer.java:900)
	at org.apache.pig.PigServer.openIterator(PigServer.java:813)
	at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
	at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
	at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
	at org.apache.pig.Main.run(Main.java:604)
	at org.apache.pig.Main.main(Main.java:157)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

I think the MayReduceLauncher classes launchPig method must have an infinite loop in it somehow.  I looked in the source, that method is hundreds of lines (you really should split it up).  It does contain a while loop, I suspect that it is waiting for some other thread to start.

So you probably need to do a couple things:
1.  Split up that method, make it modular
2.  Have some kind of timeout, something that checks that it's making progress 
3.  Figure out why it's hung (I really can't figure it out, I followed all of CDH4.5's instructions very carefully)
4.  Whatever configuration setting I'm missing, probably needs to be documented a little beter.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)