WebHCat - Pig job runs in local mode

[Rajkumar S <ra...@gmail.com>]

Hello Experts,

I am using WebHCat REST interface to launch Pig jobs.

When I invoke Pig scripts using Curl, WebHCat launches a
TempletonControllerJob which has one map task as expected. This job in turn
is NOT launching the actual job from the REST API call. In the Resource
Manager page, I only see the controller job (parent job), but PigLatin job
(child job) could not be seen.

However, the controller job is completed and the status getting succeeded,
while looking inside this parent job, the actual Pig scripts are getting
executed in the local instance. I am expecting the child jobs to be
executed as a separate MR job in Hadoop cluster.

Why is the controller job not launching a separate MR job for the pig
scripts? Am I missing any configuration changes?

Note : The pig scripts are executing fine in both Local and Mapreduce mode
when I run it from command line.

Templeton Controller Job - Map task logs

HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
2.6.0   0.15.0  hduser  2016-04-16 20:56:54 2016-04-16 20:57:10 LIMIT

Success!

Job Stats (time in seconds):
JobId   Maps    Reduces MaxMapTime  MinMapTime  AvgMapTime
MedianMapTime   MaxReduceTime   MinReduceTime   AvgReduceTime
MedianReducetime    Alias   Feature Outputs
job_local1288160920_0002    1   1   n/a n/a n/a n/a n/a n/a n/a n/a
     /tempo,
job_local384530329_0001 1   1   n/a n/a n/a n/a n/a n/a n/a n/a a,b

Input(s):
Successfully read 1 records (10500218 bytes) from: "/pigouttest"

Output(s):
Successfully stored 1 records (20999942 bytes) in: "/tempo"

Counters:
Total records written : 1
Total bytes written : 20999942
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_local384530329_0001 ->  job_local1288160920_0002,
job_local1288160920_0002


2016-04-16 20:57:10,759 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM
Metrics with processName=JobTracker, sessionId= - already initialized
2016-04-16 20:57:10,766 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM
Metrics with processName=JobTracker, sessionId= - already initialized
2016-04-16 20:57:10,768 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM
Metrics with processName=JobTracker, sessionId= - already initialized
2016-04-16 20:57:10,827 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM
Metrics with processName=JobTracker, sessionId= - already initialized
2016-04-16 20:57:10,829 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM
Metrics with processName=JobTracker, sessionId= - already initialized
2016-04-16 20:57:10,830 [main] INFO
org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM
Metrics with processName=JobTracker, sessionId= - already initialized
2016-04-16 20:57:10,839 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Success!
2016-04-16 20:57:10,886 [main] INFO  org.apache.pig.Main - Pig script
completed in 22 seconds and 866 milliseconds (22866 ms)


Thanks!