You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Raman Yakkala <ry...@gmail.com> on 2010/08/21 01:22:03 UTC

Help with PIG 0.7 and JOINs

Hi Guys,



I am trying to do join tow data sets and the jobs are failing. There are
some warnings reported and I am not good at understanding them. I am seeking
your help in adjusting any parameters to the job so that the job might
succeed.

Here is the grunt shell output:

grunt> STORE fjid INTO '/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks'
USING PigStorage();
2010-08-20 14:43:47,056 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for
odl: $1, $2, $4, $5, $6, $8
2010-08-20 14:43:47,056 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for odl
2010-08-20 14:43:47,059 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for
oil: $2, $3, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18,
$19, $20, $21, $22, $23, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35,
$36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $51,
$53, $55
2010-08-20 14:43:47,059 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for oil
2010-08-20 14:43:47,070 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for qif
2010-08-20 14:43:47,070 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for qif
2010-08-20 14:43:47,221 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name:
Store(hdfs://srwaishdc1nn0001/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks:PigStorage)
- 1-799 Operator Key: 1-799)
*2010-08-20 14:43:47,269 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 3 time(s).
2010-08-20 14:43:47,274 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning MULTI_LEAF_MAP 1 time(s).
*2010-08-20 14:43:47,284 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-20 14:43:47,292 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 4
2010-08-20 14:43:47,292 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 4
2010-08-20 14:43:47,333 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 14:43:48,645 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 14:43:48,715 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 14:43:49,884 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 14:43:49,896 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 2 map-reduce job(s) waiting for submission.
*2010-08-20 14:43:49,902 [Thread-13] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
*2010-08-20 14:43:50,398 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2010-08-20 14:43:51,082 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 5800
2010-08-20 14:44:02,525 [Thread-13] WARN  org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-20 14:44:02,871 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 950
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201007221306_7119
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201007221306_7120
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
2010-08-20 14:48:41,613 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 20% complete
2010-08-20 14:52:11,487 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 31% complete
2010-08-20 14:56:47,404 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 38% complete
2010-08-20 14:58:45,888 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 42% complete
2010-08-20 15:09:23,195 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 15:09:24,339 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 15:09:24,344 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2010-08-20 15:09:24,360 [Thread-31] WARN  org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-20 15:09:24,630 [Thread-31] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2010-08-20 15:09:24,678 [Thread-31] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 200
2010-08-20 15:09:24,847 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
*2010-08-20 15:09:24,847 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map reduce job(s) failed!
*2010-08-20 15:09:26,620 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Some jobs have failed! Stop running all dependent jobs
grunt>

Re: Help with PIG 0.7 and JOINs

Posted by Raman Yakkala <ry...@gmail.com>.
Hi All,



It could be an intermittent issue with the cluster, and the same PIG scripts
ran successfully at a later time.



Thanks,

Raman

On Fri, Aug 20, 2010 at 5:22 PM, Raman Yakkala <ry...@gmail.com> wrote:

> Thanks Thejas. Here are the details of the jobs. Not sure about the
> location of the log files ...
>
> According to the job detail page, it says the job completed successfully...
>  Hadoop job_201007221306_7119 on srwaishdc1jn0001<https://athena/proxy/srwaishdc1jn0001/50030/jobtracker.jsp>
> *User:* ryakkala
> *Job Name:* PigLatin:DefaultJobName
> *Job File:*
> hdfs://srwaishdc1nn0001/tmp/hadoop-hadoop/mapred/system/job_201007221306_7119/job.xml<https://athena/proxy/srwaishdc1jn0001/50030/jobconf.jsp?jobid=job_201007221306_7119>
> *Job Setup:* Successful<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=setup&pagenum=1&state=completed>
> *Status:* Succeeded
> *Started at:* Fri Aug 20 14:43:55 GMT-07:00 2010
> *Finished at:* Fri Aug 20 15:09:18 GMT-07:00 2010
> *Finished in:* 25mins, 22sec
> *Job Cleanup:* Successful<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=cleanup&pagenum=1&state=completed>
> ------------------------------
>   Kind % Complete Num Tasks Pending Running Complete Killed Failed/Killed
> Task Attempts<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7119>
> map<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=map&pagenum=1> 100.00%
>   14263 0 0 14263<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=map&pagenum=1&state=completed>
> 0 0 / 442<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7119&kind=map&cause=killed>
> reduce<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=reduce&pagenum=1> 100.00%
>   2000 0 0 2000<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=reduce&pagenum=1&state=completed>
> 0 0 / 301<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7119&kind=reduce&cause=killed>
>
>
> Counter Map Reduce Total Job Counters Launched reduce tasks 0 0 2,301 Rack-local
> map tasks 0 0 218 Launched map tasks 0 0 14,705 Data-local map tasks 0 0
> 13,585 FileSystemCounters FILE_BYTES_READ 9,680,966,524,451
> 2,319,902,502,764 12,000,869,027,215 HDFS_BYTES_READ 1,614,305,647,406 0
> 1,614,305,647,406 FILE_BYTES_WRITTEN 4,813,545,935,138 2,319,902,502,764
> 7,133,448,437,902 HDFS_BYTES_WRITTEN 0 5,268,568,327,064 5,268,568,327,064 Map-Reduce
> Framework Reduce input groups 0 298,706,692 298,706,692 Combine output
> records 0 0 0 Map input records 398,774,974 0 398,774,974 Reduce shuffle
> bytes 0 2,413,604,558,982 2,413,604,558,982 Reduce output records 0
> 298,706,692 298,706,692 Spilled Records 790,418,867 398,774,974
> 1,189,193,841 Map output bytes 6,899,498,353,361 0 6,899,498,353,361 Map
> output records 398,774,974 0 398,774,974 Combine input records 0 0 0 Reduce
> input records 0 398,774,974 398,774,974
> ------------------------------
> Map Completion Graph - close
> <https://athena/proxy/srwaishdc1jn0001/50030/jobdetails.jsp?jobid=job_201007221306_7119&refresh=0&map.graph=off>
> ------------------------------
> Reduce Completion Graph - close
> <https://athena/proxy/srwaishdc1jn0001/50030/jobdetails.jsp?jobid=job_201007221306_7119&refresh=0&reduce.graph=off>
> ------------------------------
>   ------------------------------
> ------------------------------
> Go back to JobTracker<https://athena/proxy/srwaishdc1jn0001/50030/jobtracker.jsp>
> ------------------------------
> Cloudera's Distribution for Hadoop<https://athena/proxy/www.cloudera.com/80/hadoop/>,
> 2010.
>
> *User:* ryakkala
> *Job Name:* PigLatin:DefaultJobName
> *Job File:*
> hdfs://srwaishdc1nn0001/tmp/hadoop-hadoop/mapred/system/job_201007221306_7120/job.xml<https://athena/proxy/srwaishdc1jn0001/50030/jobconf.jsp?jobid=job_201007221306_7120>
> *Job Setup:* Successful<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=setup&pagenum=1&state=completed>
> *Status:* Succeeded
> *Started at:* Fri Aug 20 14:44:03 GMT-07:00 2010
> *Finished at:* Fri Aug 20 14:50:05 GMT-07:00 2010
> *Finished in:* 6mins, 1sec
> *Job Cleanup:* Successful<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=cleanup&pagenum=1&state=completed>
> ------------------------------
>   Kind % Complete Num Tasks Pending Running Complete Killed Failed/Killed
> Task Attempts<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7120>
> map<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=map&pagenum=1> 100.00%
>   950 0 0 950<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=map&pagenum=1&state=completed>
> 0 0 / 153<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7120&kind=map&cause=killed>
> reduce<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=reduce&pagenum=1> 100.00%
>   2000 0 0 2000<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=reduce&pagenum=1&state=completed>
> 0 0 / 148<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7120&kind=reduce&cause=killed>
>
>
> Counter Map Reduce Total Job Counters Launched reduce tasks 0 0 2,148 Rack-local
> map tasks 0 0 244 Launched map tasks 0 0 1,103 Data-local map tasks 0 0
> 772 FileSystemCounters FILE_BYTES_READ 0 1,609,717,850 1,609,717,850
> HDFS_BYTES_READ 64,557,004,337 0 64,557,004,337 FILE_BYTES_WRITTEN
> 2,685,517,089 1,609,717,850 4,295,234,939 HDFS_BYTES_WRITTEN 0 273,487,824
> 273,487,824 Map-Reduce Framework Reduce input groups 0 48,500,194
> 48,500,194 Combine output records 0 0 0 Map input records 1,879,762,784 0
> 1,879,762,784 Reduce shuffle bytes 0 2,638,520,072 2,638,520,072 Reduce
> output records 0 11,395,326 11,395,326 Spilled Records 77,375,834
> 77,375,834 154,751,668 Map output bytes 8,279,214,238 0 8,279,214,238 Map
> output records 77,375,834 0 77,375,834 Combine input records 0 0 0 Reduce
> input records 0 77,375,834 77,375,834
> ------------------------------
> Map Completion Graph - close
> <https://athena/proxy/srwaishdc1jn0001/50030/jobdetails.jsp?jobid=job_201007221306_7120&refresh=0&map.graph=off>
> ------------------------------
> Reduce Completion Graph - close
> <https://athena/proxy/srwaishdc1jn0001/50030/jobdetails.jsp?jobid=job_201007221306_7120&refresh=0&reduce.graph=off>
> ------------------------------
>   ------------------------------
> ------------------------------
> Go back to JobTracker<https://athena/proxy/srwaishdc1jn0001/50030/jobtracker.jsp>
> ------------------------------
> Cloudera's Distribution for Hadoop<https://athena/proxy/www.cloudera.com/80/hadoop/>,
> 2010.
>
>   On Fri, Aug 20, 2010 at 4:40 PM, Thejas M Nair <te...@yahoo-inc.com>wrote:
>
>> You would find the log for MR jobs that failed either at -
>>
>> http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
>> Or
>> http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
>>
>> You can forward the error message if you need further help with this.
>>
>> -Thejas
>>
>>
>>
>>
>> On 8/20/10 4:22 PM, "Raman Yakkala" <ry...@gmail.com> wrote:
>>
>>   Hi Guys,
>>
>>
>>
>> I am trying to do join tow data sets and the jobs are failing. There are
>> some warnings reported and I am not good at understanding them. I am
>> seeking
>> your help in adjusting any parameters to the job so that the job might
>> succeed.
>>
>> Here is the grunt shell output:
>>
>> grunt> STORE fjid INTO '/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks'
>> USING PigStorage();
>> 2010-08-20 14:43:47,056 [main] INFO
>> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned
>> for
>> odl: $1, $2, $4, $5, $6, $8
>> 2010-08-20 14:43:47,056 [main] INFO
>> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys
>> pruned
>> for odl
>> 2010-08-20 14:43:47,059 [main] INFO
>> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned
>> for
>> oil: $2, $3, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18,
>> $19, $20, $21, $22, $23, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35,
>> $36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $51,
>> $53, $55
>> 2010-08-20 14:43:47,059 [main] INFO
>> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys
>> pruned
>> for oil
>> 2010-08-20 14:43:47,070 [main] INFO
>> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
>> for qif
>> 2010-08-20 14:43:47,070 [main] INFO
>> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys
>> pruned
>> for qif
>> 2010-08-20 14:43:47,221 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name:
>>
>> Store(hdfs://srwaishdc1nn0001/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks:PigStorage)
>> - 1-799 Operator Key: 1-799)
>> *2010-08-20 14:43:47,269 [main] WARN
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 3 time(s).
>> 2010-08-20 14:43:47,274 [main] WARN
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - Encountered Warning MULTI_LEAF_MAP 1 time(s).
>> *2010-08-20 14:43:47,284 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
>> - Rewrite: POPackage->POForEach to POJoinPackage
>> 2010-08-20 14:43:47,292 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> - MR plan size before optimization: 4
>> 2010-08-20 14:43:47,292 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> - MR plan size after optimization: 4
>> 2010-08-20 14:43:47,333 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - mapred.job.reduce.markreset.buffer.percent is not set, set to default
>> 0.3
>> 2010-08-20 14:43:48,645 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - Setting up single store job
>> 2010-08-20 14:43:48,715 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - mapred.job.reduce.markreset.buffer.percent is not set, set to default
>> 0.3
>> 2010-08-20 14:43:49,884 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - Setting up single store job
>> 2010-08-20 14:43:49,896 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 2 map-reduce job(s) waiting for submission.
>> *2010-08-20 14:43:49,902 [Thread-13] WARN
>> org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
>> the arguments. Applications should implement Tool for the same.
>> *2010-08-20 14:43:50,398 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 0% complete
>> 2010-08-20 14:43:51,082 [Thread-13] INFO
>> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
>> input
>> paths to process : 5800
>> 2010-08-20 14:44:02,525 [Thread-13] WARN
>>  org.apache.hadoop.mapred.JobClient
>> - Use GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 2010-08-20 14:44:02,871 [Thread-13] INFO
>> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
>> input
>> paths to process : 950
>> 2010-08-20 14:44:10,835 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - HadoopJobId: job_201007221306_7119
>> 2010-08-20 14:44:10,835 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - More information at:
>> http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
>> 2010-08-20 14:44:10,835 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - HadoopJobId: job_201007221306_7120
>> 2010-08-20 14:44:10,835 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - More information at:
>> http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
>> 2010-08-20 14:48:41,613 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 20% complete
>> 2010-08-20 14:52:11,487 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 31% complete
>> 2010-08-20 14:56:47,404 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 38% complete
>> 2010-08-20 14:58:45,888 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 42% complete
>> 2010-08-20 15:09:23,195 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - mapred.job.reduce.markreset.buffer.percent is not set, set to default
>> 0.3
>> 2010-08-20 15:09:24,339 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - Setting up single store job
>> 2010-08-20 15:09:24,344 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 1 map-reduce job(s) waiting for submission.
>> 2010-08-20 15:09:24,360 [Thread-31] WARN
>>  org.apache.hadoop.mapred.JobClient
>> - Use GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 2010-08-20 15:09:24,630 [Thread-31] INFO
>> org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
>> to
>> process : 1
>> 2010-08-20 15:09:24,678 [Thread-31] INFO
>> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total
>> input
>> paths to process : 200
>> 2010-08-20 15:09:24,847 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 100% complete
>> *2010-08-20 15:09:24,847 [main] ERROR
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 1 map reduce job(s) failed!
>> *2010-08-20 15:09:26,620 [main] INFO
>>
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - Some jobs have failed! Stop running all dependent jobs
>> grunt>
>>
>>
>>
>

Re: Help with PIG 0.7 and JOINs

Posted by Raman Yakkala <ry...@gmail.com>.
Thanks Thejas. Here are the details of the jobs. Not sure about the location
of the log files ...

According to the job detail page, it says the job completed successfully...
 Hadoop job_201007221306_7119 on
srwaishdc1jn0001<https://athena/proxy/srwaishdc1jn0001/50030/jobtracker.jsp>
*User:* ryakkala
*Job Name:* PigLatin:DefaultJobName
*Job File:*
hdfs://srwaishdc1nn0001/tmp/hadoop-hadoop/mapred/system/job_201007221306_7119/job.xml<https://athena/proxy/srwaishdc1jn0001/50030/jobconf.jsp?jobid=job_201007221306_7119>
*Job Setup:* Successful<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=setup&pagenum=1&state=completed>
*Status:* Succeeded
*Started at:* Fri Aug 20 14:43:55 GMT-07:00 2010
*Finished at:* Fri Aug 20 15:09:18 GMT-07:00 2010
*Finished in:* 25mins, 22sec
*Job Cleanup:* Successful<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=cleanup&pagenum=1&state=completed>
------------------------------
  Kind % Complete Num Tasks Pending Running Complete Killed Failed/Killed
Task Attempts<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7119>
map<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=map&pagenum=1>
100.00%
  14263 0 0 14263<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=map&pagenum=1&state=completed>
0 0 / 442<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7119&kind=map&cause=killed>
reduce<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=reduce&pagenum=1>
100.00%
  2000 0 0 2000<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7119&type=reduce&pagenum=1&state=completed>
0 0 / 301<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7119&kind=reduce&cause=killed>


Counter Map Reduce Total Job Counters Launched reduce tasks 0 0 2,301
Rack-local
map tasks 0 0 218 Launched map tasks 0 0 14,705 Data-local map tasks 0 0
13,585 FileSystemCounters FILE_BYTES_READ 9,680,966,524,451
2,319,902,502,764 12,000,869,027,215 HDFS_BYTES_READ 1,614,305,647,406 0
1,614,305,647,406 FILE_BYTES_WRITTEN 4,813,545,935,138 2,319,902,502,764
7,133,448,437,902 HDFS_BYTES_WRITTEN 0 5,268,568,327,064
5,268,568,327,064 Map-Reduce
Framework Reduce input groups 0 298,706,692 298,706,692 Combine output
records 0 0 0 Map input records 398,774,974 0 398,774,974 Reduce shuffle
bytes 0 2,413,604,558,982 2,413,604,558,982 Reduce output records 0
298,706,692 298,706,692 Spilled Records 790,418,867 398,774,974
1,189,193,841 Map output bytes 6,899,498,353,361 0 6,899,498,353,361 Map
output records 398,774,974 0 398,774,974 Combine input records 0 0 0 Reduce
input records 0 398,774,974 398,774,974
------------------------------
Map Completion Graph - close
<https://athena/proxy/srwaishdc1jn0001/50030/jobdetails.jsp?jobid=job_201007221306_7119&refresh=0&map.graph=off>
------------------------------
Reduce Completion Graph - close
<https://athena/proxy/srwaishdc1jn0001/50030/jobdetails.jsp?jobid=job_201007221306_7119&refresh=0&reduce.graph=off>
------------------------------
  ------------------------------
------------------------------
Go back to JobTracker<https://athena/proxy/srwaishdc1jn0001/50030/jobtracker.jsp>
------------------------------
Cloudera's Distribution for
Hadoop<https://athena/proxy/www.cloudera.com/80/hadoop/>,
2010.

*User:* ryakkala
*Job Name:* PigLatin:DefaultJobName
*Job File:*
hdfs://srwaishdc1nn0001/tmp/hadoop-hadoop/mapred/system/job_201007221306_7120/job.xml<https://athena/proxy/srwaishdc1jn0001/50030/jobconf.jsp?jobid=job_201007221306_7120>
*Job Setup:* Successful<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=setup&pagenum=1&state=completed>
*Status:* Succeeded
*Started at:* Fri Aug 20 14:44:03 GMT-07:00 2010
*Finished at:* Fri Aug 20 14:50:05 GMT-07:00 2010
*Finished in:* 6mins, 1sec
*Job Cleanup:* Successful<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=cleanup&pagenum=1&state=completed>
------------------------------
  Kind % Complete Num Tasks Pending Running Complete Killed Failed/Killed
Task Attempts<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7120>
map<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=map&pagenum=1>
100.00%
  950 0 0 950<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=map&pagenum=1&state=completed>
0 0 / 153<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7120&kind=map&cause=killed>
reduce<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=reduce&pagenum=1>
100.00%
  2000 0 0 2000<https://athena/proxy/srwaishdc1jn0001/50030/jobtasks.jsp?jobid=job_201007221306_7120&type=reduce&pagenum=1&state=completed>
0 0 / 148<https://athena/proxy/srwaishdc1jn0001/50030/jobfailures.jsp?jobid=job_201007221306_7120&kind=reduce&cause=killed>


Counter Map Reduce Total Job Counters Launched reduce tasks 0 0 2,148
Rack-local
map tasks 0 0 244 Launched map tasks 0 0 1,103 Data-local map tasks 0 0 772
FileSystemCounters FILE_BYTES_READ 0 1,609,717,850 1,609,717,850
HDFS_BYTES_READ 64,557,004,337 0 64,557,004,337 FILE_BYTES_WRITTEN
2,685,517,089 1,609,717,850 4,295,234,939 HDFS_BYTES_WRITTEN 0 273,487,824
273,487,824 Map-Reduce Framework Reduce input groups 0 48,500,194
48,500,194 Combine
output records 0 0 0 Map input records 1,879,762,784 0 1,879,762,784 Reduce
shuffle bytes 0 2,638,520,072 2,638,520,072 Reduce output records 0
11,395,326 11,395,326 Spilled Records 77,375,834 77,375,834 154,751,668 Map
output bytes 8,279,214,238 0 8,279,214,238 Map output records 77,375,834 0
77,375,834 Combine input records 0 0 0 Reduce input records 0 77,375,834
77,375,834
------------------------------
Map Completion Graph - close
<https://athena/proxy/srwaishdc1jn0001/50030/jobdetails.jsp?jobid=job_201007221306_7120&refresh=0&map.graph=off>
------------------------------
Reduce Completion Graph - close
<https://athena/proxy/srwaishdc1jn0001/50030/jobdetails.jsp?jobid=job_201007221306_7120&refresh=0&reduce.graph=off>
------------------------------
  ------------------------------
------------------------------
Go back to JobTracker<https://athena/proxy/srwaishdc1jn0001/50030/jobtracker.jsp>
------------------------------
Cloudera's Distribution for
Hadoop<https://athena/proxy/www.cloudera.com/80/hadoop/>,
2010.

On Fri, Aug 20, 2010 at 4:40 PM, Thejas M Nair <te...@yahoo-inc.com> wrote:

> You would find the log for MR jobs that failed either at -
>
> http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
> Or
> http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
>
> You can forward the error message if you need further help with this.
>
> -Thejas
>
>
>
>
> On 8/20/10 4:22 PM, "Raman Yakkala" <ry...@gmail.com> wrote:
>
>   Hi Guys,
>
>
>
> I am trying to do join tow data sets and the jobs are failing. There are
> some warnings reported and I am not good at understanding them. I am
> seeking
> your help in adjusting any parameters to the job so that the job might
> succeed.
>
> Here is the grunt shell output:
>
> grunt> STORE fjid INTO '/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks'
> USING PigStorage();
> 2010-08-20 14:43:47,056 [main] INFO
> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned
> for
> odl: $1, $2, $4, $5, $6, $8
> 2010-08-20 14:43:47,056 [main] INFO
> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys
> pruned
> for odl
> 2010-08-20 14:43:47,059 [main] INFO
> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned
> for
> oil: $2, $3, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18,
> $19, $20, $21, $22, $23, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35,
> $36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $51,
> $53, $55
> 2010-08-20 14:43:47,059 [main] INFO
> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys
> pruned
> for oil
> 2010-08-20 14:43:47,070 [main] INFO
> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
> for qif
> 2010-08-20 14:43:47,070 [main] INFO
> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys
> pruned
> for qif
> 2010-08-20 14:43:47,221 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name:
>
> Store(hdfs://srwaishdc1nn0001/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks:PigStorage)
> - 1-799 Operator Key: 1-799)
> *2010-08-20 14:43:47,269 [main] WARN
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 3 time(s).
> 2010-08-20 14:43:47,274 [main] WARN
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Encountered Warning MULTI_LEAF_MAP 1 time(s).
> *2010-08-20 14:43:47,284 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
> - Rewrite: POPackage->POForEach to POJoinPackage
> 2010-08-20 14:43:47,292 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 4
> 2010-08-20 14:43:47,292 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 4
> 2010-08-20 14:43:47,333 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2010-08-20 14:43:48,645 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2010-08-20 14:43:48,715 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2010-08-20 14:43:49,884 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2010-08-20 14:43:49,896 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 2 map-reduce job(s) waiting for submission.
> *2010-08-20 14:43:49,902 [Thread-13] WARN
> org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
> the arguments. Applications should implement Tool for the same.
> *2010-08-20 14:43:50,398 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2010-08-20 14:43:51,082 [Thread-13] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths to process : 5800
> 2010-08-20 14:44:02,525 [Thread-13] WARN
>  org.apache.hadoop.mapred.JobClient
> - Use GenericOptionsParser for parsing the arguments. Applications should
> implement Tool for the same.
> 2010-08-20 14:44:02,871 [Thread-13] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths to process : 950
> 2010-08-20 14:44:10,835 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_201007221306_7119
> 2010-08-20 14:44:10,835 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - More information at:
> http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
> 2010-08-20 14:44:10,835 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - HadoopJobId: job_201007221306_7120
> 2010-08-20 14:44:10,835 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - More information at:
> http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
> 2010-08-20 14:48:41,613 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 20% complete
> 2010-08-20 14:52:11,487 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 31% complete
> 2010-08-20 14:56:47,404 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 38% complete
> 2010-08-20 14:58:45,888 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 42% complete
> 2010-08-20 15:09:23,195 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
> 2010-08-20 15:09:24,339 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2010-08-20 15:09:24,344 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2010-08-20 15:09:24,360 [Thread-31] WARN
>  org.apache.hadoop.mapred.JobClient
> - Use GenericOptionsParser for parsing the arguments. Applications should
> implement Tool for the same.
> 2010-08-20 15:09:24,630 [Thread-31] INFO
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
> to
> process : 1
> 2010-08-20 15:09:24,678 [Thread-31] INFO
> org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
> paths to process : 200
> 2010-08-20 15:09:24,847 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> *2010-08-20 15:09:24,847 [main] ERROR
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map reduce job(s) failed!
> *2010-08-20 15:09:26,620 [main] INFO
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Some jobs have failed! Stop running all dependent jobs
> grunt>
>
>
>

Re: Help with PIG 0.7 and JOINs

Posted by Thejas M Nair <te...@yahoo-inc.com>.
You would find the log for MR jobs that failed either at -
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
Or
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119

You can forward the error message if you need further help with this.

-Thejas



On 8/20/10 4:22 PM, "Raman Yakkala" <ry...@gmail.com> wrote:

Hi Guys,



I am trying to do join tow data sets and the jobs are failing. There are
some warnings reported and I am not good at understanding them. I am seeking
your help in adjusting any parameters to the job so that the job might
succeed.

Here is the grunt shell output:

grunt> STORE fjid INTO '/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks'
USING PigStorage();
2010-08-20 14:43:47,056 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for
odl: $1, $2, $4, $5, $6, $8
2010-08-20 14:43:47,056 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for odl
2010-08-20 14:43:47,059 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - Columns pruned for
oil: $2, $3, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18,
$19, $20, $21, $22, $23, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35,
$36, $37, $38, $39, $40, $41, $42, $43, $44, $45, $46, $47, $48, $49, $51,
$53, $55
2010-08-20 14:43:47,059 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for oil
2010-08-20 14:43:47,070 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column pruned
for qif
2010-08-20 14:43:47,070 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys pruned
for qif
2010-08-20 14:43:47,221 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name:
Store(hdfs://srwaishdc1nn0001/apps/sq/ryakkala/trec_fixed_price_0820_us_clicks:PigStorage)
- 1-799 Operator Key: 1-799)
*2010-08-20 14:43:47,269 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning DID_NOT_FIND_LOAD_ONLY_MAP_PLAN 3 time(s).
2010-08-20 14:43:47,274 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Encountered Warning MULTI_LEAF_MAP 1 time(s).
*2010-08-20 14:43:47,284 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler$LastInputStreamingOptimizer
- Rewrite: POPackage->POForEach to POJoinPackage
2010-08-20 14:43:47,292 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 4
2010-08-20 14:43:47,292 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 4
2010-08-20 14:43:47,333 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 14:43:48,645 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 14:43:48,715 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 14:43:49,884 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 14:43:49,896 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 2 map-reduce job(s) waiting for submission.
*2010-08-20 14:43:49,902 [Thread-13] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for parsing
the arguments. Applications should implement Tool for the same.
*2010-08-20 14:43:50,398 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2010-08-20 14:43:51,082 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 5800
2010-08-20 14:44:02,525 [Thread-13] WARN  org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-20 14:44:02,871 [Thread-13] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 950
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201007221306_7119
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7119
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_201007221306_7120
2010-08-20 14:44:10,835 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- More information at:
http://srwaishdc1jn0001:50030/jobdetails.jsp?jobid=job_201007221306_7120
2010-08-20 14:48:41,613 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 20% complete
2010-08-20 14:52:11,487 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 31% complete
2010-08-20 14:56:47,404 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 38% complete
2010-08-20 14:58:45,888 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 42% complete
2010-08-20 15:09:23,195 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2010-08-20 15:09:24,339 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-08-20 15:09:24,344 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2010-08-20 15:09:24,360 [Thread-31] WARN  org.apache.hadoop.mapred.JobClient
- Use GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
2010-08-20 15:09:24,630 [Thread-31] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to
process : 1
2010-08-20 15:09:24,678 [Thread-31] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 200
2010-08-20 15:09:24,847 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
*2010-08-20 15:09:24,847 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map reduce job(s) failed!
*2010-08-20 15:09:26,620 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Some jobs have failed! Stop running all dependent jobs
grunt>