You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Jeff Zhang <zj...@gmail.com> on 2009/11/27 08:38:13 UTC
Re: Could pig dynamic change the reduce number according the mapper
task number ?
I get the suggestion from Owen O'Malley that we can control reducer number
in InputFormat, and I have tried that, it works.
Jeff Zhang
On Sat, Nov 14, 2009 at 1:23 AM, Alan Gates <ga...@yahoo-inc.com> wrote:
>
> On Nov 12, 2009, at 2:49 PM, Scott Carey wrote:
>
> Is it possible to have a script at least use the default configured Hadoop
>> value? Or is there a way to do that already?
>>
>
> If the user doesn't specify a parallelism Pig doesn't set a value in
> JobConf for the reduce, which means it will pick up the default for the
> cluster. Unless cluster administrators change it, the default for the
> cluster is 1.
>
>
>> Alan.
>
Re: Pig batch mode vs grunt
Posted by Jeff Zhang <zj...@gmail.com>.
Maybe you can look at the job tracker UI, http://your-cluster:50030
There you can see the status of jobs and logs of tasks
Jeff Zhnag
On Fri, Nov 27, 2009 at 7:51 PM, <le...@llnl.gov> wrote:
> I'm having a weird sort of problem. When I run pig on my cluster
> (unfortunately also
> adminstered by me), it works fine with grunt, but not as a script. This
> morning I
> started a job in grunt that proceeded OK, until I lost the connection and
> grunt died.
> (screen is not installed on the cluster, and I don't have root.) So I
> decided to run the
> script like so:
>
> nohup pig <script.pig>
>
> I never get anything past:
>
> 2009-11-27 15:43:04,081 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0%
> complete
>
> The one I ran in grunt would occisionally update to 2%, 8%, 20%, etc. When
> I run it in
> batch mode, that never happens.
>
> However, I can see it's doing something, because I can tail the JobTracker
> log:
> 2009-11-27 19:16:03,093 INFO org.apache.hadoop.mapred.ResourceEstimator:
> completedMapsUpdates:12935 completedMapsInputSize:13563343495
> completedMapsOutputSize:20135178306
> 2009-11-27 19:16:03,094 INFO org.apache.hadoop.mapred.JobTracker: Adding
> task
> 'attempt_200911270748_0004_m_013052_0' to tip
> task_200911270748_0004_m_013052, for
> tracker 'tracker_tuson64:localhost.localdomain/127.0.0.1:53589'
> 2009-11-27 19:16:03,094 INFO org.apache.hadoop.mapred.JobInProgress:
> Choosing data-local
> task task_200911270748_0004_m_013052
>
> I mean, it says it has completed map 12935. It actually still hasn't
> produced any result
> directories, so that seems weird. Even weirder, when pig is running in
> batch mode, I
> can't run anything else on hadoop!
>
> For example, now I can't run my usual hadoop program, which works fine
> normally. This
> little test, that should run in about 1 second, never even starts:
>
> [leek2@tuson118 dna]$ hadoop jar /home/leek2/hadoopstuff/dna/dnamapper.jar
> org.myorg.DNAMapper /dna/kmer.txt /dna/kmerres5 5
> Setting kmer size 5
> 09/11/27 19:21:17 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the
> arguments. Applications should implement Tool for the same.
> 09/11/27 19:21:17 INFO input.FileInputFormat: Total input paths to process
> : 1
> 09/11/27 19:21:18 INFO mapred.JobClient: Running job: job_200911270748_0008
> 09/11/27 19:21:19 INFO mapred.JobClient: map 0% reduce 0%
>
> What is going on here? How can I figure out if anything is happening?
>
> Jim
>
>
Pig batch mode vs grunt
Posted by le...@llnl.gov.
I'm having a weird sort of problem. When I run pig on my cluster
(unfortunately also
adminstered by me), it works fine with grunt, but not as a script.
This morning I
started a job in grunt that proceeded OK, until I lost the connection
and grunt died.
(screen is not installed on the cluster, and I don't have root.) So I
decided to run the
script like so:
nohup pig <script.pig>
I never get anything past:
2009-11-27 15:43:04,081 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0%
complete
The one I ran in grunt would occisionally update to 2%, 8%, 20%, etc.
When I run it in
batch mode, that never happens.
However, I can see it's doing something, because I can tail the
JobTracker log:
2009-11-27 19:16:03,093 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:12935 completedMapsInputSize:13563343495
completedMapsOutputSize:20135178306
2009-11-27 19:16:03,094 INFO org.apache.hadoop.mapred.JobTracker: Adding task
'attempt_200911270748_0004_m_013052_0' to tip
task_200911270748_0004_m_013052, for
tracker 'tracker_tuson64:localhost.localdomain/127.0.0.1:53589'
2009-11-27 19:16:03,094 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local
task task_200911270748_0004_m_013052
I mean, it says it has completed map 12935. It actually still hasn't
produced any result
directories, so that seems weird. Even weirder, when pig is running
in batch mode, I
can't run anything else on hadoop!
For example, now I can't run my usual hadoop program, which works fine
normally. This
little test, that should run in about 1 second, never even starts:
[leek2@tuson118 dna]$ hadoop jar /home/leek2/hadoopstuff/dna/dnamapper.jar
org.myorg.DNAMapper /dna/kmer.txt /dna/kmerres5 5
Setting kmer size 5
09/11/27 19:21:17 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the
arguments. Applications should implement Tool for the same.
09/11/27 19:21:17 INFO input.FileInputFormat: Total input paths to process : 1
09/11/27 19:21:18 INFO mapred.JobClient: Running job: job_200911270748_0008
09/11/27 19:21:19 INFO mapred.JobClient: map 0% reduce 0%
What is going on here? How can I figure out if anything is happening?
Jim
Re: Could pig dynamic change the reduce number according the mapper
task number ?
Posted by Jeff Zhang <zj...@gmail.com>.
Attach my sample code ( this InputFormat generate 1 reducer task for each 5
mapper task):
*public class MyInputFormat extends TextInputFormat {
@Override
public InputSplit[] getSplits(JobConf job, int numSplits)
throws IOException {
InputSplit[] splits = super.getSplits(job, numSplits);
int reducerNum = splits.length / 5;
if (reducerNum == 0) {
reducerNum = 1;
}
job.setNumReduceTasks(reducerNum);
return splits;
}
}*
After pig integrate the InputFormat in LoadFunc (Pig-966), it will be
possible to change the reducer task number dynamically.
Jeff Zhang
On Fri, Nov 27, 2009 at 3:38 PM, Jeff Zhang <zj...@gmail.com> wrote:
> I get the suggestion from Owen O'Malley that we can control reducer number
> in InputFormat, and I have tried that, it works.
>
>
> Jeff Zhang
>
>
>
>
> On Sat, Nov 14, 2009 at 1:23 AM, Alan Gates <ga...@yahoo-inc.com> wrote:
>
>>
>> On Nov 12, 2009, at 2:49 PM, Scott Carey wrote:
>>
>> Is it possible to have a script at least use the default configured
>>> Hadoop value? Or is there a way to do that already?
>>>
>>
>> If the user doesn't specify a parallelism Pig doesn't set a value in
>> JobConf for the reduce, which means it will pick up the default for the
>> cluster. Unless cluster administrators change it, the default for the
>> cluster is 1.
>>
>>
>>> Alan.
>>
>
>