You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Xin Jing <xi...@beyondfun.net> on 2011/10/28 06:34:50 UTC

problem when using combiner and MultipleOutputFormat

Hi all,

I am currently encountering a tough problem, my job use MultipleOutputFormat to output result into different folder, and I have to use a combiner to enhance performance. In this situation, reduce does not work, reduce cannot receive any data. I searched this issue and found a related topic, http://lucene.472066.n3.nabble.com/Combiner-and-MultipleOutputs-in-Mapreduce-td1640503.html , but not get clear what the solution is really. Seems it is the constraint of hadoop framework?

I found a interesting phenomenon, when I limit the map input record to a small number (such as 10000), the reduce is ok, it can receive data and the result is correct. But when the input is over a million record, the reduce receive nothing. I guess the reason is the combiner only be called once when data is small while combiner be called multiple time when data is huge.

To summary, how can I make combiner feasible  while using MultipleOutputFormat? Any solution or suggestion is welcome.


Thanks


Auditable Hadoop

Posted by Friso van Vollenhoven <fv...@xebia.com>.
Hi all,

I have a auditing challenge. I am looking for a quite detailed level of audit trail on MR jobs. I know that HDFS has a audit log, which you can write to a separate file through log4j config. But what I ideally need is something that allows to determine, with certainty, which jobs were run against what data and by whom. Now, with 'which jobs', I mean source code, not just the binary.

One idea I had for basic Java based MR jobs is to have devs (me) check the job's code in the some SCM and then have a tool that does a checkout of a particular branch on a machine that has cluster access and is not open to the dev team. The tool would do a checkout and a build and write the SCM tag + a hash of the binary (.jar) to a audit trail. Then I'd need some kind of hook in the job tracker that checks whether a submitted job's binary is properly audited. This way, you could always find the source that was executed.

Question is: is there a possibility for such a hook in the JT? Or do I need to patch it. It would be nice to have the auditing happen in the JT, such that the dev team can have regular access to the cluster (thus use the hadoop command line tool to copy/move files, etc.) and it would just reject jobs that have not been audited.

Also, with this model, non-Java based jobs are a problem. I probably won't be using streaming, but Pig, Hive and Mahout will likely be used. For these I'd need some additional steps: confirming that the Pig / Hive / Mahout binaries which are submitted are trusted ones and have Pig/Hive/Mahout add some config params or other info about the script / query that is being executed in the plan.

Does anyone have any ideas on this? Or relevant experiences?


Thanks,
Friso


Re: problem when using combiner and MultipleOutputFormat

Posted by Harsh J <ha...@cloudera.com>.
Xin,

No problem. Glad to know you managed to hunt it down and fix it! :)

On 28-Oct-2011, at 6:11 PM, Xin Jing wrote:

> Hi Harsh,
> 
> You are right, I went through my combiner code again carefully, and found that there is a bug in my code. For short, I add a type for each value, and check the type before processing. The type before and after multiple combiner iteration is not the same, so...
> 
> I found the root cause of the issue, thanks for your help, Harsh.
> ________________________________________
> From: Xin Jing [xinjing@beyondfun.net]
> Sent: Friday, October 28, 2011 2:43 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: RE: problem when using combiner and MultipleOutputFormat
> 
> OK, I will prepare my code to show how it works.
> 
> Here another question, my combiner DOES output some records, if it the reason as you said, my combiner's behavior is wrong when run one time and multiple times, the reason that reduce cannot receive input is because of the wrong data format of the combiner?
> 
> I attach the job result, please take a look.
> ________________________________________
> From: Harsh J [harsh@cloudera.com]
> Sent: Friday, October 28, 2011 2:33 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: Re: problem when using combiner and MultipleOutputFormat
> 
> Xin,
> 
> I guess am now confused. I was thinking that your combiner was a
> reused reducer, and that was probably eating away all the outputs when
> run (Combiners run 0…N times (yes, 0 is possible), so that may explain
> your low vs. high data volume difference).
> 
> I'm sure the issue is with your combiner implementation, but I'd have
> to see some mock code to tell you  more, as am unsure on what you are
> attempting to do in there.
> 
> On Fri, Oct 28, 2011 at 11:52 AM, Xin Jing <xi...@beyondfun.net> wrote:
>> Thanks for your answer.
>> 
>> I am using different combiner and reducer. As I have said in previous mail, when the data set is small, it works fine and the result is correct. I can tell the functionality of my job is ok, right?
>> 
>> I cannot understand what do you mean by ' Do not output to files directly from your combiner', could you give me more hints? I combiner code, I am using output.collect() to output my result, do I misuse it?
>> ________________________________________
>> From: Harsh J [harsh@cloudera.com]
>> Sent: Friday, October 28, 2011 2:11 PM
>> To: mapreduce-user@hadoop.apache.org
>> Subject: Re: problem when using combiner and MultipleOutputFormat
>> 
>> Xin,
>> 
>> You probably just need to write a special Combiner class instead of
>> reusing your Reducer class for combiner purposes. In an MR job, you
>> need to specifically guarantee that the combiner outputs the same type
>> of K-V pairs as the reducer's input. Do not output to files directly
>> from your combiner, and that is why you'd need a different class impl.
>> performing the optimization.
>> 
>> On Fri, Oct 28, 2011 at 10:04 AM, Xin Jing <xi...@beyondfun.net> wrote:
>>> 
>>> Hi all,
>>> I am currently encountering a tough problem, my job use MultipleOutputFormat
>>> to output result into different folder, and I have to use a combiner to
>>> enhance performance. In this situation, reduce does not work, reduce cannot
>>> receive any data. I searched this issue and found a related
>>> topic, http://lucene.472066.n3.nabble.com/Combiner-and-MultipleOutputs-in-Mapreduce-td1640503.html ,
>>> but not get clear what the solution is really. Seems it is the constraint of
>>> hadoop framework?
>>> I found a interesting phenomenon, when I limit the map input record to a
>>> small number (such as 10000), the reduce is ok, it can receive data and the
>>> result is correct. But when the input is over a million record, the reduce
>>> receive nothing. I guess the reason is the combiner only be called once when
>>> data is small while combiner be called multiple time when data is huge.
>>> To summary, how can I make combiner feasible  while using
>>> MultipleOutputFormat? Any solution or suggestion is welcome.
>>> 
>>> Thanks
>>> 
>> 
>> 
>> 
>> --
>> Harsh J
>> 
>> 
>> 
> 
> 
> 
> --
> Harsh J
> 
> 


RE: problem when using combiner and MultipleOutputFormat

Posted by Xin Jing <xi...@beyondfun.net>.
Hi Harsh,

You are right, I went through my combiner code again carefully, and found that there is a bug in my code. For short, I add a type for each value, and check the type before processing. The type before and after multiple combiner iteration is not the same, so...

I found the root cause of the issue, thanks for your help, Harsh.
________________________________________
From: Xin Jing [xinjing@beyondfun.net]
Sent: Friday, October 28, 2011 2:43 PM
To: mapreduce-user@hadoop.apache.org
Subject: RE: problem when using combiner and MultipleOutputFormat

OK, I will prepare my code to show how it works.

Here another question, my combiner DOES output some records, if it the reason as you said, my combiner's behavior is wrong when run one time and multiple times, the reason that reduce cannot receive input is because of the wrong data format of the combiner?

I attach the job result, please take a look.
________________________________________
From: Harsh J [harsh@cloudera.com]
Sent: Friday, October 28, 2011 2:33 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: problem when using combiner and MultipleOutputFormat

Xin,

I guess am now confused. I was thinking that your combiner was a
reused reducer, and that was probably eating away all the outputs when
run (Combiners run 0…N times (yes, 0 is possible), so that may explain
your low vs. high data volume difference).

I'm sure the issue is with your combiner implementation, but I'd have
to see some mock code to tell you  more, as am unsure on what you are
attempting to do in there.

On Fri, Oct 28, 2011 at 11:52 AM, Xin Jing <xi...@beyondfun.net> wrote:
> Thanks for your answer.
>
> I am using different combiner and reducer. As I have said in previous mail, when the data set is small, it works fine and the result is correct. I can tell the functionality of my job is ok, right?
>
> I cannot understand what do you mean by ' Do not output to files directly from your combiner', could you give me more hints? I combiner code, I am using output.collect() to output my result, do I misuse it?
> ________________________________________
> From: Harsh J [harsh@cloudera.com]
> Sent: Friday, October 28, 2011 2:11 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: Re: problem when using combiner and MultipleOutputFormat
>
> Xin,
>
> You probably just need to write a special Combiner class instead of
> reusing your Reducer class for combiner purposes. In an MR job, you
> need to specifically guarantee that the combiner outputs the same type
> of K-V pairs as the reducer's input. Do not output to files directly
> from your combiner, and that is why you'd need a different class impl.
> performing the optimization.
>
> On Fri, Oct 28, 2011 at 10:04 AM, Xin Jing <xi...@beyondfun.net> wrote:
>>
>> Hi all,
>> I am currently encountering a tough problem, my job use MultipleOutputFormat
>> to output result into different folder, and I have to use a combiner to
>> enhance performance. In this situation, reduce does not work, reduce cannot
>> receive any data. I searched this issue and found a related
>> topic, http://lucene.472066.n3.nabble.com/Combiner-and-MultipleOutputs-in-Mapreduce-td1640503.html ,
>> but not get clear what the solution is really. Seems it is the constraint of
>> hadoop framework?
>> I found a interesting phenomenon, when I limit the map input record to a
>> small number (such as 10000), the reduce is ok, it can receive data and the
>> result is correct. But when the input is over a million record, the reduce
>> receive nothing. I guess the reason is the combiner only be called once when
>> data is small while combiner be called multiple time when data is huge.
>> To summary, how can I make combiner feasible  while using
>> MultipleOutputFormat? Any solution or suggestion is welcome.
>>
>> Thanks
>>
>
>
>
> --
> Harsh J
>
>
>



--
Harsh J



RE: problem when using combiner and MultipleOutputFormat

Posted by Xin Jing <xi...@beyondfun.net>.
OK, I will prepare my code to show how it works.

Here another question, my combiner DOES output some records, if it the reason as you said, my combiner's behavior is wrong when run one time and multiple times, the reason that reduce cannot receive input is because of the wrong data format of the combiner?

I attach the job result, please take a look.
________________________________________
From: Harsh J [harsh@cloudera.com]
Sent: Friday, October 28, 2011 2:33 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: problem when using combiner and MultipleOutputFormat

Xin,

I guess am now confused. I was thinking that your combiner was a
reused reducer, and that was probably eating away all the outputs when
run (Combiners run 0…N times (yes, 0 is possible), so that may explain
your low vs. high data volume difference).

I'm sure the issue is with your combiner implementation, but I'd have
to see some mock code to tell you  more, as am unsure on what you are
attempting to do in there.

On Fri, Oct 28, 2011 at 11:52 AM, Xin Jing <xi...@beyondfun.net> wrote:
> Thanks for your answer.
>
> I am using different combiner and reducer. As I have said in previous mail, when the data set is small, it works fine and the result is correct. I can tell the functionality of my job is ok, right?
>
> I cannot understand what do you mean by ' Do not output to files directly from your combiner', could you give me more hints? I combiner code, I am using output.collect() to output my result, do I misuse it?
> ________________________________________
> From: Harsh J [harsh@cloudera.com]
> Sent: Friday, October 28, 2011 2:11 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: Re: problem when using combiner and MultipleOutputFormat
>
> Xin,
>
> You probably just need to write a special Combiner class instead of
> reusing your Reducer class for combiner purposes. In an MR job, you
> need to specifically guarantee that the combiner outputs the same type
> of K-V pairs as the reducer's input. Do not output to files directly
> from your combiner, and that is why you'd need a different class impl.
> performing the optimization.
>
> On Fri, Oct 28, 2011 at 10:04 AM, Xin Jing <xi...@beyondfun.net> wrote:
>>
>> Hi all,
>> I am currently encountering a tough problem, my job use MultipleOutputFormat
>> to output result into different folder, and I have to use a combiner to
>> enhance performance. In this situation, reduce does not work, reduce cannot
>> receive any data. I searched this issue and found a related
>> topic, http://lucene.472066.n3.nabble.com/Combiner-and-MultipleOutputs-in-Mapreduce-td1640503.html ,
>> but not get clear what the solution is really. Seems it is the constraint of
>> hadoop framework?
>> I found a interesting phenomenon, when I limit the map input record to a
>> small number (such as 10000), the reduce is ok, it can receive data and the
>> result is correct. But when the input is over a million record, the reduce
>> receive nothing. I guess the reason is the combiner only be called once when
>> data is small while combiner be called multiple time when data is huge.
>> To summary, how can I make combiner feasible  while using
>> MultipleOutputFormat? Any solution or suggestion is welcome.
>>
>> Thanks
>>
>
>
>
> --
> Harsh J
>
>
>



--
Harsh J


Re: problem when using combiner and MultipleOutputFormat

Posted by Harsh J <ha...@cloudera.com>.
Xin,

I guess am now confused. I was thinking that your combiner was a
reused reducer, and that was probably eating away all the outputs when
run (Combiners run 0…N times (yes, 0 is possible), so that may explain
your low vs. high data volume difference).

I'm sure the issue is with your combiner implementation, but I'd have
to see some mock code to tell you  more, as am unsure on what you are
attempting to do in there.

On Fri, Oct 28, 2011 at 11:52 AM, Xin Jing <xi...@beyondfun.net> wrote:
> Thanks for your answer.
>
> I am using different combiner and reducer. As I have said in previous mail, when the data set is small, it works fine and the result is correct. I can tell the functionality of my job is ok, right?
>
> I cannot understand what do you mean by ' Do not output to files directly from your combiner', could you give me more hints? I combiner code, I am using output.collect() to output my result, do I misuse it?
> ________________________________________
> From: Harsh J [harsh@cloudera.com]
> Sent: Friday, October 28, 2011 2:11 PM
> To: mapreduce-user@hadoop.apache.org
> Subject: Re: problem when using combiner and MultipleOutputFormat
>
> Xin,
>
> You probably just need to write a special Combiner class instead of
> reusing your Reducer class for combiner purposes. In an MR job, you
> need to specifically guarantee that the combiner outputs the same type
> of K-V pairs as the reducer's input. Do not output to files directly
> from your combiner, and that is why you'd need a different class impl.
> performing the optimization.
>
> On Fri, Oct 28, 2011 at 10:04 AM, Xin Jing <xi...@beyondfun.net> wrote:
>>
>> Hi all,
>> I am currently encountering a tough problem, my job use MultipleOutputFormat
>> to output result into different folder, and I have to use a combiner to
>> enhance performance. In this situation, reduce does not work, reduce cannot
>> receive any data. I searched this issue and found a related
>> topic, http://lucene.472066.n3.nabble.com/Combiner-and-MultipleOutputs-in-Mapreduce-td1640503.html ,
>> but not get clear what the solution is really. Seems it is the constraint of
>> hadoop framework?
>> I found a interesting phenomenon, when I limit the map input record to a
>> small number (such as 10000), the reduce is ok, it can receive data and the
>> result is correct. But when the input is over a million record, the reduce
>> receive nothing. I guess the reason is the combiner only be called once when
>> data is small while combiner be called multiple time when data is huge.
>> To summary, how can I make combiner feasible  while using
>> MultipleOutputFormat? Any solution or suggestion is welcome.
>>
>> Thanks
>>
>
>
>
> --
> Harsh J
>
>
>



-- 
Harsh J

RE: problem when using combiner and MultipleOutputFormat

Posted by Xin Jing <xi...@beyondfun.net>.
Thanks for your answer.

I am using different combiner and reducer. As I have said in previous mail, when the data set is small, it works fine and the result is correct. I can tell the functionality of my job is ok, right?

I cannot understand what do you mean by ' Do not output to files directly from your combiner', could you give me more hints? I combiner code, I am using output.collect() to output my result, do I misuse it?
________________________________________
From: Harsh J [harsh@cloudera.com]
Sent: Friday, October 28, 2011 2:11 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: problem when using combiner and MultipleOutputFormat

Xin,

You probably just need to write a special Combiner class instead of
reusing your Reducer class for combiner purposes. In an MR job, you
need to specifically guarantee that the combiner outputs the same type
of K-V pairs as the reducer's input. Do not output to files directly
from your combiner, and that is why you'd need a different class impl.
performing the optimization.

On Fri, Oct 28, 2011 at 10:04 AM, Xin Jing <xi...@beyondfun.net> wrote:
>
> Hi all,
> I am currently encountering a tough problem, my job use MultipleOutputFormat
> to output result into different folder, and I have to use a combiner to
> enhance performance. In this situation, reduce does not work, reduce cannot
> receive any data. I searched this issue and found a related
> topic, http://lucene.472066.n3.nabble.com/Combiner-and-MultipleOutputs-in-Mapreduce-td1640503.html ,
> but not get clear what the solution is really. Seems it is the constraint of
> hadoop framework?
> I found a interesting phenomenon, when I limit the map input record to a
> small number (such as 10000), the reduce is ok, it can receive data and the
> result is correct. But when the input is over a million record, the reduce
> receive nothing. I guess the reason is the combiner only be called once when
> data is small while combiner be called multiple time when data is huge.
> To summary, how can I make combiner feasible  while using
> MultipleOutputFormat? Any solution or suggestion is welcome.
>
> Thanks
>



--
Harsh J



Re: problem when using combiner and MultipleOutputFormat

Posted by Harsh J <ha...@cloudera.com>.
Xin,

You probably just need to write a special Combiner class instead of
reusing your Reducer class for combiner purposes. In an MR job, you
need to specifically guarantee that the combiner outputs the same type
of K-V pairs as the reducer's input. Do not output to files directly
from your combiner, and that is why you'd need a different class impl.
performing the optimization.

On Fri, Oct 28, 2011 at 10:04 AM, Xin Jing <xi...@beyondfun.net> wrote:
>
> Hi all,
> I am currently encountering a tough problem, my job use MultipleOutputFormat
> to output result into different folder, and I have to use a combiner to
> enhance performance. In this situation, reduce does not work, reduce cannot
> receive any data. I searched this issue and found a related
> topic, http://lucene.472066.n3.nabble.com/Combiner-and-MultipleOutputs-in-Mapreduce-td1640503.html ,
> but not get clear what the solution is really. Seems it is the constraint of
> hadoop framework?
> I found a interesting phenomenon, when I limit the map input record to a
> small number (such as 10000), the reduce is ok, it can receive data and the
> result is correct. But when the input is over a million record, the reduce
> receive nothing. I guess the reason is the combiner only be called once when
> data is small while combiner be called multiple time when data is huge.
> To summary, how can I make combiner feasible  while using
> MultipleOutputFormat? Any solution or suggestion is welcome.
>
> Thanks
>



-- 
Harsh J