You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Sudip Sinha <su...@gmail.com> on 2012/04/19 14:35:37 UTC
Required condition for the combiner to run
Hi,
I'm reposting this as I've not received any reply to my earlier post on the
same issue.
I've read that the combiner only works if it is specified AND the sort
memory buffer overflows in the mapper.
http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201107.mbox/%3C374D8F3F-B8B1-499F-BEDB-BFEE3219010C@hortonworks.com%3E
But when I run a Hadoop streaming job in R using RHadoop, the combiner
always runs when specified. This is on a very small dataset.
Is this the desired behaviour?
More on this: https://github.com/RevolutionAnalytics/RHadoop/issues/70
Thanks,
Sudip Sinha
Re: Required condition for the combiner to run
Posted by Sudip Sinha <su...@gmail.com>.
Code, input and output - http://pastebin.com/AG2DyZ22
userlogs - http://pastebin.com/bLv2Ad3J, http://pastebin.com/RzzEre1R
On Thu, Apr 19, 2012 at 7:04 PM, Harsh J <ha...@cloudera.com> wrote:
> Can you pastebin and provide your specific mapper userlog
> (syslogs/stderr/stdout)?
>
> On Thu, Apr 19, 2012 at 6:05 PM, Sudip Sinha <su...@gmail.com>
> wrote:
> > Hi,
> >
> > I'm reposting this as I've not received any reply to my earlier post on
> the
> > same issue.
> >
> > I've read that the combiner only works if it is specified AND the sort
> > memory buffer overflows in the mapper.
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201107.mbox/%3C374D8F3F-B8B1-499F-BEDB-BFEE3219010C@hortonworks.com%3E
> >
> > But when I run a Hadoop streaming job in R using RHadoop, the combiner
> > always runs when specified. This is on a very small dataset.
> >
> > Is this the desired behaviour?
> >
> > More on this: https://github.com/RevolutionAnalytics/RHadoop/issues/70
> >
> > Thanks,
> > Sudip Sinha
>
>
>
> --
> Harsh J
>
Re: Required condition for the combiner to run
Posted by Harsh J <ha...@cloudera.com>.
Can you pastebin and provide your specific mapper userlog
(syslogs/stderr/stdout)?
On Thu, Apr 19, 2012 at 6:05 PM, Sudip Sinha <su...@gmail.com> wrote:
> Hi,
>
> I'm reposting this as I've not received any reply to my earlier post on the
> same issue.
>
> I've read that the combiner only works if it is specified AND the sort
> memory buffer overflows in the mapper.
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201107.mbox/%3C374D8F3F-B8B1-499F-BEDB-BFEE3219010C@hortonworks.com%3E
>
> But when I run a Hadoop streaming job in R using RHadoop, the combiner
> always runs when specified. This is on a very small dataset.
>
> Is this the desired behaviour?
>
> More on this: https://github.com/RevolutionAnalytics/RHadoop/issues/70
>
> Thanks,
> Sudip Sinha
--
Harsh J