You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by Dierickx Dominique <d....@gmail.com> on 2014/01/16 17:40:19 UTC
Fwd: Too many Crunch output counters
We're having some trouble with the amount of counters that Crunch creates
when writing to a lot of different output files (slightly more than 120).
This wouldn't be an issue if we were able to configure the maximum number
of allowed counters but unfortunately, because we are running an older
version of Hadoop, doing this is not an option and we are required to patch
Crunch locally when using a new release to leave out the counters. The
required patch (one line...) can be found in the attachment.
I'm not saying the counters should be removed but maybe it is an option to
make them configurable without paying too much of a performance penalty?
Regards,
Dominique Dierickx
Re: Too many Crunch output counters
Posted by Dierickx Dominique <d....@gmail.com>.
OK now I am subscribed! @Josh I will try and implement this this week and
submit a patch in the Jira tracker. Thanks.
2014/1/20 Stefan De Smit <st...@gmail.com>
> That's what he wants, but he can't reply as he wasn't yet subscribed to the
> mailing list.
>
>
> On Thu, Jan 16, 2014 at 7:27 PM, Josh Wills <jw...@cloudera.com> wrote:
>
> > That seems relatively benign; do you need a crunch param that would
> control
> > the usage of the named outputs counter?
> >
> >
> > On Thu, Jan 16, 2014 at 8:40 AM, Dierickx Dominique <
> d.dierickx@gmail.com
> > >wrote:
> >
> > > We're having some trouble with the amount of counters that Crunch
> creates
> > > when writing to a lot of different output files (slightly more than
> 120).
> > > This wouldn't be an issue if we were able to configure the maximum
> number
> > > of allowed counters but unfortunately, because we are running an older
> > > version of Hadoop, doing this is not an option and we are required to
> > patch
> > > Crunch locally when using a new release to leave out the counters. The
> > > required patch (one line...) can be found in the attachment.
> > >
> > > I'm not saying the counters should be removed but maybe it is an option
> > to
> > > make them configurable without paying too much of a performance
> penalty?
> > >
> > > Regards,
> > > Dominique Dierickx
> > >
> > >
> >
> >
> > --
> > Director of Data Science
> > Cloudera <http://www.cloudera.com>
> > Twitter: @josh_wills <http://twitter.com/josh_wills>
> >
>
Re: Too many Crunch output counters
Posted by Stefan De Smit <st...@gmail.com>.
That's what he wants, but he can't reply as he wasn't yet subscribed to the
mailing list.
On Thu, Jan 16, 2014 at 7:27 PM, Josh Wills <jw...@cloudera.com> wrote:
> That seems relatively benign; do you need a crunch param that would control
> the usage of the named outputs counter?
>
>
> On Thu, Jan 16, 2014 at 8:40 AM, Dierickx Dominique <d.dierickx@gmail.com
> >wrote:
>
> > We're having some trouble with the amount of counters that Crunch creates
> > when writing to a lot of different output files (slightly more than 120).
> > This wouldn't be an issue if we were able to configure the maximum number
> > of allowed counters but unfortunately, because we are running an older
> > version of Hadoop, doing this is not an option and we are required to
> patch
> > Crunch locally when using a new release to leave out the counters. The
> > required patch (one line...) can be found in the attachment.
> >
> > I'm not saying the counters should be removed but maybe it is an option
> to
> > make them configurable without paying too much of a performance penalty?
> >
> > Regards,
> > Dominique Dierickx
> >
> >
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>
Re: Too many Crunch output counters
Posted by Josh Wills <jw...@cloudera.com>.
That seems relatively benign; do you need a crunch param that would control
the usage of the named outputs counter?
On Thu, Jan 16, 2014 at 8:40 AM, Dierickx Dominique <d....@gmail.com>wrote:
> We're having some trouble with the amount of counters that Crunch creates
> when writing to a lot of different output files (slightly more than 120).
> This wouldn't be an issue if we were able to configure the maximum number
> of allowed counters but unfortunately, because we are running an older
> version of Hadoop, doing this is not an option and we are required to patch
> Crunch locally when using a new release to leave out the counters. The
> required patch (one line...) can be found in the attachment.
>
> I'm not saying the counters should be removed but maybe it is an option to
> make them configurable without paying too much of a performance penalty?
>
> Regards,
> Dominique Dierickx
>
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>