You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Daniel Siegmann <da...@velos.io> on 2014/02/07 20:55:58 UTC
Equivalent to Hadoop's standard counters
When I run Hadoop jobs, the job tracker shows a bunch of standard counters
(in addition to any others I create). For example, in Hadoop I can get the
count of "Map input records" without needing to write any code explicitly.
Is there any equivalent to this with Spark's accumulators?
--
Daniel Siegmann, Software Developer
Velos
440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
E: daniel.siegmann@velos.io W: www.velos.io
Re: Equivalent to Hadoop's standard counters
Posted by Daniel Siegmann <da...@velos.io>.
Thanks for the response Matei. Guess I'll have to give it some thought.
~Daniel
On Sat, Feb 8, 2014 at 6:38 PM, Matei Zaharia <ma...@gmail.com>wrote:
> Hi Daniel,
>
> There isn't any built-in equivalent of this right now, though some
> metrics, such as shuffle bytes, are tracked on the application UI (
> http://driver:4040). In this particular case I guess you could add a map
> that updates an accumulator, but it's kind of awkward. Feel free to open a
> JIRA about this and maybe someone will implement it, since it sounds like a
> useful metric.
>
> Matei
>
> On Feb 7, 2014, at 11:55 AM, Daniel Siegmann <da...@velos.io>
> wrote:
>
> When I run Hadoop jobs, the job tracker shows a bunch of standard counters
> (in addition to any others I create). For example, in Hadoop I can get the
> count of "Map input records" without needing to write any code explicitly.
>
> Is there any equivalent to this with Spark's accumulators?
>
> --
> Daniel Siegmann, Software Developer
> Velos
>
> 440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
> E: daniel.siegmann@velos.io W: www.velos.io
>
>
>
--
Daniel Siegmann, Software Developer
Velos
440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
E: daniel.siegmann@velos.io W: www.velos.io
Re: Equivalent to Hadoop's standard counters
Posted by Matei Zaharia <ma...@gmail.com>.
Hi Daniel,
There isn’t any built-in equivalent of this right now, though some metrics, such as shuffle bytes, are tracked on the application UI (http://driver:4040). In this particular case I guess you could add a map that updates an accumulator, but it’s kind of awkward. Feel free to open a JIRA about this and maybe someone will implement it, since it sounds like a useful metric.
Matei
On Feb 7, 2014, at 11:55 AM, Daniel Siegmann <da...@velos.io> wrote:
> When I run Hadoop jobs, the job tracker shows a bunch of standard counters (in addition to any others I create). For example, in Hadoop I can get the count of "Map input records" without needing to write any code explicitly.
>
> Is there any equivalent to this with Spark's accumulators?
>
> --
> Daniel Siegmann, Software Developer
> Velos
>
> 440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
> E: daniel.siegmann@velos.io W: www.velos.io