You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Chandra Mohan, Ananda Vel Murugan" <An...@honeywell.com> on 2015/11/17 11:48:00 UTC

Count of streams processed

HI,

Is it possible to have a running count of number of kafka messages processed in a spark streaming application? Thanks

Regards,
Anand.C

Re: Count of streams processed

Posted by Cody Koeninger <co...@koeninger.org>.
Sure, just call count on each rdd and track it in your driver however you
want.

If count is called directly on a kafkardd (e.g. createDirectStream, then
foreachRDD before doing any other transformations), it should just be using
the beginning and ending offsets rather than doing any real work.

On Tue, Nov 17, 2015 at 4:48 AM, Chandra Mohan, Ananda Vel Murugan <
Ananda.Murugan@honeywell.com> wrote:

> HI,
>
>
>
> Is it possible to have a running count of number of kafka messages
> processed in a spark streaming application? Thanks
>
>
>
> Regards,
>
> Anand.C
>