You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Chen Song <ch...@gmail.com> on 2014/06/30 18:28:55 UTC
spark streaming counter metrics
I am new to spark streaming and wondering if spark streaming tracks
counters (e.g., how many rows in each consumer, how many rows routed to an
individual reduce task, etc.) in any form so I can get an idea of how data
is skewed? I checked spark job page but don't seem to find any.
--
Chen Song
Re: spark streaming counter metrics
Posted by Chen Song <ch...@gmail.com>.
Thanks Mayur. I will take a look at StreamingListener.
Is there any example you have handy?
On Wed, Jul 2, 2014 at 2:32 AM, Mayur Rustagi <ma...@gmail.com>
wrote:
> You may be able to mix StreamingListener & SparkListener to get meaningful
> information about your task. however you need to connect a lot of pieces to
> make sense of the flow..
>
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>
>
>
> On Mon, Jun 30, 2014 at 9:58 PM, Chen Song <ch...@gmail.com> wrote:
>
>> I am new to spark streaming and wondering if spark streaming tracks
>> counters (e.g., how many rows in each consumer, how many rows routed to an
>> individual reduce task, etc.) in any form so I can get an idea of how data
>> is skewed? I checked spark job page but don't seem to find any.
>>
>>
>>
>> --
>> Chen Song
>>
>>
>
--
Chen Song
Re: spark streaming counter metrics
Posted by Mayur Rustagi <ma...@gmail.com>.
You may be able to mix StreamingListener & SparkListener to get meaningful
information about your task. however you need to connect a lot of pieces to
make sense of the flow..
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>
On Mon, Jun 30, 2014 at 9:58 PM, Chen Song <ch...@gmail.com> wrote:
> I am new to spark streaming and wondering if spark streaming tracks
> counters (e.g., how many rows in each consumer, how many rows routed to an
> individual reduce task, etc.) in any form so I can get an idea of how data
> is skewed? I checked spark job page but don't seem to find any.
>
>
>
> --
> Chen Song
>
>