Re: spark streaming counter metrics

[Mayur Rustagi <ma...@gmail.com>]

You may be able to mix StreamingListener & SparkListener to get meaningful
information about your task. however you need to connect a lot of pieces to
make sense of the flow..

Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Mon, Jun 30, 2014 at 9:58 PM, Chen Song <ch...@gmail.com> wrote:

> I am new to spark streaming and wondering if spark streaming tracks
> counters (e.g., how many rows in each consumer, how many rows routed to an
> individual reduce task, etc.) in any form so I can get an idea of how data
> is skewed? I checked spark job page but don't seem to find any.
>
>
>
> --
> Chen Song
>
>

Re: spark streaming counter metrics

[Chen Song <ch...@gmail.com>]

Thanks Mayur. I will take a look at StreamingListener.

Is there any example you have handy?


On Wed, Jul 2, 2014 at 2:32 AM, Mayur Rustagi <ma...@gmail.com>
wrote:

> You may be able to mix StreamingListener & SparkListener to get meaningful
> information about your task. however you need to connect a lot of pieces to
> make sense of the flow..
>
> Mayur Rustagi
> Ph: +1 (760) 203 3257
> http://www.sigmoidanalytics.com
> @mayur_rustagi <https://twitter.com/mayur_rustagi>
>
>
>
> On Mon, Jun 30, 2014 at 9:58 PM, Chen Song <ch...@gmail.com> wrote:
>
>> I am new to spark streaming and wondering if spark streaming tracks
>> counters (e.g., how many rows in each consumer, how many rows routed to an
>> individual reduce task, etc.) in any form so I can get an idea of how data
>> is skewed? I checked spark job page but don't seem to find any.
>>
>>
>>
>> --
>> Chen Song
>>
>>
>


-- 
Chen Song