You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Sourav Chandra <so...@livestream.com> on 2014/02/05 07:32:41 UTC
Message processing rate of spark
HI,
We are currently evaluating spark streaming for our analytics application.
It reads from Kafka, processes and then persists into cassandra
As part of poc project, we need to see the message processing rate of spark
i.e. end to end time taken.
I looked into metrics but did not find any way how to capture this info.
Is there any way to see these metrics?
I am using spark 0.9.0
Thanks,
--
Sourav Chandra
Senior Software Engineer
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
sourav.chandra@livestream.com
o: +91 80 4121 8723
m: +91 988 699 3746
skype: sourav.chandra
Livestream
"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,
Bangalore 560034
www.livestream.com
Re: Message processing rate of spark
Posted by Tathagata Das <ta...@gmail.com>.
Yes! Sorry for the confusion but batchInterval is a more appropriate term.
It is the same batch interval/duration that you provide when you are
creating the StreamingContext.
TD
On Wed, Feb 5, 2014 at 2:44 AM, Pankaj Mittal
<pa...@livestream.com>wrote:
> Isn't it println("Current rate = " + (count / *batchInterval*) + "
> records / second") ?
>
>
>
>
> On Wed, Feb 5, 2014 at 2:05 PM, Sourav Chandra <
> sourav.chandra@livestream.com> wrote:
>
>> Hi Tathagata,
>>
>> How can i find the batch size?
>>
>> Thanks,
>> Sourav
>>
>>
>>
>>
>> On Wed, Feb 5, 2014 at 2:02 PM, Tathagata Das <
>> tathagata.das1565@gmail.com> wrote:
>>
>>> Hi Sourav,
>>>
>>> For number of records received per second, you could use something like
>>> this to calculate number of records in each batch, and divide it by your
>>> batch size.
>>>
>>> yourKafkaStream.foreachRDD(rdd => {
>>> val count = rdd.count
>>> println("Current rate = " + (count / batchSize) + " records / second")
>>> })
>>>
>>> For the delay metrics, take a look at the StreamingListener interface<http://spark.incubator.apache.org/docs/latest/api/streaming/index.html#org.apache.spark.streaming.scheduler.StreamingListener>.
>>> You can create a streaming listener object and use
>>> streamingContext.addListener(object) to attach the listener.
>>>
>>>
>>>
>>> On Tue, Feb 4, 2014 at 10:32 PM, Sourav Chandra <
>>> sourav.chandra@livestream.com> wrote:
>>>
>>>> HI,
>>>>
>>>> We are currently evaluating spark streaming for our analytics
>>>> application.
>>>>
>>>> It reads from Kafka, processes and then persists into cassandra
>>>>
>>>> As part of poc project, we need to see the message processing rate of
>>>> spark i.e. end to end time taken.
>>>>
>>>> I looked into metrics but did not find any way how to capture this info.
>>>>
>>>> Is there any way to see these metrics?
>>>>
>>>> I am using spark 0.9.0
>>>>
>>>> Thanks,
>>>> --
>>>>
>>>> Sourav Chandra
>>>>
>>>> Senior Software Engineer
>>>>
>>>> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>>>>
>>>> sourav.chandra@livestream.com
>>>>
>>>> o: +91 80 4121 8723
>>>>
>>>> m: +91 988 699 3746
>>>>
>>>> skype: sourav.chandra
>>>>
>>>> Livestream
>>>>
>>>> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
>>>> Block, Koramangala Industrial Area,
>>>>
>>>> Bangalore 560034
>>>>
>>>> www.livestream.com
>>>>
>>>
>>>
>>
>>
>> --
>>
>> Sourav Chandra
>>
>> Senior Software Engineer
>>
>> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>>
>> sourav.chandra@livestream.com
>>
>> o: +91 80 4121 8723
>>
>> m: +91 988 699 3746
>>
>> skype: sourav.chandra
>>
>> Livestream
>>
>> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
>> Block, Koramangala Industrial Area,
>>
>> Bangalore 560034
>>
>> www.livestream.com
>>
>
>
Re: Message processing rate of spark
Posted by Pankaj Mittal <pa...@livestream.com>.
Isn't it println("Current rate = " + (count / *batchInterval*) + " records
/ second") ?
On Wed, Feb 5, 2014 at 2:05 PM, Sourav Chandra <
sourav.chandra@livestream.com> wrote:
> Hi Tathagata,
>
> How can i find the batch size?
>
> Thanks,
> Sourav
>
>
>
>
> On Wed, Feb 5, 2014 at 2:02 PM, Tathagata Das <tathagata.das1565@gmail.com
> > wrote:
>
>> Hi Sourav,
>>
>> For number of records received per second, you could use something like
>> this to calculate number of records in each batch, and divide it by your
>> batch size.
>>
>> yourKafkaStream.foreachRDD(rdd => {
>> val count = rdd.count
>> println("Current rate = " + (count / batchSize) + " records / second")
>> })
>>
>> For the delay metrics, take a look at the StreamingListener interface<http://spark.incubator.apache.org/docs/latest/api/streaming/index.html#org.apache.spark.streaming.scheduler.StreamingListener>.
>> You can create a streaming listener object and use
>> streamingContext.addListener(object) to attach the listener.
>>
>>
>>
>> On Tue, Feb 4, 2014 at 10:32 PM, Sourav Chandra <
>> sourav.chandra@livestream.com> wrote:
>>
>>> HI,
>>>
>>> We are currently evaluating spark streaming for our analytics
>>> application.
>>>
>>> It reads from Kafka, processes and then persists into cassandra
>>>
>>> As part of poc project, we need to see the message processing rate of
>>> spark i.e. end to end time taken.
>>>
>>> I looked into metrics but did not find any way how to capture this info.
>>>
>>> Is there any way to see these metrics?
>>>
>>> I am using spark 0.9.0
>>>
>>> Thanks,
>>> --
>>>
>>> Sourav Chandra
>>>
>>> Senior Software Engineer
>>>
>>> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>>>
>>> sourav.chandra@livestream.com
>>>
>>> o: +91 80 4121 8723
>>>
>>> m: +91 988 699 3746
>>>
>>> skype: sourav.chandra
>>>
>>> Livestream
>>>
>>> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
>>> Block, Koramangala Industrial Area,
>>>
>>> Bangalore 560034
>>>
>>> www.livestream.com
>>>
>>
>>
>
>
> --
>
> Sourav Chandra
>
> Senior Software Engineer
>
> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>
> sourav.chandra@livestream.com
>
> o: +91 80 4121 8723
>
> m: +91 988 699 3746
>
> skype: sourav.chandra
>
> Livestream
>
> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
> Block, Koramangala Industrial Area,
>
> Bangalore 560034
>
> www.livestream.com
>
Re: Message processing rate of spark
Posted by Sourav Chandra <so...@livestream.com>.
Hi Tathagata,
How can i find the batch size?
Thanks,
Sourav
On Wed, Feb 5, 2014 at 2:02 PM, Tathagata Das
<ta...@gmail.com>wrote:
> Hi Sourav,
>
> For number of records received per second, you could use something like
> this to calculate number of records in each batch, and divide it by your
> batch size.
>
> yourKafkaStream.foreachRDD(rdd => {
> val count = rdd.count
> println("Current rate = " + (count / batchSize) + " records / second")
> })
>
> For the delay metrics, take a look at the StreamingListener interface<http://spark.incubator.apache.org/docs/latest/api/streaming/index.html#org.apache.spark.streaming.scheduler.StreamingListener>.
> You can create a streaming listener object and use
> streamingContext.addListener(object) to attach the listener.
>
>
>
> On Tue, Feb 4, 2014 at 10:32 PM, Sourav Chandra <
> sourav.chandra@livestream.com> wrote:
>
>> HI,
>>
>> We are currently evaluating spark streaming for our analytics application.
>>
>> It reads from Kafka, processes and then persists into cassandra
>>
>> As part of poc project, we need to see the message processing rate of
>> spark i.e. end to end time taken.
>>
>> I looked into metrics but did not find any way how to capture this info.
>>
>> Is there any way to see these metrics?
>>
>> I am using spark 0.9.0
>>
>> Thanks,
>> --
>>
>> Sourav Chandra
>>
>> Senior Software Engineer
>>
>> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>>
>> sourav.chandra@livestream.com
>>
>> o: +91 80 4121 8723
>>
>> m: +91 988 699 3746
>>
>> skype: sourav.chandra
>>
>> Livestream
>>
>> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
>> Block, Koramangala Industrial Area,
>>
>> Bangalore 560034
>>
>> www.livestream.com
>>
>
>
--
Sourav Chandra
Senior Software Engineer
· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
sourav.chandra@livestream.com
o: +91 80 4121 8723
m: +91 988 699 3746
skype: sourav.chandra
Livestream
"Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
Block, Koramangala Industrial Area,
Bangalore 560034
www.livestream.com
Re: Message processing rate of spark
Posted by Tathagata Das <ta...@gmail.com>.
Hi Sourav,
For number of records received per second, you could use something like
this to calculate number of records in each batch, and divide it by your
batch size.
yourKafkaStream.foreachRDD(rdd => {
val count = rdd.count
println("Current rate = " + (count / batchSize) + " records / second")
})
For the delay metrics, take a look at the StreamingListener
interface<http://spark.incubator.apache.org/docs/latest/api/streaming/index.html#org.apache.spark.streaming.scheduler.StreamingListener>.
You can create a streaming listener object and use
streamingContext.addListener(object) to attach the listener.
On Tue, Feb 4, 2014 at 10:32 PM, Sourav Chandra <
sourav.chandra@livestream.com> wrote:
> HI,
>
> We are currently evaluating spark streaming for our analytics application.
>
> It reads from Kafka, processes and then persists into cassandra
>
> As part of poc project, we need to see the message processing rate of
> spark i.e. end to end time taken.
>
> I looked into metrics but did not find any way how to capture this info.
>
> Is there any way to see these metrics?
>
> I am using spark 0.9.0
>
> Thanks,
> --
>
> Sourav Chandra
>
> Senior Software Engineer
>
> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · ·
>
> sourav.chandra@livestream.com
>
> o: +91 80 4121 8723
>
> m: +91 988 699 3746
>
> skype: sourav.chandra
>
> Livestream
>
> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd
> Block, Koramangala Industrial Area,
>
> Bangalore 560034
>
> www.livestream.com
>