You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "Prateek ." <pr...@aricent.com> on 2015/10/09 16:34:52 UTC
RE: Streaming Application Unable to get Stream from Kafka
Hi All,
In my application I have a serializable class which is taking InputDStream from Kafka. The inputDStream contains JSON which is stored in serializable case class. Transformations are applied and saveToCassandra() is executed.
I was getting task not serializable exception , so I made the class serializable.
Now ,the application is working fine in standalone mode, but not able to receive data in local mode with the below mentioned log.
What is internally happening?, if anyone have some insights Please share!
Thank You in advance
Regards,
Prateek
From: Prateek .
Sent: Friday, October 09, 2015 6:55 PM
To: user@spark.apache.org
Subject: Streaming Application Unable to get Stream from Kafka
Hi,
I have Spark Streaming application running with the following log on console, I don’t get any exception but I am not able to receive the data from Kafka Stream.
Can anyone please provide any insight what is happening with Spark Streaming. Is Receiver is not able to read the stream? How shall I debug it?
JobScheduler: Added jobs for time 1444396043000 ms
15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(922) called with curMem=1009312, maxMem=278302556
15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043000 stored as bytes in memory (estimated size 922.0 B, free 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043000 in memory on webanalytics03:51843 (size: 922.0 B, free: 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043000
15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043000 already exists on this machine; not re-adding it
15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043000
15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(926) called with curMem=1010234, maxMem=278302556
15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043200 stored as bytes in memory (estimated size 926.0 B, free 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043200 in memory on webanalytics03:51843 (size: 926.0 B, free: 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043200
15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043200 already exists on this machine; not re-adding it
15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043200
15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(923) called with curMem=1011160, maxMem=278302556
15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043400 stored as bytes in memory (estimated size 923.0 B, free 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043400 in memory on webanalytics03:51843 (size: 923.0 B, free: 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043400
15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043400 already exists on this machine; not re-adding it
15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043400
15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(917) called with curMem=1012083, maxMem=278302556
15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043600 stored as bytes in memory (estimated size 917.0 B, free 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043600 in memory on webanalytics03:51843 (size: 917.0 B, free: 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043600
15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043600 already exists on this machine; not re-adding it
15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043600
15/10/09 18:37:24 INFO ReceiverTracker: Stream 0 received 5 blocks
15/10/09 18:37:24 INFO MemoryStore: ensureFreeSpace(922) called with curMem=1013000, maxMem=278302556
15/10/09 18:37:24 INFO MemoryStore: Block input-0-1444396043800 stored as bytes in memory (estimated size 922.0 B, free 264.4 MB)
15/10/09 18:37:24 INFO BlockManagerInfo: Added input-0-1444396043800 in memory on webanalytics03:51843 (size: 922.0 B, free: 264.4 MB)
15/10/09 18:37:24 INFO BlockManagerMaster: Updated info of block input-0-1444396043800
15/10/09 18:37:24 WARN BlockManager: Block input-0-1444396043800 already exists on this machine; not re-adding it
15/10/09 18:37:24 INFO BlockGenerator: Pushed block input-0-1444396043800
Thanks in advance
Prateek
"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."
RE: Streaming Application Unable to get Stream from Kafka
Posted by "Prateek ." <pr...@aricent.com>.
Hi Terry,
Thanks a lot. It was the resource problem , Spark was able to get only one thread. It’s working fine now with local[*].
Cheers,
Prateek
From: Terry Hoo [mailto:hujie.eagle@gmail.com]
Sent: Saturday, October 10, 2015 9:51 AM
To: Prateek . <pr...@aricent.com>
Cc: user@spark.apache.org
Subject: Re: Streaming Application Unable to get Stream from Kafka
Hi Prateek,
How many cores (threads) do you assign to spark in local mode? It is very likely the local spark does not have enough resource to proceed. You can check http://yourip:4040 to check the details.
Thanks!
Terry
On Fri, Oct 9, 2015 at 10:34 PM, Prateek . <pr...@aricent.com>> wrote:
Hi All,
In my application I have a serializable class which is taking InputDStream from Kafka. The inputDStream contains JSON which is stored in serializable case class. Transformations are applied and saveToCassandra() is executed.
I was getting task not serializable exception , so I made the class serializable.
Now ,the application is working fine in standalone mode, but not able to receive data in local mode with the below mentioned log.
What is internally happening?, if anyone have some insights Please share!
Thank You in advance
Regards,
Prateek
From: Prateek .
Sent: Friday, October 09, 2015 6:55 PM
To: user@spark.apache.org<ma...@spark.apache.org>
Subject: Streaming Application Unable to get Stream from Kafka
Hi,
I have Spark Streaming application running with the following log on console, I don’t get any exception but I am not able to receive the data from Kafka Stream.
Can anyone please provide any insight what is happening with Spark Streaming. Is Receiver is not able to read the stream? How shall I debug it?
JobScheduler: Added jobs for time 1444396043000 ms
15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(922) called with curMem=1009312, maxMem=278302556
15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043000 stored as bytes in memory (estimated size 922.0 B, free 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043000 in memory on webanalytics03:51843 (size: 922.0 B, free: 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043000
15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043000 already exists on this machine; not re-adding it
15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043000
15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(926) called with curMem=1010234, maxMem=278302556
15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043200 stored as bytes in memory (estimated size 926.0 B, free 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043200 in memory on webanalytics03:51843 (size: 926.0 B, free: 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043200
15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043200 already exists on this machine; not re-adding it
15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043200
15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(923) called with curMem=1011160, maxMem=278302556
15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043400 stored as bytes in memory (estimated size 923.0 B, free 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043400 in memory on webanalytics03:51843 (size: 923.0 B, free: 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043400
15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043400 already exists on this machine; not re-adding it
15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043400
15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(917) called with curMem=1012083, maxMem=278302556
15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043600 stored as bytes in memory (estimated size 917.0 B, free 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043600 in memory on webanalytics03:51843 (size: 917.0 B, free: 264.4 MB)
15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block input-0-1444396043600
15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043600 already exists on this machine; not re-adding it
15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043600
15/10/09 18:37:24 INFO ReceiverTracker: Stream 0 received 5 blocks
15/10/09 18:37:24 INFO MemoryStore: ensureFreeSpace(922) called with curMem=1013000, maxMem=278302556
15/10/09 18:37:24 INFO MemoryStore: Block input-0-1444396043800 stored as bytes in memory (estimated size 922.0 B, free 264.4 MB)
15/10/09 18:37:24 INFO BlockManagerInfo: Added input-0-1444396043800 in memory on webanalytics03:51843 (size: 922.0 B, free: 264.4 MB)
15/10/09 18:37:24 INFO BlockManagerMaster: Updated info of block input-0-1444396043800
15/10/09 18:37:24 WARN BlockManager: Block input-0-1444396043800 already exists on this machine; not re-adding it
15/10/09 18:37:24 INFO BlockGenerator: Pushed block input-0-1444396043800
Thanks in advance
Prateek
"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."
"DISCLAIMER: This message is proprietary to Aricent and is intended solely for the use of the individual to whom it is addressed. It may contain privileged or confidential information and should not be circulated or used for any purpose other than for what it is intended. If you have received this message in error, please notify the originator immediately. If you are not the intended recipient, you are notified that you are strictly prohibited from using, copying, altering, or disclosing the contents of this message. Aricent accepts no responsibility for loss or damage arising from the use of the information transmitted by this email including damage from virus."
Re: Streaming Application Unable to get Stream from Kafka
Posted by Terry Hoo <hu...@gmail.com>.
Hi Prateek,
How many cores (threads) do you assign to spark in local mode? It is very
likely the local spark does not have enough resource to proceed. You can
check http://yourip:4040 to check the details.
Thanks!
Terry
On Fri, Oct 9, 2015 at 10:34 PM, Prateek . <pr...@aricent.com> wrote:
> Hi All,
>
>
>
> In my application I have a serializable class which is taking
> InputDStream from Kafka. The inputDStream contains JSON which is stored in
> serializable case class. Transformations are applied and saveToCassandra()
> is executed.
>
> I was getting task not serializable exception , so I made the class
> serializable.
>
>
>
> Now ,the application is working fine in standalone mode, but not able to
> receive data in local mode with the below mentioned log.
>
>
>
> What is internally happening?, if anyone have some insights Please share!
>
>
>
> Thank You in advance
>
> Regards,
>
> Prateek
>
>
>
>
>
> *From:* Prateek .
> *Sent:* Friday, October 09, 2015 6:55 PM
> *To:* user@spark.apache.org
> *Subject:* Streaming Application Unable to get Stream from Kafka
>
>
>
> Hi,
>
>
>
> I have Spark Streaming application running with the following log on
> console, I don’t get any exception but I am not able to receive the data
> from Kafka Stream.
>
>
>
> Can anyone please provide any insight what is happening with Spark
> Streaming. Is Receiver is not able to read the stream? How shall I debug
> it?
>
>
>
> JobScheduler: Added jobs for time 1444396043000 ms
>
> 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(922) called with
> curMem=1009312, maxMem=278302556
>
> 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043000 stored as
> bytes in memory (estimated size 922.0 B, free 264.4 MB)
>
> 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043000 in
> memory on webanalytics03:51843 (size: 922.0 B, free: 264.4 MB)
>
> 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block
> input-0-1444396043000
>
> 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043000 already
> exists on this machine; not re-adding it
>
> 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043000
>
> 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(926) called with
> curMem=1010234, maxMem=278302556
>
> 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043200 stored as
> bytes in memory (estimated size 926.0 B, free 264.4 MB)
>
> 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043200 in
> memory on webanalytics03:51843 (size: 926.0 B, free: 264.4 MB)
>
> 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block
> input-0-1444396043200
>
> 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043200 already
> exists on this machine; not re-adding it
>
> 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043200
>
> 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(923) called with
> curMem=1011160, maxMem=278302556
>
> 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043400 stored as
> bytes in memory (estimated size 923.0 B, free 264.4 MB)
>
> 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043400 in
> memory on webanalytics03:51843 (size: 923.0 B, free: 264.4 MB)
>
> 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block
> input-0-1444396043400
>
> 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043400 already
> exists on this machine; not re-adding it
>
> 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043400
>
> 15/10/09 18:37:23 INFO MemoryStore: ensureFreeSpace(917) called with
> curMem=1012083, maxMem=278302556
>
> 15/10/09 18:37:23 INFO MemoryStore: Block input-0-1444396043600 stored as
> bytes in memory (estimated size 917.0 B, free 264.4 MB)
>
> 15/10/09 18:37:23 INFO BlockManagerInfo: Added input-0-1444396043600 in
> memory on webanalytics03:51843 (size: 917.0 B, free: 264.4 MB)
>
> 15/10/09 18:37:23 INFO BlockManagerMaster: Updated info of block
> input-0-1444396043600
>
> 15/10/09 18:37:23 WARN BlockManager: Block input-0-1444396043600 already
> exists on this machine; not re-adding it
>
> 15/10/09 18:37:23 INFO BlockGenerator: Pushed block input-0-1444396043600
>
> 15/10/09 18:37:24 INFO ReceiverTracker: Stream 0 received 5 blocks
>
> 15/10/09 18:37:24 INFO MemoryStore: ensureFreeSpace(922) called with
> curMem=1013000, maxMem=278302556
>
> 15/10/09 18:37:24 INFO MemoryStore: Block input-0-1444396043800 stored as
> bytes in memory (estimated size 922.0 B, free 264.4 MB)
>
> 15/10/09 18:37:24 INFO BlockManagerInfo: Added input-0-1444396043800 in
> memory on webanalytics03:51843 (size: 922.0 B, free: 264.4 MB)
>
> 15/10/09 18:37:24 INFO BlockManagerMaster: Updated info of block
> input-0-1444396043800
>
> 15/10/09 18:37:24 WARN BlockManager: Block input-0-1444396043800 already
> exists on this machine; not re-adding it
>
> 15/10/09 18:37:24 INFO BlockGenerator: Pushed block input-0-1444396043800
>
>
>
> Thanks in advance
>
> Prateek
>
> "DISCLAIMER: This message is proprietary to Aricent and is intended solely
> for the use of the individual to whom it is addressed. It may contain
> privileged or confidential information and should not be circulated or used
> for any purpose other than for what it is intended. If you have received
> this message in error, please notify the originator immediately. If you are
> not the intended recipient, you are notified that you are strictly
> prohibited from using, copying, altering, or disclosing the contents of
> this message. Aricent accepts no responsibility for loss or damage arising
> from the use of the information transmitted by this email including damage
> from virus."
>