Spark Streaming NeteorkReceiver problems

[zzzzzqf12345 <zz...@gmail.com>]

hi,
here is problem description, I write a custom networkreceiver to receive
image data from camera. I had confirmed all the data received correctly.

1)when data received, only the networkreceiver node run at full speed, while
other nodes keep idle, my spark cluster has 6 nodes.

2)And every image data is calculated many times, which I expected to
calculate once.

3)How to distribute tasks to the whole cluster?

I tried dstream.repartitioin.map(), problems in 2) is still existed ,and
sometimes job faidled.

here is spark configration,
conf.set("spark.executor.memory", "3g");
conf.set("spark.shuffle.netty.connect.timeout", "300000")
conf.set("spark.storage.blockManagerSlaveTimeoutMs", "300000")
conf.set("spark.akka.timeout","600")
conf.set("spark.akka.threads","8")

node : 8 cores, mem 6G.
NetWork : 1Gb/bps

Any suggestions will be appreciated. 

thanks,
QingFeng



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-NeteorkReceiver-problems-tp7109.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.