You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shuai Zhang <sm...@yahoo.com.INVALID> on 2015/06/16 17:15:43 UTC
The problem when share data inside Dstream
Hello guys,
I faced one problem that I cannot pass my data inside rdd partition when I was trying to develop spark streaming feature.I'm the newcomer of Spark, could you please give me any suggestion on this problem?
The figure in the attachment is the code I used in my program:
After I run my code, I found these logs:Inside foreachRDD scope I can get the right indexGeneratorMap value which includes right kva instance, "CassandraAccessDatastax@5adc7292"
15/06/16 12:07:35 INFO streaming.AbstractSparkStreamingMain: [RDD][Index Generator Map]: {class com.worksap.company.framework.autoindex.edp.job.SparkStreamingTestEntity=AbstractIndexGenerator{kva=com.worksap.company.access.cassandra.CassandraAccessDatastax@5adc7292, indexName='spark_streaming_test_index'}}
But in the foreachPartition scope, I found my kva instance is null. I totally have no idea about it.
15/06/16 12:07:28 INFO streaming.AbstractSparkStreamingMain: [Index Generator Map]: {class com.worksap.company.framework.autoindex.edp.job.SparkStreamingTestEntity=AbstractIndexGenerator{kva=null, indexName='spark_streaming_test_index'}}
If someone can give me any tips, I will really appreciate your help.
Thanks in advance.
Best Regards,
--
Gabriel Zhang