convert List to dstream

[Josh J <jo...@gmail.com>]

Hi,

I have some data generated by some utilities that returns the results as
a List<String>. I would like to join this with a Dstream of strings. How
can I do this? I tried the following though get scala compiler errors

val list_scalaconverted = ssc.sparkContext.parallelize(listvalues.toArray())
   list_queue.add(list_scalaconverted)
    val list_stream = ssc.queueStream(list_queue)

 found   : org.apache.spark.rdd.RDD[Object]

 required: org.apache.spark.rdd.RDD[String]

Note: Object >: String, but class RDD is invariant in type T.

You may wish to define T as -T instead. (SLS 4.5)


and


 found   : java.util.LinkedList[org.apache.spark.rdd.RDD[String]]

 required: scala.collection.mutable.Queue[org.apache.spark.rdd.RDD[?]]


Thanks,

Josh

Re: convert List to dstream

[Tobias Pfeiffer <tg...@preferred.jp>]

Josh,

On Tue, Nov 11, 2014 at 7:43 AM, Josh J <jo...@gmail.com> wrote:
>
> I have some data generated by some utilities that returns the results as
> a List<String>. I would like to join this with a Dstream of strings. How
> can I do this? I tried the following though get scala compiler errors
>
> val list_scalaconverted =
> ssc.sparkContext.parallelize(listvalues.toArray())
>

Your `listvalues` seems to be a java.util.List, not a
scala.collection.immutable.List, right? In that case, toArray() will return
a Array[Object], not an Array[String], which leads to the error you see.
Have a look at
http://www.scala-lang.org/api/current/index.html#scala.collection.JavaConversions$
and convert your Java list to a Scala list.

Tobias