You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Adrian Mocanu <am...@verticalscope.com> on 2014/03/27 16:57:00 UTC

StreamingContext.transform on a DStream

Found this transform fn in StreamingContext which takes in a DStream[_] and a function which acts on each of its RDDs
Unfortunately I can't figure out how to transform my DStream[(String,Int)] into DStream[_]


/*** Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams. */
  def transform[T: ClassTag](
      dstreams: Seq[DStream[_]],
      transformFunc: (Seq[RDD[_]], Time) => RDD[T]
    ): DStream[T] = {
    new TransformedDStream[T](dstreams, sparkContext.clean(transformFunc))
  }

-Adrian

RE: StreamingContext.transform on a DStream

Posted by Adrian Mocanu <am...@verticalscope.com>.

Please disregard I didn't see the Seq wrapper.

From: Adrian Mocanu [mailto:amocanu@verticalscope.com]
Sent: March-27-14 11:57 AM
To: user@spark.incubator.apache.org
Subject: StreamingContext.transform on a DStream

Found this transform fn in StreamingContext which takes in a DStream[_] and a function which acts on each of its RDDs
Unfortunately I can't figure out how to transform my DStream[(String,Int)] into DStream[_]


/*** Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams. */
  def transform[T: ClassTag](
      dstreams: Seq[DStream[_]],
      transformFunc: (Seq[RDD[_]], Time) => RDD[T]
    ): DStream[T] = {
    new TransformedDStream[T](dstreams, sparkContext.clean(transformFunc))
  }

-Adrian