You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by spr <sp...@yarcdata.com> on 2014/10/29 23:15:41 UTC
what does DStream.union() do?
The documentation at
https://spark.apache.org/docs/1.0.0/api/scala/index.html#org.apache.spark.streaming.dstream.DStream
describes the union() method as
"Return a new DStream by unifying data of another DStream with this
DStream."
Can somebody provide a clear definition of what "unifying" means in this
context? Does it append corresponding elements together? Inside a wider
tuple if need be?
I'm hoping for something clear enough that it could just be added to the doc
page if the developers so chose.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/what-does-DStream-union-do-tp17673.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: what does DStream.union() do?
Posted by Holden Karau <ho...@pigscanfly.ca>.
On Wed, Oct 29, 2014 at 3:39 PM, spr <sp...@yarcdata.com> wrote:
> I need more precision to understand. If the elements of one DStream/RDD
> are
> (String) and the elements of the other are (Time, Int), what does "union"
> mean? I'm hoping for (String, Time, Int) but that appears optimistic. :)
>
It won't compile.
> Do the elements have to be of homogeneous type?
>
Yes. From the scaladoc (
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.dstream.DStream
) you can see DStreams are generic/templated on a type (T) and the union
function works on a DStream of the same templated type. If you have
hetrogeneous data you can first map each DStream it to a case class with
options or try something like
http://stackoverflow.com/questions/3508077/does-scala-have-type-disjunction-union-types
>
>
> Holden Karau wrote
> > The union function simply returns a DStream with the elements from both.
> > This is the same behavior as when we call union on RDDs :) (You can think
> > of union as similar to the union operator on sets except without the
> > unique
> > element restrictions).
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/what-does-DStream-union-do-tp17673p17682.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
Cell : 425-233-8271
Re: what does DStream.union() do?
Posted by spr <sp...@yarcdata.com>.
I need more precision to understand. If the elements of one DStream/RDD are
(String) and the elements of the other are (Time, Int), what does "union"
mean? I'm hoping for (String, Time, Int) but that appears optimistic. :)
Do the elements have to be of homogeneous type?
Holden Karau wrote
> The union function simply returns a DStream with the elements from both.
> This is the same behavior as when we call union on RDDs :) (You can think
> of union as similar to the union operator on sets except without the
> unique
> element restrictions).
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/what-does-DStream-union-do-tp17673p17682.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: what does DStream.union() do?
Posted by Holden Karau <ho...@pigscanfly.ca>.
The union function simply returns a DStream with the elements from both.
This is the same behavior as when we call union on RDDs :) (You can think
of union as similar to the union operator on sets except without the unique
element restrictions).
On Wed, Oct 29, 2014 at 3:15 PM, spr <sp...@yarcdata.com> wrote:
> The documentation at
>
> https://spark.apache.org/docs/1.0.0/api/scala/index.html#org.apache.spark.streaming.dstream.DStream
> describes the union() method as
>
> "Return a new DStream by unifying data of another DStream with this
> DStream."
>
> Can somebody provide a clear definition of what "unifying" means in this
> context? Does it append corresponding elements together? Inside a wider
> tuple if need be?
>
> I'm hoping for something clear enough that it could just be added to the
> doc
> page if the developers so chose.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/what-does-DStream-union-do-tp17673.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
Cell : 425-233-8271