You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jeff Nadler <jn...@srcginc.com> on 2015/01/19 07:04:17 UTC

Streaming with Java: Expected ReduceByWindow to Return JavaDStream

Can anyone tell me if my expectations are sane?

I'm trying to do a reduceByWindow using the 3-arg signature (not providing
an inverse reduce function):

JavaDStream<whatevs> reducedStream = messages.reduceByWindow((x, y) ->
reduce(x, y), Durations.seconds(5), Durations.seconds(5));

This isn't building; looks like it's returning DStream not JavaDStream.

>From JavaDStreamLike.scala, looks like this sig returns DStream, the
4-arg sig with the inverse reduce returns JavaDStream.

def reduceByWindow(
    reduceFunc: (T, T) => T,
    windowDuration: Duration,
    slideDuration: Duration
  ): DStream[T] = {
  dstream.reduceByWindow(reduceFunc, windowDuration, slideDuration)
}

So I'm just a noob.  Is this a bug or am I missing something?

Thanks!

Jeff Nadler

Re: Streaming with Java: Expected ReduceByWindow to Return JavaDStream

Posted by Jeff Nadler <jn...@srcginc.com>.
For anyone who finds this later, looks like Jerry already took care of this
here:  https://issues.apache.org/jira/browse/SPARK-5315

Thanks!

On Sun, Jan 18, 2015 at 10:28 PM, Shao, Saisai <sa...@intel.com>
wrote:

>  Hi Jeff,
>
>
>
> From my understanding it seems more like a bug, since JavaDStreamLike is
> used for Java code, return a Scala DStream is not reasonable. You can fix
> this by submitting a PR, or I can help you to fix this.
>
>
>
> Thanks
>
> Jerry
>
>
>
> *From:* Jeff Nadler [mailto:jnadler@srcginc.com]
> *Sent:* Monday, January 19, 2015 2:04 PM
> *To:* user@spark.apache.org
> *Subject:* Streaming with Java: Expected ReduceByWindow to Return
> JavaDStream
>
>
>
>
>
> Can anyone tell me if my expectations are sane?
>
>
>
> I'm trying to do a reduceByWindow using the 3-arg signature (not providing
> an inverse reduce function):
>
> JavaDStream<whatevs> reducedStream = messages.reduceByWindow((x, y) -> reduce(x, y), Durations.*seconds*(5), Durations.*seconds*(5));
>
> This isn't building; looks like it's returning DStream not JavaDStream.
>
> From JavaDStreamLike.scala, looks like this sig returns DStream, the 4-arg sig with the inverse reduce returns JavaDStream.
>
> *def *reduceByWindow(
>     reduceFunc: (T, T) => T,
>     windowDuration: Duration,
>     slideDuration: Duration
>   ): DStream[T] = {
>   dstream.reduceByWindow(reduceFunc, windowDuration, slideDuration)
> }
>
> So I'm just a noob.  Is this a bug or am I missing something?
>
> Thanks!
>
> Jeff Nadler
>
>
>
>

RE: Streaming with Java: Expected ReduceByWindow to Return JavaDStream

Posted by "Shao, Saisai" <sa...@intel.com>.
Hi Jeff,

>From my understanding it seems more like a bug, since JavaDStreamLike is used for Java code, return a Scala DStream is not reasonable. You can fix this by submitting a PR, or I can help you to fix this.

Thanks
Jerry

From: Jeff Nadler [mailto:jnadler@srcginc.com]
Sent: Monday, January 19, 2015 2:04 PM
To: user@spark.apache.org
Subject: Streaming with Java: Expected ReduceByWindow to Return JavaDStream


Can anyone tell me if my expectations are sane?

I'm trying to do a reduceByWindow using the 3-arg signature (not providing an inverse reduce function):

JavaDStream<whatevs> reducedStream = messages.reduceByWindow((x, y) -> reduce(x, y), Durations.seconds(5), Durations.seconds(5));

This isn't building; looks like it's returning DStream not JavaDStream.

>From JavaDStreamLike.scala, looks like this sig returns DStream, the 4-arg sig with the inverse reduce returns JavaDStream.

def reduceByWindow(
    reduceFunc: (T, T) => T,
    windowDuration: Duration,
    slideDuration: Duration
  ): DStream[T] = {
  dstream.reduceByWindow(reduceFunc, windowDuration, slideDuration)
}

So I'm just a noob.  Is this a bug or am I missing something?

Thanks!

Jeff Nadler