Design documents for consolidated DataStream API

[Stephan Ewen <se...@apache.org>]

Hi all!

As many of you know, there are a ongoing efforts to consolidate the
streaming API for the next release, and then graduate it (from beta status).

In the process of this consolidation, we want to achieve the following
goals.

 - Make the code more robust and simplify it in parts

 - Clearly define the semantics of the constructs.

 - Prepare it for support of more advanced concepts, like partitionable
state, and event time.

 - Cut support for certain corner cases that were prototyped, but turned
out to be not efficiently doable


Based on prior discussions on the mailing list, Aljoscha and me drafted the
design documents below, which outline how the consolidated API would like.
We focused in constructs, time, and window semantics.


Design document on how to restructure the Streaming API:
https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams

Design document on definitions of time, order, and the resulting semantics:
https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams



Note: The design of the interfaces and concepts for advanced state in
functions is not in here. That is part of a separate design discussion and
orthogonal to the designs drafted here.


Please have a look and voice questions and concerns. Since we should not
break the streaming API more than once, we should make sure this
consolidation brings it into the shape we want it to be in.


Greetings,
Stephan

Re: Design documents for consolidated DataStream API

[Kostas Tzoumas <kt...@apache.org>]

I think the though was to explicitly not have the same terminology as the
batch API to not confuse people.

But this is a minor naming issue IMO.

On Tue, Jul 14, 2015 at 12:40 PM, Gyula Fóra <gy...@gmail.com> wrote:

> I see your point, reduceByKey is much clearer.
>
> The question is whether we want to introduce this inconsistency across the
> two api-s or stick with what we have.
> On Tue, Jul 14, 2015 at 10:57 AM Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > I agree, the groupBy, in the batch API is misleading, since a
> > ds.groupBy().reduce() does not really build any groups, it is really a
> > ds.keyBy().reduceByKey(). In the streaming API we can still fix this,
> IMHO.
> >
> > On Tue, 14 Jul 2015 at 10:56 Stephan Ewen <se...@apache.org> wrote:
> >
> > > It is not a bit different than the batch API, because streaming
> semantics
> > > are a bit different ;-)
> > >
> > > One good thing is that we can make things better that were sub-optimal
> in
> > > the Batch API.
> > >
> > > On Tue, Jul 14, 2015 at 10:55 AM, Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > > > keyBy() does not do any grouping. Grouping in streams in not defined
> > > > without windows.
> > > >
> > > > On Tue, Jul 14, 2015 at 10:48 AM, Gyula Fóra <gy...@gmail.com>
> > > wrote:
> > > >
> > > >> If we only want to have either keyBy or groupBy, why not keep
> groupBy?
> > > >> That
> > > >> would be more consistent with the batch api.
> > > >> On Tue, Jul 14, 2015 at 10:35 AM Stephan Ewen <se...@apache.org>
> > wrote:
> > > >>
> > > >> > Concerning your comments:
> > > >> >
> > > >> > 1) In the new design, there is no grouping without windowing. The
> > > >> > KeyedDataStream subsumes the grouping and key-ing for partitioned
> > > state.
> > > >> >
> > > >> >     The keyBy() + window() makes a parallel grouped window
> > > >> >     keyBy() alone allows access to partitioned state.
> > > >> >
> > > >> >     My thought was that this is simpler, because it needs not
> > > groupBy()
> > > >> and
> > > >> > keyBy(), but one construct to handle both cases.
> > > >> >
> > > >> > 2) The discretization is a rough thought and is nothing for the
> > short
> > > >> term.
> > > >> > It totally needs more thoughts. I put it there to have it as a
> > sketch
> > > >> for
> > > >> > how to evolve this.
> > > >> >
> > > >> >     The idea is of course to not have a single data set, but a
> > series
> > > of
> > > >> > data set. In each discrete time slice, the data set can be treated
> > > like
> > > >> a
> > > >> > regular data set.
> > > >> >
> > > >> >     Let's kick off a separate design for the discretization. Joins
> > are
> > > >> good
> > > >> > to talk about (data sets can be joined with data set), and I am
> sure
> > > >> there
> > > >> > are more questions coming up.
> > > >> >
> > > >> >
> > > >> > Does that make sense?
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Tue, Jul 14, 2015 at 10:05 AM, Gyula Fóra <
> gyula.fora@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > I think Marton has some good points here.
> > > >> > >
> > > >> > > 1) Is KeyedDataStream a better name if this is only a renaming?
> > > >> > >
> > > >> > > 2) the discretize semantics is unclear indeed. Are we operating
> > on a
> > > >> > single
> > > >> > > or sequence of datasets? If the latter why not call it something
> > > else
> > > >> > > (dstream). How are joins and other binary operators defined for
> > > >> different
> > > >> > > discretizations etc.
> > > >> > > On Mon, Jul 13, 2015 at 7:37 PM Márton Balassi <
> > mbalassi@apache.org
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Generally I agree with the new design. Two concerns:
> > > >> > > >
> > > >> > > > 1) Does KeyedDataStream replace GroupedDataStream or is it the
> > > >> latter a
> > > >> > > > special case of the former?
> > > >> > > >
> > > >> > > > The KeyedDataStream as described in the design document is a
> bit
> > > >> > unclear
> > > >> > > > for me. It lists the following usages:
> > > >> > > >   a) It is the first step in building a window stream, on top
> of
> > > >> which
> > > >> > > the
> > > >> > > > grouped/windowed aggregation and reduce-style function can be
> > > >> applied
> > > >> > > >   b) It allows to use the "by-key" state of functions. Here,
> > every
> > > >> > record
> > > >> > > > has access to a state that is scoped by its key. Key-scoped
> > state
> > > >> can
> > > >> > be
> > > >> > > > automatically redistributed and repartitioned.
> > > >> > > >
> > > >> > > > The code snippet describes a use case where the computation
> and
> > > the
> > > >> > > access
> > > >> > > > of the state is used the way currently the GroupedDataStream
> > > should
> > > >> > > work. I
> > > >> > > > suppose this is the example for case b). Would case a) also
> > window
> > > >> > > elements
> > > >> > > > by key? If yes, then this is practically a renaming and
> > > enhancement
> > > >> of
> > > >> > > the
> > > >> > > > GroupedDataStream functionality with keyed state. Then the
> > > >> > > > StreamExecutionEnvironment.createKeyedStream(Partitioner,
> > > >> > > > KeySelector)construction does not make much sense as the user
> > only
> > > >> > > operates
> > > >> > > > within the scope of the keyselector and not the partitioner
> > > anyway.
> > > >> > > >
> > > >> > > > I personally think KeyedDataStream as a name does not
> > necessarily
> > > >> > suggest
> > > >> > > > that the records are grouped by key, it only suggests
> > partitioning
> > > >> by
> > > >> > > key -
> > > >> > > > at least for me. :)
> > > >> > > >
> > > >> > > > 2) The API for discretization is not convenient IMHO
> > > >> > > >
> > > >> > > > The discretization part declares that the output of
> > > >> > > DataStream.discretize()
> > > >> > > > is a sequence of DataSets. I love this approach, but then in
> the
> > > >> code
> > > >> > > > snippet the return value of this function is simply a DataSet
> > and
> > > >> uses
> > > >> > it
> > > >> > > > as such. The take home message of that code is the following:
> > this
> > > >> is
> > > >> > > > actually the way you would like to program on these sequence
> of
> > > >> > DataSets,
> > > >> > > > most probably you would like to do the same with each of them.
> > If
> > > >> that
> > > >> > is
> > > >> > > > the case we should provide a nice utility for that. I think
> > Spark
> > > >> > > > Streaming's DStream.foreachRDD() is fairly useful for this
> > > purpose.
> > > >> > > >
> > > >> > > > On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <
> > gyula.fora@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > +1
> > > >> > > > > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <
> > sewen@apache.org>
> > > >> > wrote:
> > > >> > > > >
> > > >> > > > > > If naming is the only concern, then we should go ahead,
> > > because
> > > >> we
> > > >> > > can
> > > >> > > > > > change names easily (before the release).
> > > >> > > > > >
> > > >> > > > > > In fact, I don't think it leaves a bad impression. Global
> > > >> windows
> > > >> > are
> > > >> > > > > > non-parallel windows. There are also parallel windows.
> Pick
> > > what
> > > >> > you
> > > >> > > > need
> > > >> > > > > > and what works.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <
> > > >> gyula.fora@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > I think we agree on everything its more of a naming
> issue
> > :)
> > > >> > > > > > >
> > > >> > > > > > > I thought it might be misleading that global time
> windows
> > > are
> > > >> > > > > > > "non-parallel" windows. We dont want to give a bad
> > > impression.
> > > >> > > (Also
> > > >> > > > we
> > > >> > > > > > > dont want them to think that every global window is
> > parallel
> > > >> but
> > > >> > > > thats
> > > >> > > > > > not
> > > >> > > > > > > a problem here)
> > > >> > > > > > >
> > > >> > > > > > > Gyula
> > > >> > > > > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <
> > > >> sewen@apache.org>
> > > >> > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Okay, what is missing about the windowing in your
> > opinion?
> > > >> > > > > > > >
> > > >> > > > > > > > The core points of the document are:
> > > >> > > > > > > >
> > > >> > > > > > > >   - The parallel windows are per group only.
> > > >> > > > > > > >
> > > >> > > > > > > >   - The implementation of the parallel windows holds
> > > window
> > > >> > data
> > > >> > > in
> > > >> > > > > the
> > > >> > > > > > > > group buffers.
> > > >> > > > > > > >
> > > >> > > > > > > >   - The global windows are non-parallel. May have
> > parallel
> > > >> > > > > > > pre-aggregation,
> > > >> > > > > > > > if they are time windows.
> > > >> > > > > > > >
> > > >> > > > > > > >   - Time may be operator time (timer thread), or
> > watermark
> > > >> > time.
> > > >> > > > > > > Watermark
> > > >> > > > > > > > time can refer to ingress or event time.
> > > >> > > > > > > >
> > > >> > > > > > > >   - Windows that do not pre-aggregate may require
> > elements
> > > >> in
> > > >> > > > order.
> > > >> > > > > > Not
> > > >> > > > > > > > part of the first prototype.
> > > >> > > > > > > >
> > > >> > > > > > > > Do we agree on those points?
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <
> > > >> > > gyula.fora@gmail.com>
> > > >> > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > In general I like it, although the main difference
> > > between
> > > >> > the
> > > >> > > > > > current
> > > >> > > > > > > > and
> > > >> > > > > > > > > the new one is the windowing and that is still not
> > very
> > > >> > clear.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Where do we have the full stream time windows for
> > > >> > > instance?(which
> > > >> > > > > is
> > > >> > > > > > > > > parallel but not keyed)
> > > >> > > > > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> > > >> > > > > > aljoscha@apache.org>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > +1 I like it as well.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <
> > > >> > > > ktzoumas@apache.org
> > > >> > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > +1 from my side
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <
> > > >> > > > > sewen@apache.org>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Do we have consensus on these designs?
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > If we have, we should get to implementing this
> > > soon,
> > > >> > > > because
> > > >> > > > > > > > > basically
> > > >> > > > > > > > > > > all
> > > >> > > > > > > > > > > > streaming patches will have to be revisited in
> > > >> light of
> > > >> > > > > this...
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> > > >> > > > > > gyula.fora@gmail.com
> > > >> > > > > > > >
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > You are right thats an important issue.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > And I think we should also do some renaming
> > with
> > > >> the
> > > >> > > > > > > "iterations"
> > > >> > > > > > > > > > > because
> > > >> > > > > > > > > > > > > they are not really iterations like in the
> > batch
> > > >> case
> > > >> > > and
> > > >> > > > > it
> > > >> > > > > > > > might
> > > >> > > > > > > > > > > > confuse
> > > >> > > > > > > > > > > > > some users.
> > > >> > > > > > > > > > > > > Maybe we can call them loops or cycles and
> > > rename
> > > >> the
> > > >> > > api
> > > >> > > > > > calls
> > > >> > > > > > > > to
> > > >> > > > > > > > > > make
> > > >> > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > more intuitive what happens. It is really
> > just a
> > > >> > cyclic
> > > >> > > > > > > dataflow.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Aljoscha Krettek <al...@apache.org> ezt
> > írta
> > > >> > > > (időpont:
> > > >> > > > > > > 2015.
> > > >> > > > > > > > > júl.
> > > >> > > > > > > > > > > 7.,
> > > >> > > > > > > > > > > > > K,
> > > >> > > > > > > > > > > > > 15:35):
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Hi,
> > > >> > > > > > > > > > > > > > I just noticed that we don't have anything
> > > about
> > > >> > how
> > > >> > > > > > > iterations
> > > >> > > > > > > > > and
> > > >> > > > > > > > > > > > > > timestamps/watermarks should interact.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > > > Aljoscha
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <
> > > >> > > > > sewen@apache.org
> > > >> > > > > > >
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Hi all!
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > As many of you know, there are a ongoing
> > > >> efforts
> > > >> > to
> > > >> > > > > > > > consolidate
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > streaming API for the next release, and
> > then
> > > >> > > graduate
> > > >> > > > > it
> > > >> > > > > > > > (from
> > > >> > > > > > > > > > beta
> > > >> > > > > > > > > > > > > > > status).
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > In the process of this consolidation, we
> > > want
> > > >> to
> > > >> > > > > achieve
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > following
> > > >> > > > > > > > > > > > > > > goals.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >  - Make the code more robust and
> simplify
> > it
> > > >> in
> > > >> > > parts
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >  - Clearly define the semantics of the
> > > >> > constructs.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >  - Prepare it for support of more
> advanced
> > > >> > > concepts,
> > > >> > > > > like
> > > >> > > > > > > > > > > > partitionable
> > > >> > > > > > > > > > > > > > > state, and event time.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >  - Cut support for certain corner cases
> > that
> > > >> were
> > > >> > > > > > > prototyped,
> > > >> > > > > > > > > but
> > > >> > > > > > > > > > > > > turned
> > > >> > > > > > > > > > > > > > > out to be not efficiently doable
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Based on prior discussions on the
> mailing
> > > >> list,
> > > >> > > > > Aljoscha
> > > >> > > > > > > and
> > > >> > > > > > > > me
> > > >> > > > > > > > > > > > drafted
> > > >> > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > design documents below, which outline
> how
> > > the
> > > >> > > > > > consolidated
> > > >> > > > > > > > API
> > > >> > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > like.
> > > >> > > > > > > > > > > > > > > We focused in constructs, time, and
> window
> > > >> > > semantics.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Design document on how to restructure
> the
> > > >> > Streaming
> > > >> > > > > API:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Design document on definitions of time,
> > > order,
> > > >> > and
> > > >> > > > the
> > > >> > > > > > > > > resulting
> > > >> > > > > > > > > > > > > > semantics:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Note: The design of the interfaces and
> > > >> concepts
> > > >> > for
> > > >> > > > > > > advanced
> > > >> > > > > > > > > > state
> > > >> > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > functions is not in here. That is part
> of
> > a
> > > >> > > separate
> > > >> > > > > > design
> > > >> > > > > > > > > > > > discussion
> > > >> > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > orthogonal to the designs drafted here.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Please have a look and voice questions
> and
> > > >> > > concerns.
> > > >> > > > > > Since
> > > >> > > > > > > we
> > > >> > > > > > > > > > > should
> > > >> > > > > > > > > > > > > not
> > > >> > > > > > > > > > > > > > > break the streaming API more than once,
> we
> > > >> should
> > > >> > > > make
> > > >> > > > > > sure
> > > >> > > > > > > > > this
> > > >> > > > > > > > > > > > > > > consolidation brings it into the shape
> we
> > > >> want it
> > > >> > > to
> > > >> > > > be
> > > >> > > > > > in.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Greetings,
> > > >> > > > > > > > > > > > > > > Stephan
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Márton Balassi <ba...@gmail.com>]

Ok, thanks for the clarification. Let us try to document it in a way that
those thoughts are reflected then. Discretization will not happen upfront
we can wait with that.

On Tue, Jul 14, 2015 at 12:49 PM, Stephan Ewen <se...@apache.org> wrote:

> There is no inconsistency between the Batch and Streaming API. They have
> different semantics - the batch API is implicitly always windowed.
>
> There is a naming difference between the two APIs.
>
> There is a strong inconsistency within the Streaming API right now.
> Grouping and aggregating without windows is plain dangerous in streaming.
> It either blows up or is undefined in its behavior.
>
>
>
> On Tue, Jul 14, 2015 at 12:40 PM, Gyula Fóra <gy...@gmail.com> wrote:
>
> > I see your point, reduceByKey is much clearer.
> >
> > The question is whether we want to introduce this inconsistency across
> the
> > two api-s or stick with what we have.
> > On Tue, Jul 14, 2015 at 10:57 AM Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> > > I agree, the groupBy, in the batch API is misleading, since a
> > > ds.groupBy().reduce() does not really build any groups, it is really a
> > > ds.keyBy().reduceByKey(). In the streaming API we can still fix this,
> > IMHO.
> > >
> > > On Tue, 14 Jul 2015 at 10:56 Stephan Ewen <se...@apache.org> wrote:
> > >
> > > > It is not a bit different than the batch API, because streaming
> > semantics
> > > > are a bit different ;-)
> > > >
> > > > One good thing is that we can make things better that were
> sub-optimal
> > in
> > > > the Batch API.
> > > >
> > > > On Tue, Jul 14, 2015 at 10:55 AM, Stephan Ewen <se...@apache.org>
> > wrote:
> > > >
> > > > > keyBy() does not do any grouping. Grouping in streams in not
> defined
> > > > > without windows.
> > > > >
> > > > > On Tue, Jul 14, 2015 at 10:48 AM, Gyula Fóra <gyula.fora@gmail.com
> >
> > > > wrote:
> > > > >
> > > > >> If we only want to have either keyBy or groupBy, why not keep
> > groupBy?
> > > > >> That
> > > > >> would be more consistent with the batch api.
> > > > >> On Tue, Jul 14, 2015 at 10:35 AM Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >>
> > > > >> > Concerning your comments:
> > > > >> >
> > > > >> > 1) In the new design, there is no grouping without windowing.
> The
> > > > >> > KeyedDataStream subsumes the grouping and key-ing for
> partitioned
> > > > state.
> > > > >> >
> > > > >> >     The keyBy() + window() makes a parallel grouped window
> > > > >> >     keyBy() alone allows access to partitioned state.
> > > > >> >
> > > > >> >     My thought was that this is simpler, because it needs not
> > > > groupBy()
> > > > >> and
> > > > >> > keyBy(), but one construct to handle both cases.
> > > > >> >
> > > > >> > 2) The discretization is a rough thought and is nothing for the
> > > short
> > > > >> term.
> > > > >> > It totally needs more thoughts. I put it there to have it as a
> > > sketch
> > > > >> for
> > > > >> > how to evolve this.
> > > > >> >
> > > > >> >     The idea is of course to not have a single data set, but a
> > > series
> > > > of
> > > > >> > data set. In each discrete time slice, the data set can be
> treated
> > > > like
> > > > >> a
> > > > >> > regular data set.
> > > > >> >
> > > > >> >     Let's kick off a separate design for the discretization.
> Joins
> > > are
> > > > >> good
> > > > >> > to talk about (data sets can be joined with data set), and I am
> > sure
> > > > >> there
> > > > >> > are more questions coming up.
> > > > >> >
> > > > >> >
> > > > >> > Does that make sense?
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Tue, Jul 14, 2015 at 10:05 AM, Gyula Fóra <
> > gyula.fora@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > I think Marton has some good points here.
> > > > >> > >
> > > > >> > > 1) Is KeyedDataStream a better name if this is only a
> renaming?
> > > > >> > >
> > > > >> > > 2) the discretize semantics is unclear indeed. Are we
> operating
> > > on a
> > > > >> > single
> > > > >> > > or sequence of datasets? If the latter why not call it
> something
> > > > else
> > > > >> > > (dstream). How are joins and other binary operators defined
> for
> > > > >> different
> > > > >> > > discretizations etc.
> > > > >> > > On Mon, Jul 13, 2015 at 7:37 PM Márton Balassi <
> > > mbalassi@apache.org
> > > > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Generally I agree with the new design. Two concerns:
> > > > >> > > >
> > > > >> > > > 1) Does KeyedDataStream replace GroupedDataStream or is it
> the
> > > > >> latter a
> > > > >> > > > special case of the former?
> > > > >> > > >
> > > > >> > > > The KeyedDataStream as described in the design document is a
> > bit
> > > > >> > unclear
> > > > >> > > > for me. It lists the following usages:
> > > > >> > > >   a) It is the first step in building a window stream, on
> top
> > of
> > > > >> which
> > > > >> > > the
> > > > >> > > > grouped/windowed aggregation and reduce-style function can
> be
> > > > >> applied
> > > > >> > > >   b) It allows to use the "by-key" state of functions. Here,
> > > every
> > > > >> > record
> > > > >> > > > has access to a state that is scoped by its key. Key-scoped
> > > state
> > > > >> can
> > > > >> > be
> > > > >> > > > automatically redistributed and repartitioned.
> > > > >> > > >
> > > > >> > > > The code snippet describes a use case where the computation
> > and
> > > > the
> > > > >> > > access
> > > > >> > > > of the state is used the way currently the GroupedDataStream
> > > > should
> > > > >> > > work. I
> > > > >> > > > suppose this is the example for case b). Would case a) also
> > > window
> > > > >> > > elements
> > > > >> > > > by key? If yes, then this is practically a renaming and
> > > > enhancement
> > > > >> of
> > > > >> > > the
> > > > >> > > > GroupedDataStream functionality with keyed state. Then the
> > > > >> > > > StreamExecutionEnvironment.createKeyedStream(Partitioner,
> > > > >> > > > KeySelector)construction does not make much sense as the
> user
> > > only
> > > > >> > > operates
> > > > >> > > > within the scope of the keyselector and not the partitioner
> > > > anyway.
> > > > >> > > >
> > > > >> > > > I personally think KeyedDataStream as a name does not
> > > necessarily
> > > > >> > suggest
> > > > >> > > > that the records are grouped by key, it only suggests
> > > partitioning
> > > > >> by
> > > > >> > > key -
> > > > >> > > > at least for me. :)
> > > > >> > > >
> > > > >> > > > 2) The API for discretization is not convenient IMHO
> > > > >> > > >
> > > > >> > > > The discretization part declares that the output of
> > > > >> > > DataStream.discretize()
> > > > >> > > > is a sequence of DataSets. I love this approach, but then in
> > the
> > > > >> code
> > > > >> > > > snippet the return value of this function is simply a
> DataSet
> > > and
> > > > >> uses
> > > > >> > it
> > > > >> > > > as such. The take home message of that code is the
> following:
> > > this
> > > > >> is
> > > > >> > > > actually the way you would like to program on these sequence
> > of
> > > > >> > DataSets,
> > > > >> > > > most probably you would like to do the same with each of
> them.
> > > If
> > > > >> that
> > > > >> > is
> > > > >> > > > the case we should provide a nice utility for that. I think
> > > Spark
> > > > >> > > > Streaming's DStream.foreachRDD() is fairly useful for this
> > > > purpose.
> > > > >> > > >
> > > > >> > > > On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <
> > > gyula.fora@gmail.com
> > > > >
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > > > +1
> > > > >> > > > > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <
> > > sewen@apache.org>
> > > > >> > wrote:
> > > > >> > > > >
> > > > >> > > > > > If naming is the only concern, then we should go ahead,
> > > > because
> > > > >> we
> > > > >> > > can
> > > > >> > > > > > change names easily (before the release).
> > > > >> > > > > >
> > > > >> > > > > > In fact, I don't think it leaves a bad impression.
> Global
> > > > >> windows
> > > > >> > are
> > > > >> > > > > > non-parallel windows. There are also parallel windows.
> > Pick
> > > > what
> > > > >> > you
> > > > >> > > > need
> > > > >> > > > > > and what works.
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <
> > > > >> gyula.fora@gmail.com>
> > > > >> > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > I think we agree on everything its more of a naming
> > issue
> > > :)
> > > > >> > > > > > >
> > > > >> > > > > > > I thought it might be misleading that global time
> > windows
> > > > are
> > > > >> > > > > > > "non-parallel" windows. We dont want to give a bad
> > > > impression.
> > > > >> > > (Also
> > > > >> > > > we
> > > > >> > > > > > > dont want them to think that every global window is
> > > parallel
> > > > >> but
> > > > >> > > > thats
> > > > >> > > > > > not
> > > > >> > > > > > > a problem here)
> > > > >> > > > > > >
> > > > >> > > > > > > Gyula
> > > > >> > > > > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <
> > > > >> sewen@apache.org>
> > > > >> > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Okay, what is missing about the windowing in your
> > > opinion?
> > > > >> > > > > > > >
> > > > >> > > > > > > > The core points of the document are:
> > > > >> > > > > > > >
> > > > >> > > > > > > >   - The parallel windows are per group only.
> > > > >> > > > > > > >
> > > > >> > > > > > > >   - The implementation of the parallel windows holds
> > > > window
> > > > >> > data
> > > > >> > > in
> > > > >> > > > > the
> > > > >> > > > > > > > group buffers.
> > > > >> > > > > > > >
> > > > >> > > > > > > >   - The global windows are non-parallel. May have
> > > parallel
> > > > >> > > > > > > pre-aggregation,
> > > > >> > > > > > > > if they are time windows.
> > > > >> > > > > > > >
> > > > >> > > > > > > >   - Time may be operator time (timer thread), or
> > > watermark
> > > > >> > time.
> > > > >> > > > > > > Watermark
> > > > >> > > > > > > > time can refer to ingress or event time.
> > > > >> > > > > > > >
> > > > >> > > > > > > >   - Windows that do not pre-aggregate may require
> > > elements
> > > > >> in
> > > > >> > > > order.
> > > > >> > > > > > Not
> > > > >> > > > > > > > part of the first prototype.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Do we agree on those points?
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <
> > > > >> > > gyula.fora@gmail.com>
> > > > >> > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > In general I like it, although the main difference
> > > > between
> > > > >> > the
> > > > >> > > > > > current
> > > > >> > > > > > > > and
> > > > >> > > > > > > > > the new one is the windowing and that is still not
> > > very
> > > > >> > clear.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Where do we have the full stream time windows for
> > > > >> > > instance?(which
> > > > >> > > > > is
> > > > >> > > > > > > > > parallel but not keyed)
> > > > >> > > > > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> > > > >> > > > > > aljoscha@apache.org>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > +1 I like it as well.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <
> > > > >> > > > ktzoumas@apache.org
> > > > >> > > > > >
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > +1 from my side
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen
> <
> > > > >> > > > > sewen@apache.org>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > Do we have consensus on these designs?
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > If we have, we should get to implementing
> this
> > > > soon,
> > > > >> > > > because
> > > > >> > > > > > > > > basically
> > > > >> > > > > > > > > > > all
> > > > >> > > > > > > > > > > > streaming patches will have to be revisited
> in
> > > > >> light of
> > > > >> > > > > this...
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> > > > >> > > > > > gyula.fora@gmail.com
> > > > >> > > > > > > >
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > You are right thats an important issue.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > And I think we should also do some
> renaming
> > > with
> > > > >> the
> > > > >> > > > > > > "iterations"
> > > > >> > > > > > > > > > > because
> > > > >> > > > > > > > > > > > > they are not really iterations like in the
> > > batch
> > > > >> case
> > > > >> > > and
> > > > >> > > > > it
> > > > >> > > > > > > > might
> > > > >> > > > > > > > > > > > confuse
> > > > >> > > > > > > > > > > > > some users.
> > > > >> > > > > > > > > > > > > Maybe we can call them loops or cycles and
> > > > rename
> > > > >> the
> > > > >> > > api
> > > > >> > > > > > calls
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > > make
> > > > >> > > > > > > > > > > > it
> > > > >> > > > > > > > > > > > > more intuitive what happens. It is really
> > > just a
> > > > >> > cyclic
> > > > >> > > > > > > dataflow.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Aljoscha Krettek <al...@apache.org>
> ezt
> > > írta
> > > > >> > > > (időpont:
> > > > >> > > > > > > 2015.
> > > > >> > > > > > > > > júl.
> > > > >> > > > > > > > > > > 7.,
> > > > >> > > > > > > > > > > > > K,
> > > > >> > > > > > > > > > > > > 15:35):
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Hi,
> > > > >> > > > > > > > > > > > > > I just noticed that we don't have
> anything
> > > > about
> > > > >> > how
> > > > >> > > > > > > iterations
> > > > >> > > > > > > > > and
> > > > >> > > > > > > > > > > > > > timestamps/watermarks should interact.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Cheers,
> > > > >> > > > > > > > > > > > > > Aljoscha
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan
> Ewen <
> > > > >> > > > > sewen@apache.org
> > > > >> > > > > > >
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Hi all!
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > As many of you know, there are a
> ongoing
> > > > >> efforts
> > > > >> > to
> > > > >> > > > > > > > consolidate
> > > > >> > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > streaming API for the next release,
> and
> > > then
> > > > >> > > graduate
> > > > >> > > > > it
> > > > >> > > > > > > > (from
> > > > >> > > > > > > > > > beta
> > > > >> > > > > > > > > > > > > > > status).
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > In the process of this consolidation,
> we
> > > > want
> > > > >> to
> > > > >> > > > > achieve
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > > > following
> > > > >> > > > > > > > > > > > > > > goals.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >  - Make the code more robust and
> > simplify
> > > it
> > > > >> in
> > > > >> > > parts
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >  - Clearly define the semantics of the
> > > > >> > constructs.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >  - Prepare it for support of more
> > advanced
> > > > >> > > concepts,
> > > > >> > > > > like
> > > > >> > > > > > > > > > > > partitionable
> > > > >> > > > > > > > > > > > > > > state, and event time.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >  - Cut support for certain corner
> cases
> > > that
> > > > >> were
> > > > >> > > > > > > prototyped,
> > > > >> > > > > > > > > but
> > > > >> > > > > > > > > > > > > turned
> > > > >> > > > > > > > > > > > > > > out to be not efficiently doable
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Based on prior discussions on the
> > mailing
> > > > >> list,
> > > > >> > > > > Aljoscha
> > > > >> > > > > > > and
> > > > >> > > > > > > > me
> > > > >> > > > > > > > > > > > drafted
> > > > >> > > > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > > > design documents below, which outline
> > how
> > > > the
> > > > >> > > > > > consolidated
> > > > >> > > > > > > > API
> > > > >> > > > > > > > > > > would
> > > > >> > > > > > > > > > > > > > like.
> > > > >> > > > > > > > > > > > > > > We focused in constructs, time, and
> > window
> > > > >> > > semantics.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Design document on how to restructure
> > the
> > > > >> > Streaming
> > > > >> > > > > API:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Design document on definitions of
> time,
> > > > order,
> > > > >> > and
> > > > >> > > > the
> > > > >> > > > > > > > > resulting
> > > > >> > > > > > > > > > > > > > semantics:
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Note: The design of the interfaces and
> > > > >> concepts
> > > > >> > for
> > > > >> > > > > > > advanced
> > > > >> > > > > > > > > > state
> > > > >> > > > > > > > > > > in
> > > > >> > > > > > > > > > > > > > > functions is not in here. That is part
> > of
> > > a
> > > > >> > > separate
> > > > >> > > > > > design
> > > > >> > > > > > > > > > > > discussion
> > > > >> > > > > > > > > > > > > > and
> > > > >> > > > > > > > > > > > > > > orthogonal to the designs drafted
> here.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Please have a look and voice questions
> > and
> > > > >> > > concerns.
> > > > >> > > > > > Since
> > > > >> > > > > > > we
> > > > >> > > > > > > > > > > should
> > > > >> > > > > > > > > > > > > not
> > > > >> > > > > > > > > > > > > > > break the streaming API more than
> once,
> > we
> > > > >> should
> > > > >> > > > make
> > > > >> > > > > > sure
> > > > >> > > > > > > > > this
> > > > >> > > > > > > > > > > > > > > consolidation brings it into the shape
> > we
> > > > >> want it
> > > > >> > > to
> > > > >> > > > be
> > > > >> > > > > > in.
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > > Greetings,
> > > > >> > > > > > > > > > > > > > > Stephan
> > > > >> > > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Stephan Ewen <se...@apache.org>]

There is no inconsistency between the Batch and Streaming API. They have
different semantics - the batch API is implicitly always windowed.

There is a naming difference between the two APIs.

There is a strong inconsistency within the Streaming API right now.
Grouping and aggregating without windows is plain dangerous in streaming.
It either blows up or is undefined in its behavior.



On Tue, Jul 14, 2015 at 12:40 PM, Gyula Fóra <gy...@gmail.com> wrote:

> I see your point, reduceByKey is much clearer.
>
> The question is whether we want to introduce this inconsistency across the
> two api-s or stick with what we have.
> On Tue, Jul 14, 2015 at 10:57 AM Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > I agree, the groupBy, in the batch API is misleading, since a
> > ds.groupBy().reduce() does not really build any groups, it is really a
> > ds.keyBy().reduceByKey(). In the streaming API we can still fix this,
> IMHO.
> >
> > On Tue, 14 Jul 2015 at 10:56 Stephan Ewen <se...@apache.org> wrote:
> >
> > > It is not a bit different than the batch API, because streaming
> semantics
> > > are a bit different ;-)
> > >
> > > One good thing is that we can make things better that were sub-optimal
> in
> > > the Batch API.
> > >
> > > On Tue, Jul 14, 2015 at 10:55 AM, Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > > > keyBy() does not do any grouping. Grouping in streams in not defined
> > > > without windows.
> > > >
> > > > On Tue, Jul 14, 2015 at 10:48 AM, Gyula Fóra <gy...@gmail.com>
> > > wrote:
> > > >
> > > >> If we only want to have either keyBy or groupBy, why not keep
> groupBy?
> > > >> That
> > > >> would be more consistent with the batch api.
> > > >> On Tue, Jul 14, 2015 at 10:35 AM Stephan Ewen <se...@apache.org>
> > wrote:
> > > >>
> > > >> > Concerning your comments:
> > > >> >
> > > >> > 1) In the new design, there is no grouping without windowing. The
> > > >> > KeyedDataStream subsumes the grouping and key-ing for partitioned
> > > state.
> > > >> >
> > > >> >     The keyBy() + window() makes a parallel grouped window
> > > >> >     keyBy() alone allows access to partitioned state.
> > > >> >
> > > >> >     My thought was that this is simpler, because it needs not
> > > groupBy()
> > > >> and
> > > >> > keyBy(), but one construct to handle both cases.
> > > >> >
> > > >> > 2) The discretization is a rough thought and is nothing for the
> > short
> > > >> term.
> > > >> > It totally needs more thoughts. I put it there to have it as a
> > sketch
> > > >> for
> > > >> > how to evolve this.
> > > >> >
> > > >> >     The idea is of course to not have a single data set, but a
> > series
> > > of
> > > >> > data set. In each discrete time slice, the data set can be treated
> > > like
> > > >> a
> > > >> > regular data set.
> > > >> >
> > > >> >     Let's kick off a separate design for the discretization. Joins
> > are
> > > >> good
> > > >> > to talk about (data sets can be joined with data set), and I am
> sure
> > > >> there
> > > >> > are more questions coming up.
> > > >> >
> > > >> >
> > > >> > Does that make sense?
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Tue, Jul 14, 2015 at 10:05 AM, Gyula Fóra <
> gyula.fora@gmail.com>
> > > >> wrote:
> > > >> >
> > > >> > > I think Marton has some good points here.
> > > >> > >
> > > >> > > 1) Is KeyedDataStream a better name if this is only a renaming?
> > > >> > >
> > > >> > > 2) the discretize semantics is unclear indeed. Are we operating
> > on a
> > > >> > single
> > > >> > > or sequence of datasets? If the latter why not call it something
> > > else
> > > >> > > (dstream). How are joins and other binary operators defined for
> > > >> different
> > > >> > > discretizations etc.
> > > >> > > On Mon, Jul 13, 2015 at 7:37 PM Márton Balassi <
> > mbalassi@apache.org
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Generally I agree with the new design. Two concerns:
> > > >> > > >
> > > >> > > > 1) Does KeyedDataStream replace GroupedDataStream or is it the
> > > >> latter a
> > > >> > > > special case of the former?
> > > >> > > >
> > > >> > > > The KeyedDataStream as described in the design document is a
> bit
> > > >> > unclear
> > > >> > > > for me. It lists the following usages:
> > > >> > > >   a) It is the first step in building a window stream, on top
> of
> > > >> which
> > > >> > > the
> > > >> > > > grouped/windowed aggregation and reduce-style function can be
> > > >> applied
> > > >> > > >   b) It allows to use the "by-key" state of functions. Here,
> > every
> > > >> > record
> > > >> > > > has access to a state that is scoped by its key. Key-scoped
> > state
> > > >> can
> > > >> > be
> > > >> > > > automatically redistributed and repartitioned.
> > > >> > > >
> > > >> > > > The code snippet describes a use case where the computation
> and
> > > the
> > > >> > > access
> > > >> > > > of the state is used the way currently the GroupedDataStream
> > > should
> > > >> > > work. I
> > > >> > > > suppose this is the example for case b). Would case a) also
> > window
> > > >> > > elements
> > > >> > > > by key? If yes, then this is practically a renaming and
> > > enhancement
> > > >> of
> > > >> > > the
> > > >> > > > GroupedDataStream functionality with keyed state. Then the
> > > >> > > > StreamExecutionEnvironment.createKeyedStream(Partitioner,
> > > >> > > > KeySelector)construction does not make much sense as the user
> > only
> > > >> > > operates
> > > >> > > > within the scope of the keyselector and not the partitioner
> > > anyway.
> > > >> > > >
> > > >> > > > I personally think KeyedDataStream as a name does not
> > necessarily
> > > >> > suggest
> > > >> > > > that the records are grouped by key, it only suggests
> > partitioning
> > > >> by
> > > >> > > key -
> > > >> > > > at least for me. :)
> > > >> > > >
> > > >> > > > 2) The API for discretization is not convenient IMHO
> > > >> > > >
> > > >> > > > The discretization part declares that the output of
> > > >> > > DataStream.discretize()
> > > >> > > > is a sequence of DataSets. I love this approach, but then in
> the
> > > >> code
> > > >> > > > snippet the return value of this function is simply a DataSet
> > and
> > > >> uses
> > > >> > it
> > > >> > > > as such. The take home message of that code is the following:
> > this
> > > >> is
> > > >> > > > actually the way you would like to program on these sequence
> of
> > > >> > DataSets,
> > > >> > > > most probably you would like to do the same with each of them.
> > If
> > > >> that
> > > >> > is
> > > >> > > > the case we should provide a nice utility for that. I think
> > Spark
> > > >> > > > Streaming's DStream.foreachRDD() is fairly useful for this
> > > purpose.
> > > >> > > >
> > > >> > > > On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <
> > gyula.fora@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > +1
> > > >> > > > > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <
> > sewen@apache.org>
> > > >> > wrote:
> > > >> > > > >
> > > >> > > > > > If naming is the only concern, then we should go ahead,
> > > because
> > > >> we
> > > >> > > can
> > > >> > > > > > change names easily (before the release).
> > > >> > > > > >
> > > >> > > > > > In fact, I don't think it leaves a bad impression. Global
> > > >> windows
> > > >> > are
> > > >> > > > > > non-parallel windows. There are also parallel windows.
> Pick
> > > what
> > > >> > you
> > > >> > > > need
> > > >> > > > > > and what works.
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <
> > > >> gyula.fora@gmail.com>
> > > >> > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > I think we agree on everything its more of a naming
> issue
> > :)
> > > >> > > > > > >
> > > >> > > > > > > I thought it might be misleading that global time
> windows
> > > are
> > > >> > > > > > > "non-parallel" windows. We dont want to give a bad
> > > impression.
> > > >> > > (Also
> > > >> > > > we
> > > >> > > > > > > dont want them to think that every global window is
> > parallel
> > > >> but
> > > >> > > > thats
> > > >> > > > > > not
> > > >> > > > > > > a problem here)
> > > >> > > > > > >
> > > >> > > > > > > Gyula
> > > >> > > > > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <
> > > >> sewen@apache.org>
> > > >> > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Okay, what is missing about the windowing in your
> > opinion?
> > > >> > > > > > > >
> > > >> > > > > > > > The core points of the document are:
> > > >> > > > > > > >
> > > >> > > > > > > >   - The parallel windows are per group only.
> > > >> > > > > > > >
> > > >> > > > > > > >   - The implementation of the parallel windows holds
> > > window
> > > >> > data
> > > >> > > in
> > > >> > > > > the
> > > >> > > > > > > > group buffers.
> > > >> > > > > > > >
> > > >> > > > > > > >   - The global windows are non-parallel. May have
> > parallel
> > > >> > > > > > > pre-aggregation,
> > > >> > > > > > > > if they are time windows.
> > > >> > > > > > > >
> > > >> > > > > > > >   - Time may be operator time (timer thread), or
> > watermark
> > > >> > time.
> > > >> > > > > > > Watermark
> > > >> > > > > > > > time can refer to ingress or event time.
> > > >> > > > > > > >
> > > >> > > > > > > >   - Windows that do not pre-aggregate may require
> > elements
> > > >> in
> > > >> > > > order.
> > > >> > > > > > Not
> > > >> > > > > > > > part of the first prototype.
> > > >> > > > > > > >
> > > >> > > > > > > > Do we agree on those points?
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <
> > > >> > > gyula.fora@gmail.com>
> > > >> > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > In general I like it, although the main difference
> > > between
> > > >> > the
> > > >> > > > > > current
> > > >> > > > > > > > and
> > > >> > > > > > > > > the new one is the windowing and that is still not
> > very
> > > >> > clear.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Where do we have the full stream time windows for
> > > >> > > instance?(which
> > > >> > > > > is
> > > >> > > > > > > > > parallel but not keyed)
> > > >> > > > > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> > > >> > > > > > aljoscha@apache.org>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > +1 I like it as well.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <
> > > >> > > > ktzoumas@apache.org
> > > >> > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > +1 from my side
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <
> > > >> > > > > sewen@apache.org>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > Do we have consensus on these designs?
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > If we have, we should get to implementing this
> > > soon,
> > > >> > > > because
> > > >> > > > > > > > > basically
> > > >> > > > > > > > > > > all
> > > >> > > > > > > > > > > > streaming patches will have to be revisited in
> > > >> light of
> > > >> > > > > this...
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> > > >> > > > > > gyula.fora@gmail.com
> > > >> > > > > > > >
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > > You are right thats an important issue.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > And I think we should also do some renaming
> > with
> > > >> the
> > > >> > > > > > > "iterations"
> > > >> > > > > > > > > > > because
> > > >> > > > > > > > > > > > > they are not really iterations like in the
> > batch
> > > >> case
> > > >> > > and
> > > >> > > > > it
> > > >> > > > > > > > might
> > > >> > > > > > > > > > > > confuse
> > > >> > > > > > > > > > > > > some users.
> > > >> > > > > > > > > > > > > Maybe we can call them loops or cycles and
> > > rename
> > > >> the
> > > >> > > api
> > > >> > > > > > calls
> > > >> > > > > > > > to
> > > >> > > > > > > > > > make
> > > >> > > > > > > > > > > > it
> > > >> > > > > > > > > > > > > more intuitive what happens. It is really
> > just a
> > > >> > cyclic
> > > >> > > > > > > dataflow.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Aljoscha Krettek <al...@apache.org> ezt
> > írta
> > > >> > > > (időpont:
> > > >> > > > > > > 2015.
> > > >> > > > > > > > > júl.
> > > >> > > > > > > > > > > 7.,
> > > >> > > > > > > > > > > > > K,
> > > >> > > > > > > > > > > > > 15:35):
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Hi,
> > > >> > > > > > > > > > > > > > I just noticed that we don't have anything
> > > about
> > > >> > how
> > > >> > > > > > > iterations
> > > >> > > > > > > > > and
> > > >> > > > > > > > > > > > > > timestamps/watermarks should interact.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Cheers,
> > > >> > > > > > > > > > > > > > Aljoscha
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <
> > > >> > > > > sewen@apache.org
> > > >> > > > > > >
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Hi all!
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > As many of you know, there are a ongoing
> > > >> efforts
> > > >> > to
> > > >> > > > > > > > consolidate
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > streaming API for the next release, and
> > then
> > > >> > > graduate
> > > >> > > > > it
> > > >> > > > > > > > (from
> > > >> > > > > > > > > > beta
> > > >> > > > > > > > > > > > > > > status).
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > In the process of this consolidation, we
> > > want
> > > >> to
> > > >> > > > > achieve
> > > >> > > > > > > the
> > > >> > > > > > > > > > > > following
> > > >> > > > > > > > > > > > > > > goals.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >  - Make the code more robust and
> simplify
> > it
> > > >> in
> > > >> > > parts
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >  - Clearly define the semantics of the
> > > >> > constructs.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >  - Prepare it for support of more
> advanced
> > > >> > > concepts,
> > > >> > > > > like
> > > >> > > > > > > > > > > > partitionable
> > > >> > > > > > > > > > > > > > > state, and event time.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >  - Cut support for certain corner cases
> > that
> > > >> were
> > > >> > > > > > > prototyped,
> > > >> > > > > > > > > but
> > > >> > > > > > > > > > > > > turned
> > > >> > > > > > > > > > > > > > > out to be not efficiently doable
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Based on prior discussions on the
> mailing
> > > >> list,
> > > >> > > > > Aljoscha
> > > >> > > > > > > and
> > > >> > > > > > > > me
> > > >> > > > > > > > > > > > drafted
> > > >> > > > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > > > design documents below, which outline
> how
> > > the
> > > >> > > > > > consolidated
> > > >> > > > > > > > API
> > > >> > > > > > > > > > > would
> > > >> > > > > > > > > > > > > > like.
> > > >> > > > > > > > > > > > > > > We focused in constructs, time, and
> window
> > > >> > > semantics.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Design document on how to restructure
> the
> > > >> > Streaming
> > > >> > > > > API:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Design document on definitions of time,
> > > order,
> > > >> > and
> > > >> > > > the
> > > >> > > > > > > > > resulting
> > > >> > > > > > > > > > > > > > semantics:
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Note: The design of the interfaces and
> > > >> concepts
> > > >> > for
> > > >> > > > > > > advanced
> > > >> > > > > > > > > > state
> > > >> > > > > > > > > > > in
> > > >> > > > > > > > > > > > > > > functions is not in here. That is part
> of
> > a
> > > >> > > separate
> > > >> > > > > > design
> > > >> > > > > > > > > > > > discussion
> > > >> > > > > > > > > > > > > > and
> > > >> > > > > > > > > > > > > > > orthogonal to the designs drafted here.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Please have a look and voice questions
> and
> > > >> > > concerns.
> > > >> > > > > > Since
> > > >> > > > > > > we
> > > >> > > > > > > > > > > should
> > > >> > > > > > > > > > > > > not
> > > >> > > > > > > > > > > > > > > break the streaming API more than once,
> we
> > > >> should
> > > >> > > > make
> > > >> > > > > > sure
> > > >> > > > > > > > > this
> > > >> > > > > > > > > > > > > > > consolidation brings it into the shape
> we
> > > >> want it
> > > >> > > to
> > > >> > > > be
> > > >> > > > > > in.
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > > Greetings,
> > > >> > > > > > > > > > > > > > > Stephan
> > > >> > > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Gyula Fóra <gy...@gmail.com>]

I see your point, reduceByKey is much clearer.

The question is whether we want to introduce this inconsistency across the
two api-s or stick with what we have.
On Tue, Jul 14, 2015 at 10:57 AM Aljoscha Krettek <al...@apache.org>
wrote:

> I agree, the groupBy, in the batch API is misleading, since a
> ds.groupBy().reduce() does not really build any groups, it is really a
> ds.keyBy().reduceByKey(). In the streaming API we can still fix this, IMHO.
>
> On Tue, 14 Jul 2015 at 10:56 Stephan Ewen <se...@apache.org> wrote:
>
> > It is not a bit different than the batch API, because streaming semantics
> > are a bit different ;-)
> >
> > One good thing is that we can make things better that were sub-optimal in
> > the Batch API.
> >
> > On Tue, Jul 14, 2015 at 10:55 AM, Stephan Ewen <se...@apache.org> wrote:
> >
> > > keyBy() does not do any grouping. Grouping in streams in not defined
> > > without windows.
> > >
> > > On Tue, Jul 14, 2015 at 10:48 AM, Gyula Fóra <gy...@gmail.com>
> > wrote:
> > >
> > >> If we only want to have either keyBy or groupBy, why not keep groupBy?
> > >> That
> > >> would be more consistent with the batch api.
> > >> On Tue, Jul 14, 2015 at 10:35 AM Stephan Ewen <se...@apache.org>
> wrote:
> > >>
> > >> > Concerning your comments:
> > >> >
> > >> > 1) In the new design, there is no grouping without windowing. The
> > >> > KeyedDataStream subsumes the grouping and key-ing for partitioned
> > state.
> > >> >
> > >> >     The keyBy() + window() makes a parallel grouped window
> > >> >     keyBy() alone allows access to partitioned state.
> > >> >
> > >> >     My thought was that this is simpler, because it needs not
> > groupBy()
> > >> and
> > >> > keyBy(), but one construct to handle both cases.
> > >> >
> > >> > 2) The discretization is a rough thought and is nothing for the
> short
> > >> term.
> > >> > It totally needs more thoughts. I put it there to have it as a
> sketch
> > >> for
> > >> > how to evolve this.
> > >> >
> > >> >     The idea is of course to not have a single data set, but a
> series
> > of
> > >> > data set. In each discrete time slice, the data set can be treated
> > like
> > >> a
> > >> > regular data set.
> > >> >
> > >> >     Let's kick off a separate design for the discretization. Joins
> are
> > >> good
> > >> > to talk about (data sets can be joined with data set), and I am sure
> > >> there
> > >> > are more questions coming up.
> > >> >
> > >> >
> > >> > Does that make sense?
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Jul 14, 2015 at 10:05 AM, Gyula Fóra <gy...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > I think Marton has some good points here.
> > >> > >
> > >> > > 1) Is KeyedDataStream a better name if this is only a renaming?
> > >> > >
> > >> > > 2) the discretize semantics is unclear indeed. Are we operating
> on a
> > >> > single
> > >> > > or sequence of datasets? If the latter why not call it something
> > else
> > >> > > (dstream). How are joins and other binary operators defined for
> > >> different
> > >> > > discretizations etc.
> > >> > > On Mon, Jul 13, 2015 at 7:37 PM Márton Balassi <
> mbalassi@apache.org
> > >
> > >> > > wrote:
> > >> > >
> > >> > > > Generally I agree with the new design. Two concerns:
> > >> > > >
> > >> > > > 1) Does KeyedDataStream replace GroupedDataStream or is it the
> > >> latter a
> > >> > > > special case of the former?
> > >> > > >
> > >> > > > The KeyedDataStream as described in the design document is a bit
> > >> > unclear
> > >> > > > for me. It lists the following usages:
> > >> > > >   a) It is the first step in building a window stream, on top of
> > >> which
> > >> > > the
> > >> > > > grouped/windowed aggregation and reduce-style function can be
> > >> applied
> > >> > > >   b) It allows to use the "by-key" state of functions. Here,
> every
> > >> > record
> > >> > > > has access to a state that is scoped by its key. Key-scoped
> state
> > >> can
> > >> > be
> > >> > > > automatically redistributed and repartitioned.
> > >> > > >
> > >> > > > The code snippet describes a use case where the computation and
> > the
> > >> > > access
> > >> > > > of the state is used the way currently the GroupedDataStream
> > should
> > >> > > work. I
> > >> > > > suppose this is the example for case b). Would case a) also
> window
> > >> > > elements
> > >> > > > by key? If yes, then this is practically a renaming and
> > enhancement
> > >> of
> > >> > > the
> > >> > > > GroupedDataStream functionality with keyed state. Then the
> > >> > > > StreamExecutionEnvironment.createKeyedStream(Partitioner,
> > >> > > > KeySelector)construction does not make much sense as the user
> only
> > >> > > operates
> > >> > > > within the scope of the keyselector and not the partitioner
> > anyway.
> > >> > > >
> > >> > > > I personally think KeyedDataStream as a name does not
> necessarily
> > >> > suggest
> > >> > > > that the records are grouped by key, it only suggests
> partitioning
> > >> by
> > >> > > key -
> > >> > > > at least for me. :)
> > >> > > >
> > >> > > > 2) The API for discretization is not convenient IMHO
> > >> > > >
> > >> > > > The discretization part declares that the output of
> > >> > > DataStream.discretize()
> > >> > > > is a sequence of DataSets. I love this approach, but then in the
> > >> code
> > >> > > > snippet the return value of this function is simply a DataSet
> and
> > >> uses
> > >> > it
> > >> > > > as such. The take home message of that code is the following:
> this
> > >> is
> > >> > > > actually the way you would like to program on these sequence of
> > >> > DataSets,
> > >> > > > most probably you would like to do the same with each of them.
> If
> > >> that
> > >> > is
> > >> > > > the case we should provide a nice utility for that. I think
> Spark
> > >> > > > Streaming's DStream.foreachRDD() is fairly useful for this
> > purpose.
> > >> > > >
> > >> > > > On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <
> gyula.fora@gmail.com
> > >
> > >> > > wrote:
> > >> > > >
> > >> > > > > +1
> > >> > > > > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <
> sewen@apache.org>
> > >> > wrote:
> > >> > > > >
> > >> > > > > > If naming is the only concern, then we should go ahead,
> > because
> > >> we
> > >> > > can
> > >> > > > > > change names easily (before the release).
> > >> > > > > >
> > >> > > > > > In fact, I don't think it leaves a bad impression. Global
> > >> windows
> > >> > are
> > >> > > > > > non-parallel windows. There are also parallel windows. Pick
> > what
> > >> > you
> > >> > > > need
> > >> > > > > > and what works.
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <
> > >> gyula.fora@gmail.com>
> > >> > > > > wrote:
> > >> > > > > >
> > >> > > > > > > I think we agree on everything its more of a naming issue
> :)
> > >> > > > > > >
> > >> > > > > > > I thought it might be misleading that global time windows
> > are
> > >> > > > > > > "non-parallel" windows. We dont want to give a bad
> > impression.
> > >> > > (Also
> > >> > > > we
> > >> > > > > > > dont want them to think that every global window is
> parallel
> > >> but
> > >> > > > thats
> > >> > > > > > not
> > >> > > > > > > a problem here)
> > >> > > > > > >
> > >> > > > > > > Gyula
> > >> > > > > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <
> > >> sewen@apache.org>
> > >> > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Okay, what is missing about the windowing in your
> opinion?
> > >> > > > > > > >
> > >> > > > > > > > The core points of the document are:
> > >> > > > > > > >
> > >> > > > > > > >   - The parallel windows are per group only.
> > >> > > > > > > >
> > >> > > > > > > >   - The implementation of the parallel windows holds
> > window
> > >> > data
> > >> > > in
> > >> > > > > the
> > >> > > > > > > > group buffers.
> > >> > > > > > > >
> > >> > > > > > > >   - The global windows are non-parallel. May have
> parallel
> > >> > > > > > > pre-aggregation,
> > >> > > > > > > > if they are time windows.
> > >> > > > > > > >
> > >> > > > > > > >   - Time may be operator time (timer thread), or
> watermark
> > >> > time.
> > >> > > > > > > Watermark
> > >> > > > > > > > time can refer to ingress or event time.
> > >> > > > > > > >
> > >> > > > > > > >   - Windows that do not pre-aggregate may require
> elements
> > >> in
> > >> > > > order.
> > >> > > > > > Not
> > >> > > > > > > > part of the first prototype.
> > >> > > > > > > >
> > >> > > > > > > > Do we agree on those points?
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <
> > >> > > gyula.fora@gmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > In general I like it, although the main difference
> > between
> > >> > the
> > >> > > > > > current
> > >> > > > > > > > and
> > >> > > > > > > > > the new one is the windowing and that is still not
> very
> > >> > clear.
> > >> > > > > > > > >
> > >> > > > > > > > > Where do we have the full stream time windows for
> > >> > > instance?(which
> > >> > > > > is
> > >> > > > > > > > > parallel but not keyed)
> > >> > > > > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> > >> > > > > > aljoscha@apache.org>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > +1 I like it as well.
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <
> > >> > > > ktzoumas@apache.org
> > >> > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > +1 from my side
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <
> > >> > > > > sewen@apache.org>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > Do we have consensus on these designs?
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > If we have, we should get to implementing this
> > soon,
> > >> > > > because
> > >> > > > > > > > > basically
> > >> > > > > > > > > > > all
> > >> > > > > > > > > > > > streaming patches will have to be revisited in
> > >> light of
> > >> > > > > this...
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> > >> > > > > > gyula.fora@gmail.com
> > >> > > > > > > >
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > > You are right thats an important issue.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > And I think we should also do some renaming
> with
> > >> the
> > >> > > > > > > "iterations"
> > >> > > > > > > > > > > because
> > >> > > > > > > > > > > > > they are not really iterations like in the
> batch
> > >> case
> > >> > > and
> > >> > > > > it
> > >> > > > > > > > might
> > >> > > > > > > > > > > > confuse
> > >> > > > > > > > > > > > > some users.
> > >> > > > > > > > > > > > > Maybe we can call them loops or cycles and
> > rename
> > >> the
> > >> > > api
> > >> > > > > > calls
> > >> > > > > > > > to
> > >> > > > > > > > > > make
> > >> > > > > > > > > > > > it
> > >> > > > > > > > > > > > > more intuitive what happens. It is really
> just a
> > >> > cyclic
> > >> > > > > > > dataflow.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Aljoscha Krettek <al...@apache.org> ezt
> írta
> > >> > > > (időpont:
> > >> > > > > > > 2015.
> > >> > > > > > > > > júl.
> > >> > > > > > > > > > > 7.,
> > >> > > > > > > > > > > > > K,
> > >> > > > > > > > > > > > > 15:35):
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Hi,
> > >> > > > > > > > > > > > > > I just noticed that we don't have anything
> > about
> > >> > how
> > >> > > > > > > iterations
> > >> > > > > > > > > and
> > >> > > > > > > > > > > > > > timestamps/watermarks should interact.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Cheers,
> > >> > > > > > > > > > > > > > Aljoscha
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <
> > >> > > > > sewen@apache.org
> > >> > > > > > >
> > >> > > > > > > > > wrote:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Hi all!
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > As many of you know, there are a ongoing
> > >> efforts
> > >> > to
> > >> > > > > > > > consolidate
> > >> > > > > > > > > > the
> > >> > > > > > > > > > > > > > > streaming API for the next release, and
> then
> > >> > > graduate
> > >> > > > > it
> > >> > > > > > > > (from
> > >> > > > > > > > > > beta
> > >> > > > > > > > > > > > > > > status).
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > In the process of this consolidation, we
> > want
> > >> to
> > >> > > > > achieve
> > >> > > > > > > the
> > >> > > > > > > > > > > > following
> > >> > > > > > > > > > > > > > > goals.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >  - Make the code more robust and simplify
> it
> > >> in
> > >> > > parts
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >  - Clearly define the semantics of the
> > >> > constructs.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >  - Prepare it for support of more advanced
> > >> > > concepts,
> > >> > > > > like
> > >> > > > > > > > > > > > partitionable
> > >> > > > > > > > > > > > > > > state, and event time.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >  - Cut support for certain corner cases
> that
> > >> were
> > >> > > > > > > prototyped,
> > >> > > > > > > > > but
> > >> > > > > > > > > > > > > turned
> > >> > > > > > > > > > > > > > > out to be not efficiently doable
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Based on prior discussions on the mailing
> > >> list,
> > >> > > > > Aljoscha
> > >> > > > > > > and
> > >> > > > > > > > me
> > >> > > > > > > > > > > > drafted
> > >> > > > > > > > > > > > > > the
> > >> > > > > > > > > > > > > > > design documents below, which outline how
> > the
> > >> > > > > > consolidated
> > >> > > > > > > > API
> > >> > > > > > > > > > > would
> > >> > > > > > > > > > > > > > like.
> > >> > > > > > > > > > > > > > > We focused in constructs, time, and window
> > >> > > semantics.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Design document on how to restructure the
> > >> > Streaming
> > >> > > > > API:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Design document on definitions of time,
> > order,
> > >> > and
> > >> > > > the
> > >> > > > > > > > > resulting
> > >> > > > > > > > > > > > > > semantics:
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Note: The design of the interfaces and
> > >> concepts
> > >> > for
> > >> > > > > > > advanced
> > >> > > > > > > > > > state
> > >> > > > > > > > > > > in
> > >> > > > > > > > > > > > > > > functions is not in here. That is part of
> a
> > >> > > separate
> > >> > > > > > design
> > >> > > > > > > > > > > > discussion
> > >> > > > > > > > > > > > > > and
> > >> > > > > > > > > > > > > > > orthogonal to the designs drafted here.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Please have a look and voice questions and
> > >> > > concerns.
> > >> > > > > > Since
> > >> > > > > > > we
> > >> > > > > > > > > > > should
> > >> > > > > > > > > > > > > not
> > >> > > > > > > > > > > > > > > break the streaming API more than once, we
> > >> should
> > >> > > > make
> > >> > > > > > sure
> > >> > > > > > > > > this
> > >> > > > > > > > > > > > > > > consolidation brings it into the shape we
> > >> want it
> > >> > > to
> > >> > > > be
> > >> > > > > > in.
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > > Greetings,
> > >> > > > > > > > > > > > > > > Stephan
> > >> > > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Aljoscha Krettek <al...@apache.org>]

I agree, the groupBy, in the batch API is misleading, since a
ds.groupBy().reduce() does not really build any groups, it is really a
ds.keyBy().reduceByKey(). In the streaming API we can still fix this, IMHO.

On Tue, 14 Jul 2015 at 10:56 Stephan Ewen <se...@apache.org> wrote:

> It is not a bit different than the batch API, because streaming semantics
> are a bit different ;-)
>
> One good thing is that we can make things better that were sub-optimal in
> the Batch API.
>
> On Tue, Jul 14, 2015 at 10:55 AM, Stephan Ewen <se...@apache.org> wrote:
>
> > keyBy() does not do any grouping. Grouping in streams in not defined
> > without windows.
> >
> > On Tue, Jul 14, 2015 at 10:48 AM, Gyula Fóra <gy...@gmail.com>
> wrote:
> >
> >> If we only want to have either keyBy or groupBy, why not keep groupBy?
> >> That
> >> would be more consistent with the batch api.
> >> On Tue, Jul 14, 2015 at 10:35 AM Stephan Ewen <se...@apache.org> wrote:
> >>
> >> > Concerning your comments:
> >> >
> >> > 1) In the new design, there is no grouping without windowing. The
> >> > KeyedDataStream subsumes the grouping and key-ing for partitioned
> state.
> >> >
> >> >     The keyBy() + window() makes a parallel grouped window
> >> >     keyBy() alone allows access to partitioned state.
> >> >
> >> >     My thought was that this is simpler, because it needs not
> groupBy()
> >> and
> >> > keyBy(), but one construct to handle both cases.
> >> >
> >> > 2) The discretization is a rough thought and is nothing for the short
> >> term.
> >> > It totally needs more thoughts. I put it there to have it as a sketch
> >> for
> >> > how to evolve this.
> >> >
> >> >     The idea is of course to not have a single data set, but a series
> of
> >> > data set. In each discrete time slice, the data set can be treated
> like
> >> a
> >> > regular data set.
> >> >
> >> >     Let's kick off a separate design for the discretization. Joins are
> >> good
> >> > to talk about (data sets can be joined with data set), and I am sure
> >> there
> >> > are more questions coming up.
> >> >
> >> >
> >> > Does that make sense?
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Tue, Jul 14, 2015 at 10:05 AM, Gyula Fóra <gy...@gmail.com>
> >> wrote:
> >> >
> >> > > I think Marton has some good points here.
> >> > >
> >> > > 1) Is KeyedDataStream a better name if this is only a renaming?
> >> > >
> >> > > 2) the discretize semantics is unclear indeed. Are we operating on a
> >> > single
> >> > > or sequence of datasets? If the latter why not call it something
> else
> >> > > (dstream). How are joins and other binary operators defined for
> >> different
> >> > > discretizations etc.
> >> > > On Mon, Jul 13, 2015 at 7:37 PM Márton Balassi <mbalassi@apache.org
> >
> >> > > wrote:
> >> > >
> >> > > > Generally I agree with the new design. Two concerns:
> >> > > >
> >> > > > 1) Does KeyedDataStream replace GroupedDataStream or is it the
> >> latter a
> >> > > > special case of the former?
> >> > > >
> >> > > > The KeyedDataStream as described in the design document is a bit
> >> > unclear
> >> > > > for me. It lists the following usages:
> >> > > >   a) It is the first step in building a window stream, on top of
> >> which
> >> > > the
> >> > > > grouped/windowed aggregation and reduce-style function can be
> >> applied
> >> > > >   b) It allows to use the "by-key" state of functions. Here, every
> >> > record
> >> > > > has access to a state that is scoped by its key. Key-scoped state
> >> can
> >> > be
> >> > > > automatically redistributed and repartitioned.
> >> > > >
> >> > > > The code snippet describes a use case where the computation and
> the
> >> > > access
> >> > > > of the state is used the way currently the GroupedDataStream
> should
> >> > > work. I
> >> > > > suppose this is the example for case b). Would case a) also window
> >> > > elements
> >> > > > by key? If yes, then this is practically a renaming and
> enhancement
> >> of
> >> > > the
> >> > > > GroupedDataStream functionality with keyed state. Then the
> >> > > > StreamExecutionEnvironment.createKeyedStream(Partitioner,
> >> > > > KeySelector)construction does not make much sense as the user only
> >> > > operates
> >> > > > within the scope of the keyselector and not the partitioner
> anyway.
> >> > > >
> >> > > > I personally think KeyedDataStream as a name does not necessarily
> >> > suggest
> >> > > > that the records are grouped by key, it only suggests partitioning
> >> by
> >> > > key -
> >> > > > at least for me. :)
> >> > > >
> >> > > > 2) The API for discretization is not convenient IMHO
> >> > > >
> >> > > > The discretization part declares that the output of
> >> > > DataStream.discretize()
> >> > > > is a sequence of DataSets. I love this approach, but then in the
> >> code
> >> > > > snippet the return value of this function is simply a DataSet and
> >> uses
> >> > it
> >> > > > as such. The take home message of that code is the following: this
> >> is
> >> > > > actually the way you would like to program on these sequence of
> >> > DataSets,
> >> > > > most probably you would like to do the same with each of them. If
> >> that
> >> > is
> >> > > > the case we should provide a nice utility for that. I think Spark
> >> > > > Streaming's DStream.foreachRDD() is fairly useful for this
> purpose.
> >> > > >
> >> > > > On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <gyula.fora@gmail.com
> >
> >> > > wrote:
> >> > > >
> >> > > > > +1
> >> > > > > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <se...@apache.org>
> >> > wrote:
> >> > > > >
> >> > > > > > If naming is the only concern, then we should go ahead,
> because
> >> we
> >> > > can
> >> > > > > > change names easily (before the release).
> >> > > > > >
> >> > > > > > In fact, I don't think it leaves a bad impression. Global
> >> windows
> >> > are
> >> > > > > > non-parallel windows. There are also parallel windows. Pick
> what
> >> > you
> >> > > > need
> >> > > > > > and what works.
> >> > > > > >
> >> > > > > >
> >> > > > > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <
> >> gyula.fora@gmail.com>
> >> > > > > wrote:
> >> > > > > >
> >> > > > > > > I think we agree on everything its more of a naming issue :)
> >> > > > > > >
> >> > > > > > > I thought it might be misleading that global time windows
> are
> >> > > > > > > "non-parallel" windows. We dont want to give a bad
> impression.
> >> > > (Also
> >> > > > we
> >> > > > > > > dont want them to think that every global window is parallel
> >> but
> >> > > > thats
> >> > > > > > not
> >> > > > > > > a problem here)
> >> > > > > > >
> >> > > > > > > Gyula
> >> > > > > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <
> >> sewen@apache.org>
> >> > > > wrote:
> >> > > > > > >
> >> > > > > > > > Okay, what is missing about the windowing in your opinion?
> >> > > > > > > >
> >> > > > > > > > The core points of the document are:
> >> > > > > > > >
> >> > > > > > > >   - The parallel windows are per group only.
> >> > > > > > > >
> >> > > > > > > >   - The implementation of the parallel windows holds
> window
> >> > data
> >> > > in
> >> > > > > the
> >> > > > > > > > group buffers.
> >> > > > > > > >
> >> > > > > > > >   - The global windows are non-parallel. May have parallel
> >> > > > > > > pre-aggregation,
> >> > > > > > > > if they are time windows.
> >> > > > > > > >
> >> > > > > > > >   - Time may be operator time (timer thread), or watermark
> >> > time.
> >> > > > > > > Watermark
> >> > > > > > > > time can refer to ingress or event time.
> >> > > > > > > >
> >> > > > > > > >   - Windows that do not pre-aggregate may require elements
> >> in
> >> > > > order.
> >> > > > > > Not
> >> > > > > > > > part of the first prototype.
> >> > > > > > > >
> >> > > > > > > > Do we agree on those points?
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <
> >> > > gyula.fora@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > In general I like it, although the main difference
> between
> >> > the
> >> > > > > > current
> >> > > > > > > > and
> >> > > > > > > > > the new one is the windowing and that is still not very
> >> > clear.
> >> > > > > > > > >
> >> > > > > > > > > Where do we have the full stream time windows for
> >> > > instance?(which
> >> > > > > is
> >> > > > > > > > > parallel but not keyed)
> >> > > > > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> >> > > > > > aljoscha@apache.org>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > +1 I like it as well.
> >> > > > > > > > > >
> >> > > > > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <
> >> > > > ktzoumas@apache.org
> >> > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > +1 from my side
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <
> >> > > > > sewen@apache.org>
> >> > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > Do we have consensus on these designs?
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > If we have, we should get to implementing this
> soon,
> >> > > > because
> >> > > > > > > > > basically
> >> > > > > > > > > > > all
> >> > > > > > > > > > > > streaming patches will have to be revisited in
> >> light of
> >> > > > > this...
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> >> > > > > > gyula.fora@gmail.com
> >> > > > > > > >
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > > You are right thats an important issue.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > And I think we should also do some renaming with
> >> the
> >> > > > > > > "iterations"
> >> > > > > > > > > > > because
> >> > > > > > > > > > > > > they are not really iterations like in the batch
> >> case
> >> > > and
> >> > > > > it
> >> > > > > > > > might
> >> > > > > > > > > > > > confuse
> >> > > > > > > > > > > > > some users.
> >> > > > > > > > > > > > > Maybe we can call them loops or cycles and
> rename
> >> the
> >> > > api
> >> > > > > > calls
> >> > > > > > > > to
> >> > > > > > > > > > make
> >> > > > > > > > > > > > it
> >> > > > > > > > > > > > > more intuitive what happens. It is really just a
> >> > cyclic
> >> > > > > > > dataflow.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Aljoscha Krettek <al...@apache.org> ezt írta
> >> > > > (időpont:
> >> > > > > > > 2015.
> >> > > > > > > > > júl.
> >> > > > > > > > > > > 7.,
> >> > > > > > > > > > > > > K,
> >> > > > > > > > > > > > > 15:35):
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Hi,
> >> > > > > > > > > > > > > > I just noticed that we don't have anything
> about
> >> > how
> >> > > > > > > iterations
> >> > > > > > > > > and
> >> > > > > > > > > > > > > > timestamps/watermarks should interact.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Cheers,
> >> > > > > > > > > > > > > > Aljoscha
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <
> >> > > > > sewen@apache.org
> >> > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Hi all!
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > As many of you know, there are a ongoing
> >> efforts
> >> > to
> >> > > > > > > > consolidate
> >> > > > > > > > > > the
> >> > > > > > > > > > > > > > > streaming API for the next release, and then
> >> > > graduate
> >> > > > > it
> >> > > > > > > > (from
> >> > > > > > > > > > beta
> >> > > > > > > > > > > > > > > status).
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > In the process of this consolidation, we
> want
> >> to
> >> > > > > achieve
> >> > > > > > > the
> >> > > > > > > > > > > > following
> >> > > > > > > > > > > > > > > goals.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >  - Make the code more robust and simplify it
> >> in
> >> > > parts
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >  - Clearly define the semantics of the
> >> > constructs.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >  - Prepare it for support of more advanced
> >> > > concepts,
> >> > > > > like
> >> > > > > > > > > > > > partitionable
> >> > > > > > > > > > > > > > > state, and event time.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >  - Cut support for certain corner cases that
> >> were
> >> > > > > > > prototyped,
> >> > > > > > > > > but
> >> > > > > > > > > > > > > turned
> >> > > > > > > > > > > > > > > out to be not efficiently doable
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Based on prior discussions on the mailing
> >> list,
> >> > > > > Aljoscha
> >> > > > > > > and
> >> > > > > > > > me
> >> > > > > > > > > > > > drafted
> >> > > > > > > > > > > > > > the
> >> > > > > > > > > > > > > > > design documents below, which outline how
> the
> >> > > > > > consolidated
> >> > > > > > > > API
> >> > > > > > > > > > > would
> >> > > > > > > > > > > > > > like.
> >> > > > > > > > > > > > > > > We focused in constructs, time, and window
> >> > > semantics.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Design document on how to restructure the
> >> > Streaming
> >> > > > > API:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Design document on definitions of time,
> order,
> >> > and
> >> > > > the
> >> > > > > > > > > resulting
> >> > > > > > > > > > > > > > semantics:
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Note: The design of the interfaces and
> >> concepts
> >> > for
> >> > > > > > > advanced
> >> > > > > > > > > > state
> >> > > > > > > > > > > in
> >> > > > > > > > > > > > > > > functions is not in here. That is part of a
> >> > > separate
> >> > > > > > design
> >> > > > > > > > > > > > discussion
> >> > > > > > > > > > > > > > and
> >> > > > > > > > > > > > > > > orthogonal to the designs drafted here.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Please have a look and voice questions and
> >> > > concerns.
> >> > > > > > Since
> >> > > > > > > we
> >> > > > > > > > > > > should
> >> > > > > > > > > > > > > not
> >> > > > > > > > > > > > > > > break the streaming API more than once, we
> >> should
> >> > > > make
> >> > > > > > sure
> >> > > > > > > > > this
> >> > > > > > > > > > > > > > > consolidation brings it into the shape we
> >> want it
> >> > > to
> >> > > > be
> >> > > > > > in.
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > > Greetings,
> >> > > > > > > > > > > > > > > Stephan
> >> > > > > > > > > > > > > > >
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Design documents for consolidated DataStream API

[Stephan Ewen <se...@apache.org>]

It is not a bit different than the batch API, because streaming semantics
are a bit different ;-)

One good thing is that we can make things better that were sub-optimal in
the Batch API.

On Tue, Jul 14, 2015 at 10:55 AM, Stephan Ewen <se...@apache.org> wrote:

> keyBy() does not do any grouping. Grouping in streams in not defined
> without windows.
>
> On Tue, Jul 14, 2015 at 10:48 AM, Gyula Fóra <gy...@gmail.com> wrote:
>
>> If we only want to have either keyBy or groupBy, why not keep groupBy?
>> That
>> would be more consistent with the batch api.
>> On Tue, Jul 14, 2015 at 10:35 AM Stephan Ewen <se...@apache.org> wrote:
>>
>> > Concerning your comments:
>> >
>> > 1) In the new design, there is no grouping without windowing. The
>> > KeyedDataStream subsumes the grouping and key-ing for partitioned state.
>> >
>> >     The keyBy() + window() makes a parallel grouped window
>> >     keyBy() alone allows access to partitioned state.
>> >
>> >     My thought was that this is simpler, because it needs not groupBy()
>> and
>> > keyBy(), but one construct to handle both cases.
>> >
>> > 2) The discretization is a rough thought and is nothing for the short
>> term.
>> > It totally needs more thoughts. I put it there to have it as a sketch
>> for
>> > how to evolve this.
>> >
>> >     The idea is of course to not have a single data set, but a series of
>> > data set. In each discrete time slice, the data set can be treated like
>> a
>> > regular data set.
>> >
>> >     Let's kick off a separate design for the discretization. Joins are
>> good
>> > to talk about (data sets can be joined with data set), and I am sure
>> there
>> > are more questions coming up.
>> >
>> >
>> > Does that make sense?
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Jul 14, 2015 at 10:05 AM, Gyula Fóra <gy...@gmail.com>
>> wrote:
>> >
>> > > I think Marton has some good points here.
>> > >
>> > > 1) Is KeyedDataStream a better name if this is only a renaming?
>> > >
>> > > 2) the discretize semantics is unclear indeed. Are we operating on a
>> > single
>> > > or sequence of datasets? If the latter why not call it something else
>> > > (dstream). How are joins and other binary operators defined for
>> different
>> > > discretizations etc.
>> > > On Mon, Jul 13, 2015 at 7:37 PM Márton Balassi <mb...@apache.org>
>> > > wrote:
>> > >
>> > > > Generally I agree with the new design. Two concerns:
>> > > >
>> > > > 1) Does KeyedDataStream replace GroupedDataStream or is it the
>> latter a
>> > > > special case of the former?
>> > > >
>> > > > The KeyedDataStream as described in the design document is a bit
>> > unclear
>> > > > for me. It lists the following usages:
>> > > >   a) It is the first step in building a window stream, on top of
>> which
>> > > the
>> > > > grouped/windowed aggregation and reduce-style function can be
>> applied
>> > > >   b) It allows to use the "by-key" state of functions. Here, every
>> > record
>> > > > has access to a state that is scoped by its key. Key-scoped state
>> can
>> > be
>> > > > automatically redistributed and repartitioned.
>> > > >
>> > > > The code snippet describes a use case where the computation and the
>> > > access
>> > > > of the state is used the way currently the GroupedDataStream should
>> > > work. I
>> > > > suppose this is the example for case b). Would case a) also window
>> > > elements
>> > > > by key? If yes, then this is practically a renaming and enhancement
>> of
>> > > the
>> > > > GroupedDataStream functionality with keyed state. Then the
>> > > > StreamExecutionEnvironment.createKeyedStream(Partitioner,
>> > > > KeySelector)construction does not make much sense as the user only
>> > > operates
>> > > > within the scope of the keyselector and not the partitioner anyway.
>> > > >
>> > > > I personally think KeyedDataStream as a name does not necessarily
>> > suggest
>> > > > that the records are grouped by key, it only suggests partitioning
>> by
>> > > key -
>> > > > at least for me. :)
>> > > >
>> > > > 2) The API for discretization is not convenient IMHO
>> > > >
>> > > > The discretization part declares that the output of
>> > > DataStream.discretize()
>> > > > is a sequence of DataSets. I love this approach, but then in the
>> code
>> > > > snippet the return value of this function is simply a DataSet and
>> uses
>> > it
>> > > > as such. The take home message of that code is the following: this
>> is
>> > > > actually the way you would like to program on these sequence of
>> > DataSets,
>> > > > most probably you would like to do the same with each of them. If
>> that
>> > is
>> > > > the case we should provide a nice utility for that. I think Spark
>> > > > Streaming's DStream.foreachRDD() is fairly useful for this purpose.
>> > > >
>> > > > On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <gy...@gmail.com>
>> > > wrote:
>> > > >
>> > > > > +1
>> > > > > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <se...@apache.org>
>> > wrote:
>> > > > >
>> > > > > > If naming is the only concern, then we should go ahead, because
>> we
>> > > can
>> > > > > > change names easily (before the release).
>> > > > > >
>> > > > > > In fact, I don't think it leaves a bad impression. Global
>> windows
>> > are
>> > > > > > non-parallel windows. There are also parallel windows. Pick what
>> > you
>> > > > need
>> > > > > > and what works.
>> > > > > >
>> > > > > >
>> > > > > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <
>> gyula.fora@gmail.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > > I think we agree on everything its more of a naming issue :)
>> > > > > > >
>> > > > > > > I thought it might be misleading that global time windows are
>> > > > > > > "non-parallel" windows. We dont want to give a bad impression.
>> > > (Also
>> > > > we
>> > > > > > > dont want them to think that every global window is parallel
>> but
>> > > > thats
>> > > > > > not
>> > > > > > > a problem here)
>> > > > > > >
>> > > > > > > Gyula
>> > > > > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <
>> sewen@apache.org>
>> > > > wrote:
>> > > > > > >
>> > > > > > > > Okay, what is missing about the windowing in your opinion?
>> > > > > > > >
>> > > > > > > > The core points of the document are:
>> > > > > > > >
>> > > > > > > >   - The parallel windows are per group only.
>> > > > > > > >
>> > > > > > > >   - The implementation of the parallel windows holds window
>> > data
>> > > in
>> > > > > the
>> > > > > > > > group buffers.
>> > > > > > > >
>> > > > > > > >   - The global windows are non-parallel. May have parallel
>> > > > > > > pre-aggregation,
>> > > > > > > > if they are time windows.
>> > > > > > > >
>> > > > > > > >   - Time may be operator time (timer thread), or watermark
>> > time.
>> > > > > > > Watermark
>> > > > > > > > time can refer to ingress or event time.
>> > > > > > > >
>> > > > > > > >   - Windows that do not pre-aggregate may require elements
>> in
>> > > > order.
>> > > > > > Not
>> > > > > > > > part of the first prototype.
>> > > > > > > >
>> > > > > > > > Do we agree on those points?
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <
>> > > gyula.fora@gmail.com>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > In general I like it, although the main difference between
>> > the
>> > > > > > current
>> > > > > > > > and
>> > > > > > > > > the new one is the windowing and that is still not very
>> > clear.
>> > > > > > > > >
>> > > > > > > > > Where do we have the full stream time windows for
>> > > instance?(which
>> > > > > is
>> > > > > > > > > parallel but not keyed)
>> > > > > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
>> > > > > > aljoscha@apache.org>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > +1 I like it as well.
>> > > > > > > > > >
>> > > > > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <
>> > > > ktzoumas@apache.org
>> > > > > >
>> > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > +1 from my side
>> > > > > > > > > > >
>> > > > > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <
>> > > > > sewen@apache.org>
>> > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Do we have consensus on these designs?
>> > > > > > > > > > > >
>> > > > > > > > > > > > If we have, we should get to implementing this soon,
>> > > > because
>> > > > > > > > > basically
>> > > > > > > > > > > all
>> > > > > > > > > > > > streaming patches will have to be revisited in
>> light of
>> > > > > this...
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
>> > > > > > gyula.fora@gmail.com
>> > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > You are right thats an important issue.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > And I think we should also do some renaming with
>> the
>> > > > > > > "iterations"
>> > > > > > > > > > > because
>> > > > > > > > > > > > > they are not really iterations like in the batch
>> case
>> > > and
>> > > > > it
>> > > > > > > > might
>> > > > > > > > > > > > confuse
>> > > > > > > > > > > > > some users.
>> > > > > > > > > > > > > Maybe we can call them loops or cycles and rename
>> the
>> > > api
>> > > > > > calls
>> > > > > > > > to
>> > > > > > > > > > make
>> > > > > > > > > > > > it
>> > > > > > > > > > > > > more intuitive what happens. It is really just a
>> > cyclic
>> > > > > > > dataflow.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Aljoscha Krettek <al...@apache.org> ezt írta
>> > > > (időpont:
>> > > > > > > 2015.
>> > > > > > > > > júl.
>> > > > > > > > > > > 7.,
>> > > > > > > > > > > > > K,
>> > > > > > > > > > > > > 15:35):
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi,
>> > > > > > > > > > > > > > I just noticed that we don't have anything about
>> > how
>> > > > > > > iterations
>> > > > > > > > > and
>> > > > > > > > > > > > > > timestamps/watermarks should interact.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Cheers,
>> > > > > > > > > > > > > > Aljoscha
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <
>> > > > > sewen@apache.org
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hi all!
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > As many of you know, there are a ongoing
>> efforts
>> > to
>> > > > > > > > consolidate
>> > > > > > > > > > the
>> > > > > > > > > > > > > > > streaming API for the next release, and then
>> > > graduate
>> > > > > it
>> > > > > > > > (from
>> > > > > > > > > > beta
>> > > > > > > > > > > > > > > status).
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > In the process of this consolidation, we want
>> to
>> > > > > achieve
>> > > > > > > the
>> > > > > > > > > > > > following
>> > > > > > > > > > > > > > > goals.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >  - Make the code more robust and simplify it
>> in
>> > > parts
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >  - Clearly define the semantics of the
>> > constructs.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >  - Prepare it for support of more advanced
>> > > concepts,
>> > > > > like
>> > > > > > > > > > > > partitionable
>> > > > > > > > > > > > > > > state, and event time.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >  - Cut support for certain corner cases that
>> were
>> > > > > > > prototyped,
>> > > > > > > > > but
>> > > > > > > > > > > > > turned
>> > > > > > > > > > > > > > > out to be not efficiently doable
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Based on prior discussions on the mailing
>> list,
>> > > > > Aljoscha
>> > > > > > > and
>> > > > > > > > me
>> > > > > > > > > > > > drafted
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > design documents below, which outline how the
>> > > > > > consolidated
>> > > > > > > > API
>> > > > > > > > > > > would
>> > > > > > > > > > > > > > like.
>> > > > > > > > > > > > > > > We focused in constructs, time, and window
>> > > semantics.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Design document on how to restructure the
>> > Streaming
>> > > > > API:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Design document on definitions of time, order,
>> > and
>> > > > the
>> > > > > > > > > resulting
>> > > > > > > > > > > > > > semantics:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Note: The design of the interfaces and
>> concepts
>> > for
>> > > > > > > advanced
>> > > > > > > > > > state
>> > > > > > > > > > > in
>> > > > > > > > > > > > > > > functions is not in here. That is part of a
>> > > separate
>> > > > > > design
>> > > > > > > > > > > > discussion
>> > > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > orthogonal to the designs drafted here.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Please have a look and voice questions and
>> > > concerns.
>> > > > > > Since
>> > > > > > > we
>> > > > > > > > > > > should
>> > > > > > > > > > > > > not
>> > > > > > > > > > > > > > > break the streaming API more than once, we
>> should
>> > > > make
>> > > > > > sure
>> > > > > > > > > this
>> > > > > > > > > > > > > > > consolidation brings it into the shape we
>> want it
>> > > to
>> > > > be
>> > > > > > in.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Greetings,
>> > > > > > > > > > > > > > > Stephan
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Design documents for consolidated DataStream API

[Stephan Ewen <se...@apache.org>]

keyBy() does not do any grouping. Grouping in streams in not defined
without windows.

On Tue, Jul 14, 2015 at 10:48 AM, Gyula Fóra <gy...@gmail.com> wrote:

> If we only want to have either keyBy or groupBy, why not keep groupBy? That
> would be more consistent with the batch api.
> On Tue, Jul 14, 2015 at 10:35 AM Stephan Ewen <se...@apache.org> wrote:
>
> > Concerning your comments:
> >
> > 1) In the new design, there is no grouping without windowing. The
> > KeyedDataStream subsumes the grouping and key-ing for partitioned state.
> >
> >     The keyBy() + window() makes a parallel grouped window
> >     keyBy() alone allows access to partitioned state.
> >
> >     My thought was that this is simpler, because it needs not groupBy()
> and
> > keyBy(), but one construct to handle both cases.
> >
> > 2) The discretization is a rough thought and is nothing for the short
> term.
> > It totally needs more thoughts. I put it there to have it as a sketch for
> > how to evolve this.
> >
> >     The idea is of course to not have a single data set, but a series of
> > data set. In each discrete time slice, the data set can be treated like a
> > regular data set.
> >
> >     Let's kick off a separate design for the discretization. Joins are
> good
> > to talk about (data sets can be joined with data set), and I am sure
> there
> > are more questions coming up.
> >
> >
> > Does that make sense?
> >
> >
> >
> >
> >
> > On Tue, Jul 14, 2015 at 10:05 AM, Gyula Fóra <gy...@gmail.com>
> wrote:
> >
> > > I think Marton has some good points here.
> > >
> > > 1) Is KeyedDataStream a better name if this is only a renaming?
> > >
> > > 2) the discretize semantics is unclear indeed. Are we operating on a
> > single
> > > or sequence of datasets? If the latter why not call it something else
> > > (dstream). How are joins and other binary operators defined for
> different
> > > discretizations etc.
> > > On Mon, Jul 13, 2015 at 7:37 PM Márton Balassi <mb...@apache.org>
> > > wrote:
> > >
> > > > Generally I agree with the new design. Two concerns:
> > > >
> > > > 1) Does KeyedDataStream replace GroupedDataStream or is it the
> latter a
> > > > special case of the former?
> > > >
> > > > The KeyedDataStream as described in the design document is a bit
> > unclear
> > > > for me. It lists the following usages:
> > > >   a) It is the first step in building a window stream, on top of
> which
> > > the
> > > > grouped/windowed aggregation and reduce-style function can be applied
> > > >   b) It allows to use the "by-key" state of functions. Here, every
> > record
> > > > has access to a state that is scoped by its key. Key-scoped state can
> > be
> > > > automatically redistributed and repartitioned.
> > > >
> > > > The code snippet describes a use case where the computation and the
> > > access
> > > > of the state is used the way currently the GroupedDataStream should
> > > work. I
> > > > suppose this is the example for case b). Would case a) also window
> > > elements
> > > > by key? If yes, then this is practically a renaming and enhancement
> of
> > > the
> > > > GroupedDataStream functionality with keyed state. Then the
> > > > StreamExecutionEnvironment.createKeyedStream(Partitioner,
> > > > KeySelector)construction does not make much sense as the user only
> > > operates
> > > > within the scope of the keyselector and not the partitioner anyway.
> > > >
> > > > I personally think KeyedDataStream as a name does not necessarily
> > suggest
> > > > that the records are grouped by key, it only suggests partitioning by
> > > key -
> > > > at least for me. :)
> > > >
> > > > 2) The API for discretization is not convenient IMHO
> > > >
> > > > The discretization part declares that the output of
> > > DataStream.discretize()
> > > > is a sequence of DataSets. I love this approach, but then in the code
> > > > snippet the return value of this function is simply a DataSet and
> uses
> > it
> > > > as such. The take home message of that code is the following: this is
> > > > actually the way you would like to program on these sequence of
> > DataSets,
> > > > most probably you would like to do the same with each of them. If
> that
> > is
> > > > the case we should provide a nice utility for that. I think Spark
> > > > Streaming's DStream.foreachRDD() is fairly useful for this purpose.
> > > >
> > > > On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <gy...@gmail.com>
> > > wrote:
> > > >
> > > > > +1
> > > > > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <se...@apache.org>
> > wrote:
> > > > >
> > > > > > If naming is the only concern, then we should go ahead, because
> we
> > > can
> > > > > > change names easily (before the release).
> > > > > >
> > > > > > In fact, I don't think it leaves a bad impression. Global windows
> > are
> > > > > > non-parallel windows. There are also parallel windows. Pick what
> > you
> > > > need
> > > > > > and what works.
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <
> gyula.fora@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > I think we agree on everything its more of a naming issue :)
> > > > > > >
> > > > > > > I thought it might be misleading that global time windows are
> > > > > > > "non-parallel" windows. We dont want to give a bad impression.
> > > (Also
> > > > we
> > > > > > > dont want them to think that every global window is parallel
> but
> > > > thats
> > > > > > not
> > > > > > > a problem here)
> > > > > > >
> > > > > > > Gyula
> > > > > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <sewen@apache.org
> >
> > > > wrote:
> > > > > > >
> > > > > > > > Okay, what is missing about the windowing in your opinion?
> > > > > > > >
> > > > > > > > The core points of the document are:
> > > > > > > >
> > > > > > > >   - The parallel windows are per group only.
> > > > > > > >
> > > > > > > >   - The implementation of the parallel windows holds window
> > data
> > > in
> > > > > the
> > > > > > > > group buffers.
> > > > > > > >
> > > > > > > >   - The global windows are non-parallel. May have parallel
> > > > > > > pre-aggregation,
> > > > > > > > if they are time windows.
> > > > > > > >
> > > > > > > >   - Time may be operator time (timer thread), or watermark
> > time.
> > > > > > > Watermark
> > > > > > > > time can refer to ingress or event time.
> > > > > > > >
> > > > > > > >   - Windows that do not pre-aggregate may require elements in
> > > > order.
> > > > > > Not
> > > > > > > > part of the first prototype.
> > > > > > > >
> > > > > > > > Do we agree on those points?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <
> > > gyula.fora@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > In general I like it, although the main difference between
> > the
> > > > > > current
> > > > > > > > and
> > > > > > > > > the new one is the windowing and that is still not very
> > clear.
> > > > > > > > >
> > > > > > > > > Where do we have the full stream time windows for
> > > instance?(which
> > > > > is
> > > > > > > > > parallel but not keyed)
> > > > > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> > > > > > aljoscha@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 I like it as well.
> > > > > > > > > >
> > > > > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <
> > > > ktzoumas@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1 from my side
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <
> > > > > sewen@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Do we have consensus on these designs?
> > > > > > > > > > > >
> > > > > > > > > > > > If we have, we should get to implementing this soon,
> > > > because
> > > > > > > > > basically
> > > > > > > > > > > all
> > > > > > > > > > > > streaming patches will have to be revisited in light
> of
> > > > > this...
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> > > > > > gyula.fora@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > You are right thats an important issue.
> > > > > > > > > > > > >
> > > > > > > > > > > > > And I think we should also do some renaming with
> the
> > > > > > > "iterations"
> > > > > > > > > > > because
> > > > > > > > > > > > > they are not really iterations like in the batch
> case
> > > and
> > > > > it
> > > > > > > > might
> > > > > > > > > > > > confuse
> > > > > > > > > > > > > some users.
> > > > > > > > > > > > > Maybe we can call them loops or cycles and rename
> the
> > > api
> > > > > > calls
> > > > > > > > to
> > > > > > > > > > make
> > > > > > > > > > > > it
> > > > > > > > > > > > > more intuitive what happens. It is really just a
> > cyclic
> > > > > > > dataflow.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Aljoscha Krettek <al...@apache.org> ezt írta
> > > > (időpont:
> > > > > > > 2015.
> > > > > > > > > júl.
> > > > > > > > > > > 7.,
> > > > > > > > > > > > > K,
> > > > > > > > > > > > > 15:35):
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > > I just noticed that we don't have anything about
> > how
> > > > > > > iterations
> > > > > > > > > and
> > > > > > > > > > > > > > timestamps/watermarks should interact.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <
> > > > > sewen@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As many of you know, there are a ongoing
> efforts
> > to
> > > > > > > > consolidate
> > > > > > > > > > the
> > > > > > > > > > > > > > > streaming API for the next release, and then
> > > graduate
> > > > > it
> > > > > > > > (from
> > > > > > > > > > beta
> > > > > > > > > > > > > > > status).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In the process of this consolidation, we want
> to
> > > > > achieve
> > > > > > > the
> > > > > > > > > > > > following
> > > > > > > > > > > > > > > goals.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  - Make the code more robust and simplify it in
> > > parts
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  - Clearly define the semantics of the
> > constructs.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  - Prepare it for support of more advanced
> > > concepts,
> > > > > like
> > > > > > > > > > > > partitionable
> > > > > > > > > > > > > > > state, and event time.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  - Cut support for certain corner cases that
> were
> > > > > > > prototyped,
> > > > > > > > > but
> > > > > > > > > > > > > turned
> > > > > > > > > > > > > > > out to be not efficiently doable
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Based on prior discussions on the mailing list,
> > > > > Aljoscha
> > > > > > > and
> > > > > > > > me
> > > > > > > > > > > > drafted
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > design documents below, which outline how the
> > > > > > consolidated
> > > > > > > > API
> > > > > > > > > > > would
> > > > > > > > > > > > > > like.
> > > > > > > > > > > > > > > We focused in constructs, time, and window
> > > semantics.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Design document on how to restructure the
> > Streaming
> > > > > API:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Design document on definitions of time, order,
> > and
> > > > the
> > > > > > > > > resulting
> > > > > > > > > > > > > > semantics:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Note: The design of the interfaces and concepts
> > for
> > > > > > > advanced
> > > > > > > > > > state
> > > > > > > > > > > in
> > > > > > > > > > > > > > > functions is not in here. That is part of a
> > > separate
> > > > > > design
> > > > > > > > > > > > discussion
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > orthogonal to the designs drafted here.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Please have a look and voice questions and
> > > concerns.
> > > > > > Since
> > > > > > > we
> > > > > > > > > > > should
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > break the streaming API more than once, we
> should
> > > > make
> > > > > > sure
> > > > > > > > > this
> > > > > > > > > > > > > > > consolidation brings it into the shape we want
> it
> > > to
> > > > be
> > > > > > in.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Greetings,
> > > > > > > > > > > > > > > Stephan
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Gyula Fóra <gy...@gmail.com>]

If we only want to have either keyBy or groupBy, why not keep groupBy? That
would be more consistent with the batch api.
On Tue, Jul 14, 2015 at 10:35 AM Stephan Ewen <se...@apache.org> wrote:

> Concerning your comments:
>
> 1) In the new design, there is no grouping without windowing. The
> KeyedDataStream subsumes the grouping and key-ing for partitioned state.
>
>     The keyBy() + window() makes a parallel grouped window
>     keyBy() alone allows access to partitioned state.
>
>     My thought was that this is simpler, because it needs not groupBy() and
> keyBy(), but one construct to handle both cases.
>
> 2) The discretization is a rough thought and is nothing for the short term.
> It totally needs more thoughts. I put it there to have it as a sketch for
> how to evolve this.
>
>     The idea is of course to not have a single data set, but a series of
> data set. In each discrete time slice, the data set can be treated like a
> regular data set.
>
>     Let's kick off a separate design for the discretization. Joins are good
> to talk about (data sets can be joined with data set), and I am sure there
> are more questions coming up.
>
>
> Does that make sense?
>
>
>
>
>
> On Tue, Jul 14, 2015 at 10:05 AM, Gyula Fóra <gy...@gmail.com> wrote:
>
> > I think Marton has some good points here.
> >
> > 1) Is KeyedDataStream a better name if this is only a renaming?
> >
> > 2) the discretize semantics is unclear indeed. Are we operating on a
> single
> > or sequence of datasets? If the latter why not call it something else
> > (dstream). How are joins and other binary operators defined for different
> > discretizations etc.
> > On Mon, Jul 13, 2015 at 7:37 PM Márton Balassi <mb...@apache.org>
> > wrote:
> >
> > > Generally I agree with the new design. Two concerns:
> > >
> > > 1) Does KeyedDataStream replace GroupedDataStream or is it the latter a
> > > special case of the former?
> > >
> > > The KeyedDataStream as described in the design document is a bit
> unclear
> > > for me. It lists the following usages:
> > >   a) It is the first step in building a window stream, on top of which
> > the
> > > grouped/windowed aggregation and reduce-style function can be applied
> > >   b) It allows to use the "by-key" state of functions. Here, every
> record
> > > has access to a state that is scoped by its key. Key-scoped state can
> be
> > > automatically redistributed and repartitioned.
> > >
> > > The code snippet describes a use case where the computation and the
> > access
> > > of the state is used the way currently the GroupedDataStream should
> > work. I
> > > suppose this is the example for case b). Would case a) also window
> > elements
> > > by key? If yes, then this is practically a renaming and enhancement of
> > the
> > > GroupedDataStream functionality with keyed state. Then the
> > > StreamExecutionEnvironment.createKeyedStream(Partitioner,
> > > KeySelector)construction does not make much sense as the user only
> > operates
> > > within the scope of the keyselector and not the partitioner anyway.
> > >
> > > I personally think KeyedDataStream as a name does not necessarily
> suggest
> > > that the records are grouped by key, it only suggests partitioning by
> > key -
> > > at least for me. :)
> > >
> > > 2) The API for discretization is not convenient IMHO
> > >
> > > The discretization part declares that the output of
> > DataStream.discretize()
> > > is a sequence of DataSets. I love this approach, but then in the code
> > > snippet the return value of this function is simply a DataSet and uses
> it
> > > as such. The take home message of that code is the following: this is
> > > actually the way you would like to program on these sequence of
> DataSets,
> > > most probably you would like to do the same with each of them. If that
> is
> > > the case we should provide a nice utility for that. I think Spark
> > > Streaming's DStream.foreachRDD() is fairly useful for this purpose.
> > >
> > > On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <gy...@gmail.com>
> > wrote:
> > >
> > > > +1
> > > > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <se...@apache.org>
> wrote:
> > > >
> > > > > If naming is the only concern, then we should go ahead, because we
> > can
> > > > > change names easily (before the release).
> > > > >
> > > > > In fact, I don't think it leaves a bad impression. Global windows
> are
> > > > > non-parallel windows. There are also parallel windows. Pick what
> you
> > > need
> > > > > and what works.
> > > > >
> > > > >
> > > > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <gy...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I think we agree on everything its more of a naming issue :)
> > > > > >
> > > > > > I thought it might be misleading that global time windows are
> > > > > > "non-parallel" windows. We dont want to give a bad impression.
> > (Also
> > > we
> > > > > > dont want them to think that every global window is parallel but
> > > thats
> > > > > not
> > > > > > a problem here)
> > > > > >
> > > > > > Gyula
> > > > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > Okay, what is missing about the windowing in your opinion?
> > > > > > >
> > > > > > > The core points of the document are:
> > > > > > >
> > > > > > >   - The parallel windows are per group only.
> > > > > > >
> > > > > > >   - The implementation of the parallel windows holds window
> data
> > in
> > > > the
> > > > > > > group buffers.
> > > > > > >
> > > > > > >   - The global windows are non-parallel. May have parallel
> > > > > > pre-aggregation,
> > > > > > > if they are time windows.
> > > > > > >
> > > > > > >   - Time may be operator time (timer thread), or watermark
> time.
> > > > > > Watermark
> > > > > > > time can refer to ingress or event time.
> > > > > > >
> > > > > > >   - Windows that do not pre-aggregate may require elements in
> > > order.
> > > > > Not
> > > > > > > part of the first prototype.
> > > > > > >
> > > > > > > Do we agree on those points?
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <
> > gyula.fora@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > In general I like it, although the main difference between
> the
> > > > > current
> > > > > > > and
> > > > > > > > the new one is the windowing and that is still not very
> clear.
> > > > > > > >
> > > > > > > > Where do we have the full stream time windows for
> > instance?(which
> > > > is
> > > > > > > > parallel but not keyed)
> > > > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> > > > > aljoscha@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 I like it as well.
> > > > > > > > >
> > > > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <
> > > ktzoumas@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 from my side
> > > > > > > > > >
> > > > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <
> > > > sewen@apache.org>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Do we have consensus on these designs?
> > > > > > > > > > >
> > > > > > > > > > > If we have, we should get to implementing this soon,
> > > because
> > > > > > > > basically
> > > > > > > > > > all
> > > > > > > > > > > streaming patches will have to be revisited in light of
> > > > this...
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> > > > > gyula.fora@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > You are right thats an important issue.
> > > > > > > > > > > >
> > > > > > > > > > > > And I think we should also do some renaming with the
> > > > > > "iterations"
> > > > > > > > > > because
> > > > > > > > > > > > they are not really iterations like in the batch case
> > and
> > > > it
> > > > > > > might
> > > > > > > > > > > confuse
> > > > > > > > > > > > some users.
> > > > > > > > > > > > Maybe we can call them loops or cycles and rename the
> > api
> > > > > calls
> > > > > > > to
> > > > > > > > > make
> > > > > > > > > > > it
> > > > > > > > > > > > more intuitive what happens. It is really just a
> cyclic
> > > > > > dataflow.
> > > > > > > > > > > >
> > > > > > > > > > > > Aljoscha Krettek <al...@apache.org> ezt írta
> > > (időpont:
> > > > > > 2015.
> > > > > > > > júl.
> > > > > > > > > > 7.,
> > > > > > > > > > > > K,
> > > > > > > > > > > > 15:35):
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > I just noticed that we don't have anything about
> how
> > > > > > iterations
> > > > > > > > and
> > > > > > > > > > > > > timestamps/watermarks should interact.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Aljoscha
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <
> > > > sewen@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > As many of you know, there are a ongoing efforts
> to
> > > > > > > consolidate
> > > > > > > > > the
> > > > > > > > > > > > > > streaming API for the next release, and then
> > graduate
> > > > it
> > > > > > > (from
> > > > > > > > > beta
> > > > > > > > > > > > > > status).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In the process of this consolidation, we want to
> > > > achieve
> > > > > > the
> > > > > > > > > > > following
> > > > > > > > > > > > > > goals.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  - Make the code more robust and simplify it in
> > parts
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  - Clearly define the semantics of the
> constructs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  - Prepare it for support of more advanced
> > concepts,
> > > > like
> > > > > > > > > > > partitionable
> > > > > > > > > > > > > > state, and event time.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  - Cut support for certain corner cases that were
> > > > > > prototyped,
> > > > > > > > but
> > > > > > > > > > > > turned
> > > > > > > > > > > > > > out to be not efficiently doable
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Based on prior discussions on the mailing list,
> > > > Aljoscha
> > > > > > and
> > > > > > > me
> > > > > > > > > > > drafted
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > design documents below, which outline how the
> > > > > consolidated
> > > > > > > API
> > > > > > > > > > would
> > > > > > > > > > > > > like.
> > > > > > > > > > > > > > We focused in constructs, time, and window
> > semantics.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Design document on how to restructure the
> Streaming
> > > > API:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Design document on definitions of time, order,
> and
> > > the
> > > > > > > > resulting
> > > > > > > > > > > > > semantics:
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Note: The design of the interfaces and concepts
> for
> > > > > > advanced
> > > > > > > > > state
> > > > > > > > > > in
> > > > > > > > > > > > > > functions is not in here. That is part of a
> > separate
> > > > > design
> > > > > > > > > > > discussion
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > orthogonal to the designs drafted here.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Please have a look and voice questions and
> > concerns.
> > > > > Since
> > > > > > we
> > > > > > > > > > should
> > > > > > > > > > > > not
> > > > > > > > > > > > > > break the streaming API more than once, we should
> > > make
> > > > > sure
> > > > > > > > this
> > > > > > > > > > > > > > consolidation brings it into the shape we want it
> > to
> > > be
> > > > > in.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Greetings,
> > > > > > > > > > > > > > Stephan
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Stephan Ewen <se...@apache.org>]

Concerning your comments:

1) In the new design, there is no grouping without windowing. The
KeyedDataStream subsumes the grouping and key-ing for partitioned state.

    The keyBy() + window() makes a parallel grouped window
    keyBy() alone allows access to partitioned state.

    My thought was that this is simpler, because it needs not groupBy() and
keyBy(), but one construct to handle both cases.

2) The discretization is a rough thought and is nothing for the short term.
It totally needs more thoughts. I put it there to have it as a sketch for
how to evolve this.

    The idea is of course to not have a single data set, but a series of
data set. In each discrete time slice, the data set can be treated like a
regular data set.

    Let's kick off a separate design for the discretization. Joins are good
to talk about (data sets can be joined with data set), and I am sure there
are more questions coming up.


Does that make sense?





On Tue, Jul 14, 2015 at 10:05 AM, Gyula Fóra <gy...@gmail.com> wrote:

> I think Marton has some good points here.
>
> 1) Is KeyedDataStream a better name if this is only a renaming?
>
> 2) the discretize semantics is unclear indeed. Are we operating on a single
> or sequence of datasets? If the latter why not call it something else
> (dstream). How are joins and other binary operators defined for different
> discretizations etc.
> On Mon, Jul 13, 2015 at 7:37 PM Márton Balassi <mb...@apache.org>
> wrote:
>
> > Generally I agree with the new design. Two concerns:
> >
> > 1) Does KeyedDataStream replace GroupedDataStream or is it the latter a
> > special case of the former?
> >
> > The KeyedDataStream as described in the design document is a bit unclear
> > for me. It lists the following usages:
> >   a) It is the first step in building a window stream, on top of which
> the
> > grouped/windowed aggregation and reduce-style function can be applied
> >   b) It allows to use the "by-key" state of functions. Here, every record
> > has access to a state that is scoped by its key. Key-scoped state can be
> > automatically redistributed and repartitioned.
> >
> > The code snippet describes a use case where the computation and the
> access
> > of the state is used the way currently the GroupedDataStream should
> work. I
> > suppose this is the example for case b). Would case a) also window
> elements
> > by key? If yes, then this is practically a renaming and enhancement of
> the
> > GroupedDataStream functionality with keyed state. Then the
> > StreamExecutionEnvironment.createKeyedStream(Partitioner,
> > KeySelector)construction does not make much sense as the user only
> operates
> > within the scope of the keyselector and not the partitioner anyway.
> >
> > I personally think KeyedDataStream as a name does not necessarily suggest
> > that the records are grouped by key, it only suggests partitioning by
> key -
> > at least for me. :)
> >
> > 2) The API for discretization is not convenient IMHO
> >
> > The discretization part declares that the output of
> DataStream.discretize()
> > is a sequence of DataSets. I love this approach, but then in the code
> > snippet the return value of this function is simply a DataSet and uses it
> > as such. The take home message of that code is the following: this is
> > actually the way you would like to program on these sequence of DataSets,
> > most probably you would like to do the same with each of them. If that is
> > the case we should provide a nice utility for that. I think Spark
> > Streaming's DStream.foreachRDD() is fairly useful for this purpose.
> >
> > On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <gy...@gmail.com>
> wrote:
> >
> > > +1
> > > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <se...@apache.org> wrote:
> > >
> > > > If naming is the only concern, then we should go ahead, because we
> can
> > > > change names easily (before the release).
> > > >
> > > > In fact, I don't think it leaves a bad impression. Global windows are
> > > > non-parallel windows. There are also parallel windows. Pick what you
> > need
> > > > and what works.
> > > >
> > > >
> > > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <gy...@gmail.com>
> > > wrote:
> > > >
> > > > > I think we agree on everything its more of a naming issue :)
> > > > >
> > > > > I thought it might be misleading that global time windows are
> > > > > "non-parallel" windows. We dont want to give a bad impression.
> (Also
> > we
> > > > > dont want them to think that every global window is parallel but
> > thats
> > > > not
> > > > > a problem here)
> > > > >
> > > > > Gyula
> > > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <se...@apache.org>
> > wrote:
> > > > >
> > > > > > Okay, what is missing about the windowing in your opinion?
> > > > > >
> > > > > > The core points of the document are:
> > > > > >
> > > > > >   - The parallel windows are per group only.
> > > > > >
> > > > > >   - The implementation of the parallel windows holds window data
> in
> > > the
> > > > > > group buffers.
> > > > > >
> > > > > >   - The global windows are non-parallel. May have parallel
> > > > > pre-aggregation,
> > > > > > if they are time windows.
> > > > > >
> > > > > >   - Time may be operator time (timer thread), or watermark time.
> > > > > Watermark
> > > > > > time can refer to ingress or event time.
> > > > > >
> > > > > >   - Windows that do not pre-aggregate may require elements in
> > order.
> > > > Not
> > > > > > part of the first prototype.
> > > > > >
> > > > > > Do we agree on those points?
> > > > > >
> > > > > >
> > > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <
> gyula.fora@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > In general I like it, although the main difference between the
> > > > current
> > > > > > and
> > > > > > > the new one is the windowing and that is still not very clear.
> > > > > > >
> > > > > > > Where do we have the full stream time windows for
> instance?(which
> > > is
> > > > > > > parallel but not keyed)
> > > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> > > > aljoscha@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 I like it as well.
> > > > > > > >
> > > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <
> > ktzoumas@apache.org
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 from my side
> > > > > > > > >
> > > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <
> > > sewen@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Do we have consensus on these designs?
> > > > > > > > > >
> > > > > > > > > > If we have, we should get to implementing this soon,
> > because
> > > > > > > basically
> > > > > > > > > all
> > > > > > > > > > streaming patches will have to be revisited in light of
> > > this...
> > > > > > > > > >
> > > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> > > > gyula.fora@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > You are right thats an important issue.
> > > > > > > > > > >
> > > > > > > > > > > And I think we should also do some renaming with the
> > > > > "iterations"
> > > > > > > > > because
> > > > > > > > > > > they are not really iterations like in the batch case
> and
> > > it
> > > > > > might
> > > > > > > > > > confuse
> > > > > > > > > > > some users.
> > > > > > > > > > > Maybe we can call them loops or cycles and rename the
> api
> > > > calls
> > > > > > to
> > > > > > > > make
> > > > > > > > > > it
> > > > > > > > > > > more intuitive what happens. It is really just a cyclic
> > > > > dataflow.
> > > > > > > > > > >
> > > > > > > > > > > Aljoscha Krettek <al...@apache.org> ezt írta
> > (időpont:
> > > > > 2015.
> > > > > > > júl.
> > > > > > > > > 7.,
> > > > > > > > > > > K,
> > > > > > > > > > > 15:35):
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > > I just noticed that we don't have anything about how
> > > > > iterations
> > > > > > > and
> > > > > > > > > > > > timestamps/watermarks should interact.
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Aljoscha
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <
> > > sewen@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all!
> > > > > > > > > > > > >
> > > > > > > > > > > > > As many of you know, there are a ongoing efforts to
> > > > > > consolidate
> > > > > > > > the
> > > > > > > > > > > > > streaming API for the next release, and then
> graduate
> > > it
> > > > > > (from
> > > > > > > > beta
> > > > > > > > > > > > > status).
> > > > > > > > > > > > >
> > > > > > > > > > > > > In the process of this consolidation, we want to
> > > achieve
> > > > > the
> > > > > > > > > > following
> > > > > > > > > > > > > goals.
> > > > > > > > > > > > >
> > > > > > > > > > > > >  - Make the code more robust and simplify it in
> parts
> > > > > > > > > > > > >
> > > > > > > > > > > > >  - Clearly define the semantics of the constructs.
> > > > > > > > > > > > >
> > > > > > > > > > > > >  - Prepare it for support of more advanced
> concepts,
> > > like
> > > > > > > > > > partitionable
> > > > > > > > > > > > > state, and event time.
> > > > > > > > > > > > >
> > > > > > > > > > > > >  - Cut support for certain corner cases that were
> > > > > prototyped,
> > > > > > > but
> > > > > > > > > > > turned
> > > > > > > > > > > > > out to be not efficiently doable
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Based on prior discussions on the mailing list,
> > > Aljoscha
> > > > > and
> > > > > > me
> > > > > > > > > > drafted
> > > > > > > > > > > > the
> > > > > > > > > > > > > design documents below, which outline how the
> > > > consolidated
> > > > > > API
> > > > > > > > > would
> > > > > > > > > > > > like.
> > > > > > > > > > > > > We focused in constructs, time, and window
> semantics.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Design document on how to restructure the Streaming
> > > API:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > > > > > > > > > >
> > > > > > > > > > > > > Design document on definitions of time, order, and
> > the
> > > > > > > resulting
> > > > > > > > > > > > semantics:
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Note: The design of the interfaces and concepts for
> > > > > advanced
> > > > > > > > state
> > > > > > > > > in
> > > > > > > > > > > > > functions is not in here. That is part of a
> separate
> > > > design
> > > > > > > > > > discussion
> > > > > > > > > > > > and
> > > > > > > > > > > > > orthogonal to the designs drafted here.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please have a look and voice questions and
> concerns.
> > > > Since
> > > > > we
> > > > > > > > > should
> > > > > > > > > > > not
> > > > > > > > > > > > > break the streaming API more than once, we should
> > make
> > > > sure
> > > > > > > this
> > > > > > > > > > > > > consolidation brings it into the shape we want it
> to
> > be
> > > > in.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Greetings,
> > > > > > > > > > > > > Stephan
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Gyula Fóra <gy...@gmail.com>]

I think Marton has some good points here.

1) Is KeyedDataStream a better name if this is only a renaming?

2) the discretize semantics is unclear indeed. Are we operating on a single
or sequence of datasets? If the latter why not call it something else
(dstream). How are joins and other binary operators defined for different
discretizations etc.
On Mon, Jul 13, 2015 at 7:37 PM Márton Balassi <mb...@apache.org> wrote:

> Generally I agree with the new design. Two concerns:
>
> 1) Does KeyedDataStream replace GroupedDataStream or is it the latter a
> special case of the former?
>
> The KeyedDataStream as described in the design document is a bit unclear
> for me. It lists the following usages:
>   a) It is the first step in building a window stream, on top of which the
> grouped/windowed aggregation and reduce-style function can be applied
>   b) It allows to use the "by-key" state of functions. Here, every record
> has access to a state that is scoped by its key. Key-scoped state can be
> automatically redistributed and repartitioned.
>
> The code snippet describes a use case where the computation and the access
> of the state is used the way currently the GroupedDataStream should work. I
> suppose this is the example for case b). Would case a) also window elements
> by key? If yes, then this is practically a renaming and enhancement of the
> GroupedDataStream functionality with keyed state. Then the
> StreamExecutionEnvironment.createKeyedStream(Partitioner,
> KeySelector)construction does not make much sense as the user only operates
> within the scope of the keyselector and not the partitioner anyway.
>
> I personally think KeyedDataStream as a name does not necessarily suggest
> that the records are grouped by key, it only suggests partitioning by key -
> at least for me. :)
>
> 2) The API for discretization is not convenient IMHO
>
> The discretization part declares that the output of DataStream.discretize()
> is a sequence of DataSets. I love this approach, but then in the code
> snippet the return value of this function is simply a DataSet and uses it
> as such. The take home message of that code is the following: this is
> actually the way you would like to program on these sequence of DataSets,
> most probably you would like to do the same with each of them. If that is
> the case we should provide a nice utility for that. I think Spark
> Streaming's DStream.foreachRDD() is fairly useful for this purpose.
>
> On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <gy...@gmail.com> wrote:
>
> > +1
> > On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <se...@apache.org> wrote:
> >
> > > If naming is the only concern, then we should go ahead, because we can
> > > change names easily (before the release).
> > >
> > > In fact, I don't think it leaves a bad impression. Global windows are
> > > non-parallel windows. There are also parallel windows. Pick what you
> need
> > > and what works.
> > >
> > >
> > > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <gy...@gmail.com>
> > wrote:
> > >
> > > > I think we agree on everything its more of a naming issue :)
> > > >
> > > > I thought it might be misleading that global time windows are
> > > > "non-parallel" windows. We dont want to give a bad impression. (Also
> we
> > > > dont want them to think that every global window is parallel but
> thats
> > > not
> > > > a problem here)
> > > >
> > > > Gyula
> > > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <se...@apache.org>
> wrote:
> > > >
> > > > > Okay, what is missing about the windowing in your opinion?
> > > > >
> > > > > The core points of the document are:
> > > > >
> > > > >   - The parallel windows are per group only.
> > > > >
> > > > >   - The implementation of the parallel windows holds window data in
> > the
> > > > > group buffers.
> > > > >
> > > > >   - The global windows are non-parallel. May have parallel
> > > > pre-aggregation,
> > > > > if they are time windows.
> > > > >
> > > > >   - Time may be operator time (timer thread), or watermark time.
> > > > Watermark
> > > > > time can refer to ingress or event time.
> > > > >
> > > > >   - Windows that do not pre-aggregate may require elements in
> order.
> > > Not
> > > > > part of the first prototype.
> > > > >
> > > > > Do we agree on those points?
> > > > >
> > > > >
> > > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <gy...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > In general I like it, although the main difference between the
> > > current
> > > > > and
> > > > > > the new one is the windowing and that is still not very clear.
> > > > > >
> > > > > > Where do we have the full stream time windows for instance?(which
> > is
> > > > > > parallel but not keyed)
> > > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> > > aljoscha@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 I like it as well.
> > > > > > >
> > > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <
> ktzoumas@apache.org
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > +1 from my side
> > > > > > > >
> > > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <
> > sewen@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Do we have consensus on these designs?
> > > > > > > > >
> > > > > > > > > If we have, we should get to implementing this soon,
> because
> > > > > > basically
> > > > > > > > all
> > > > > > > > > streaming patches will have to be revisited in light of
> > this...
> > > > > > > > >
> > > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> > > gyula.fora@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > You are right thats an important issue.
> > > > > > > > > >
> > > > > > > > > > And I think we should also do some renaming with the
> > > > "iterations"
> > > > > > > > because
> > > > > > > > > > they are not really iterations like in the batch case and
> > it
> > > > > might
> > > > > > > > > confuse
> > > > > > > > > > some users.
> > > > > > > > > > Maybe we can call them loops or cycles and rename the api
> > > calls
> > > > > to
> > > > > > > make
> > > > > > > > > it
> > > > > > > > > > more intuitive what happens. It is really just a cyclic
> > > > dataflow.
> > > > > > > > > >
> > > > > > > > > > Aljoscha Krettek <al...@apache.org> ezt írta
> (időpont:
> > > > 2015.
> > > > > > júl.
> > > > > > > > 7.,
> > > > > > > > > > K,
> > > > > > > > > > 15:35):
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > > I just noticed that we don't have anything about how
> > > > iterations
> > > > > > and
> > > > > > > > > > > timestamps/watermarks should interact.
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Aljoscha
> > > > > > > > > > >
> > > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <
> > sewen@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all!
> > > > > > > > > > > >
> > > > > > > > > > > > As many of you know, there are a ongoing efforts to
> > > > > consolidate
> > > > > > > the
> > > > > > > > > > > > streaming API for the next release, and then graduate
> > it
> > > > > (from
> > > > > > > beta
> > > > > > > > > > > > status).
> > > > > > > > > > > >
> > > > > > > > > > > > In the process of this consolidation, we want to
> > achieve
> > > > the
> > > > > > > > > following
> > > > > > > > > > > > goals.
> > > > > > > > > > > >
> > > > > > > > > > > >  - Make the code more robust and simplify it in parts
> > > > > > > > > > > >
> > > > > > > > > > > >  - Clearly define the semantics of the constructs.
> > > > > > > > > > > >
> > > > > > > > > > > >  - Prepare it for support of more advanced concepts,
> > like
> > > > > > > > > partitionable
> > > > > > > > > > > > state, and event time.
> > > > > > > > > > > >
> > > > > > > > > > > >  - Cut support for certain corner cases that were
> > > > prototyped,
> > > > > > but
> > > > > > > > > > turned
> > > > > > > > > > > > out to be not efficiently doable
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Based on prior discussions on the mailing list,
> > Aljoscha
> > > > and
> > > > > me
> > > > > > > > > drafted
> > > > > > > > > > > the
> > > > > > > > > > > > design documents below, which outline how the
> > > consolidated
> > > > > API
> > > > > > > > would
> > > > > > > > > > > like.
> > > > > > > > > > > > We focused in constructs, time, and window semantics.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Design document on how to restructure the Streaming
> > API:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > > > > > > > > >
> > > > > > > > > > > > Design document on definitions of time, order, and
> the
> > > > > > resulting
> > > > > > > > > > > semantics:
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Note: The design of the interfaces and concepts for
> > > > advanced
> > > > > > > state
> > > > > > > > in
> > > > > > > > > > > > functions is not in here. That is part of a separate
> > > design
> > > > > > > > > discussion
> > > > > > > > > > > and
> > > > > > > > > > > > orthogonal to the designs drafted here.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Please have a look and voice questions and concerns.
> > > Since
> > > > we
> > > > > > > > should
> > > > > > > > > > not
> > > > > > > > > > > > break the streaming API more than once, we should
> make
> > > sure
> > > > > > this
> > > > > > > > > > > > consolidation brings it into the shape we want it to
> be
> > > in.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Greetings,
> > > > > > > > > > > > Stephan
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Márton Balassi <mb...@apache.org>]

Generally I agree with the new design. Two concerns:

1) Does KeyedDataStream replace GroupedDataStream or is it the latter a
special case of the former?

The KeyedDataStream as described in the design document is a bit unclear
for me. It lists the following usages:
  a) It is the first step in building a window stream, on top of which the
grouped/windowed aggregation and reduce-style function can be applied
  b) It allows to use the "by-key" state of functions. Here, every record
has access to a state that is scoped by its key. Key-scoped state can be
automatically redistributed and repartitioned.

The code snippet describes a use case where the computation and the access
of the state is used the way currently the GroupedDataStream should work. I
suppose this is the example for case b). Would case a) also window elements
by key? If yes, then this is practically a renaming and enhancement of the
GroupedDataStream functionality with keyed state. Then the
StreamExecutionEnvironment.createKeyedStream(Partitioner,
KeySelector)construction does not make much sense as the user only operates
within the scope of the keyselector and not the partitioner anyway.

I personally think KeyedDataStream as a name does not necessarily suggest
that the records are grouped by key, it only suggests partitioning by key -
at least for me. :)

2) The API for discretization is not convenient IMHO

The discretization part declares that the output of DataStream.discretize()
is a sequence of DataSets. I love this approach, but then in the code
snippet the return value of this function is simply a DataSet and uses it
as such. The take home message of that code is the following: this is
actually the way you would like to program on these sequence of DataSets,
most probably you would like to do the same with each of them. If that is
the case we should provide a nice utility for that. I think Spark
Streaming's DStream.foreachRDD() is fairly useful for this purpose.

On Mon, Jul 13, 2015 at 6:30 PM, Gyula Fóra <gy...@gmail.com> wrote:

> +1
> On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <se...@apache.org> wrote:
>
> > If naming is the only concern, then we should go ahead, because we can
> > change names easily (before the release).
> >
> > In fact, I don't think it leaves a bad impression. Global windows are
> > non-parallel windows. There are also parallel windows. Pick what you need
> > and what works.
> >
> >
> > On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <gy...@gmail.com>
> wrote:
> >
> > > I think we agree on everything its more of a naming issue :)
> > >
> > > I thought it might be misleading that global time windows are
> > > "non-parallel" windows. We dont want to give a bad impression. (Also we
> > > dont want them to think that every global window is parallel but thats
> > not
> > > a problem here)
> > >
> > > Gyula
> > > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <se...@apache.org> wrote:
> > >
> > > > Okay, what is missing about the windowing in your opinion?
> > > >
> > > > The core points of the document are:
> > > >
> > > >   - The parallel windows are per group only.
> > > >
> > > >   - The implementation of the parallel windows holds window data in
> the
> > > > group buffers.
> > > >
> > > >   - The global windows are non-parallel. May have parallel
> > > pre-aggregation,
> > > > if they are time windows.
> > > >
> > > >   - Time may be operator time (timer thread), or watermark time.
> > > Watermark
> > > > time can refer to ingress or event time.
> > > >
> > > >   - Windows that do not pre-aggregate may require elements in order.
> > Not
> > > > part of the first prototype.
> > > >
> > > > Do we agree on those points?
> > > >
> > > >
> > > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <gy...@gmail.com>
> > > wrote:
> > > >
> > > > > In general I like it, although the main difference between the
> > current
> > > > and
> > > > > the new one is the windowing and that is still not very clear.
> > > > >
> > > > > Where do we have the full stream time windows for instance?(which
> is
> > > > > parallel but not keyed)
> > > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> > aljoscha@apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1 I like it as well.
> > > > > >
> > > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <ktzoumas@apache.org
> >
> > > > wrote:
> > > > > >
> > > > > > > +1 from my side
> > > > > > >
> > > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <
> sewen@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Do we have consensus on these designs?
> > > > > > > >
> > > > > > > > If we have, we should get to implementing this soon, because
> > > > > basically
> > > > > > > all
> > > > > > > > streaming patches will have to be revisited in light of
> this...
> > > > > > > >
> > > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> > gyula.fora@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > You are right thats an important issue.
> > > > > > > > >
> > > > > > > > > And I think we should also do some renaming with the
> > > "iterations"
> > > > > > > because
> > > > > > > > > they are not really iterations like in the batch case and
> it
> > > > might
> > > > > > > > confuse
> > > > > > > > > some users.
> > > > > > > > > Maybe we can call them loops or cycles and rename the api
> > calls
> > > > to
> > > > > > make
> > > > > > > > it
> > > > > > > > > more intuitive what happens. It is really just a cyclic
> > > dataflow.
> > > > > > > > >
> > > > > > > > > Aljoscha Krettek <al...@apache.org> ezt írta (időpont:
> > > 2015.
> > > > > júl.
> > > > > > > 7.,
> > > > > > > > > K,
> > > > > > > > > 15:35):
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > > I just noticed that we don't have anything about how
> > > iterations
> > > > > and
> > > > > > > > > > timestamps/watermarks should interact.
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Aljoscha
> > > > > > > > > >
> > > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <
> sewen@apache.org
> > >
> > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all!
> > > > > > > > > > >
> > > > > > > > > > > As many of you know, there are a ongoing efforts to
> > > > consolidate
> > > > > > the
> > > > > > > > > > > streaming API for the next release, and then graduate
> it
> > > > (from
> > > > > > beta
> > > > > > > > > > > status).
> > > > > > > > > > >
> > > > > > > > > > > In the process of this consolidation, we want to
> achieve
> > > the
> > > > > > > > following
> > > > > > > > > > > goals.
> > > > > > > > > > >
> > > > > > > > > > >  - Make the code more robust and simplify it in parts
> > > > > > > > > > >
> > > > > > > > > > >  - Clearly define the semantics of the constructs.
> > > > > > > > > > >
> > > > > > > > > > >  - Prepare it for support of more advanced concepts,
> like
> > > > > > > > partitionable
> > > > > > > > > > > state, and event time.
> > > > > > > > > > >
> > > > > > > > > > >  - Cut support for certain corner cases that were
> > > prototyped,
> > > > > but
> > > > > > > > > turned
> > > > > > > > > > > out to be not efficiently doable
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Based on prior discussions on the mailing list,
> Aljoscha
> > > and
> > > > me
> > > > > > > > drafted
> > > > > > > > > > the
> > > > > > > > > > > design documents below, which outline how the
> > consolidated
> > > > API
> > > > > > > would
> > > > > > > > > > like.
> > > > > > > > > > > We focused in constructs, time, and window semantics.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Design document on how to restructure the Streaming
> API:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > > > > > > > >
> > > > > > > > > > > Design document on definitions of time, order, and the
> > > > > resulting
> > > > > > > > > > semantics:
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Note: The design of the interfaces and concepts for
> > > advanced
> > > > > > state
> > > > > > > in
> > > > > > > > > > > functions is not in here. That is part of a separate
> > design
> > > > > > > > discussion
> > > > > > > > > > and
> > > > > > > > > > > orthogonal to the designs drafted here.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Please have a look and voice questions and concerns.
> > Since
> > > we
> > > > > > > should
> > > > > > > > > not
> > > > > > > > > > > break the streaming API more than once, we should make
> > sure
> > > > > this
> > > > > > > > > > > consolidation brings it into the shape we want it to be
> > in.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Greetings,
> > > > > > > > > > > Stephan
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Paris Carbone <pa...@kth.se>]

+1 
No further concerns from my side either

> On 13 Jul 2015, at 18:30, Gyula Fóra <gy...@gmail.com> wrote:
> 
> +1
> On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <se...@apache.org> wrote:
> 
>> If naming is the only concern, then we should go ahead, because we can
>> change names easily (before the release).
>> 
>> In fact, I don't think it leaves a bad impression. Global windows are
>> non-parallel windows. There are also parallel windows. Pick what you need
>> and what works.
>> 
>> 
>> On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <gy...@gmail.com> wrote:
>> 
>>> I think we agree on everything its more of a naming issue :)
>>> 
>>> I thought it might be misleading that global time windows are
>>> "non-parallel" windows. We dont want to give a bad impression. (Also we
>>> dont want them to think that every global window is parallel but thats
>> not
>>> a problem here)
>>> 
>>> Gyula
>>> On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <se...@apache.org> wrote:
>>> 
>>>> Okay, what is missing about the windowing in your opinion?
>>>> 
>>>> The core points of the document are:
>>>> 
>>>>  - The parallel windows are per group only.
>>>> 
>>>>  - The implementation of the parallel windows holds window data in the
>>>> group buffers.
>>>> 
>>>>  - The global windows are non-parallel. May have parallel
>>> pre-aggregation,
>>>> if they are time windows.
>>>> 
>>>>  - Time may be operator time (timer thread), or watermark time.
>>> Watermark
>>>> time can refer to ingress or event time.
>>>> 
>>>>  - Windows that do not pre-aggregate may require elements in order.
>> Not
>>>> part of the first prototype.
>>>> 
>>>> Do we agree on those points?
>>>> 
>>>> 
>>>> On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <gy...@gmail.com>
>>> wrote:
>>>> 
>>>>> In general I like it, although the main difference between the
>> current
>>>> and
>>>>> the new one is the windowing and that is still not very clear.
>>>>> 
>>>>> Where do we have the full stream time windows for instance?(which is
>>>>> parallel but not keyed)
>>>>> On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
>> aljoscha@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> +1 I like it as well.
>>>>>> 
>>>>>> On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <kt...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>>> +1 from my side
>>>>>>> 
>>>>>>> On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <se...@apache.org>
>>>>> wrote:
>>>>>>> 
>>>>>>>> Do we have consensus on these designs?
>>>>>>>> 
>>>>>>>> If we have, we should get to implementing this soon, because
>>>>> basically
>>>>>>> all
>>>>>>>> streaming patches will have to be revisited in light of this...
>>>>>>>> 
>>>>>>>> On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
>> gyula.fora@gmail.com
>>>> 
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> You are right thats an important issue.
>>>>>>>>> 
>>>>>>>>> And I think we should also do some renaming with the
>>> "iterations"
>>>>>>> because
>>>>>>>>> they are not really iterations like in the batch case and it
>>>> might
>>>>>>>> confuse
>>>>>>>>> some users.
>>>>>>>>> Maybe we can call them loops or cycles and rename the api
>> calls
>>>> to
>>>>>> make
>>>>>>>> it
>>>>>>>>> more intuitive what happens. It is really just a cyclic
>>> dataflow.
>>>>>>>>> 
>>>>>>>>> Aljoscha Krettek <al...@apache.org> ezt írta (időpont:
>>> 2015.
>>>>> júl.
>>>>>>> 7.,
>>>>>>>>> K,
>>>>>>>>> 15:35):
>>>>>>>>> 
>>>>>>>>>> Hi,
>>>>>>>>>> I just noticed that we don't have anything about how
>>> iterations
>>>>> and
>>>>>>>>>> timestamps/watermarks should interact.
>>>>>>>>>> 
>>>>>>>>>> Cheers,
>>>>>>>>>> Aljoscha
>>>>>>>>>> 
>>>>>>>>>> On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <sewen@apache.org
>>> 
>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hi all!
>>>>>>>>>>> 
>>>>>>>>>>> As many of you know, there are a ongoing efforts to
>>>> consolidate
>>>>>> the
>>>>>>>>>>> streaming API for the next release, and then graduate it
>>>> (from
>>>>>> beta
>>>>>>>>>>> status).
>>>>>>>>>>> 
>>>>>>>>>>> In the process of this consolidation, we want to achieve
>>> the
>>>>>>>> following
>>>>>>>>>>> goals.
>>>>>>>>>>> 
>>>>>>>>>>> - Make the code more robust and simplify it in parts
>>>>>>>>>>> 
>>>>>>>>>>> - Clearly define the semantics of the constructs.
>>>>>>>>>>> 
>>>>>>>>>>> - Prepare it for support of more advanced concepts, like
>>>>>>>> partitionable
>>>>>>>>>>> state, and event time.
>>>>>>>>>>> 
>>>>>>>>>>> - Cut support for certain corner cases that were
>>> prototyped,
>>>>> but
>>>>>>>>> turned
>>>>>>>>>>> out to be not efficiently doable
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Based on prior discussions on the mailing list, Aljoscha
>>> and
>>>> me
>>>>>>>> drafted
>>>>>>>>>> the
>>>>>>>>>>> design documents below, which outline how the
>> consolidated
>>>> API
>>>>>>> would
>>>>>>>>>> like.
>>>>>>>>>>> We focused in constructs, time, and window semantics.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Design document on how to restructure the Streaming API:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
>>>>>>>>>>> 
>>>>>>>>>>> Design document on definitions of time, order, and the
>>>>> resulting
>>>>>>>>>> semantics:
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Note: The design of the interfaces and concepts for
>>> advanced
>>>>>> state
>>>>>>> in
>>>>>>>>>>> functions is not in here. That is part of a separate
>> design
>>>>>>>> discussion
>>>>>>>>>> and
>>>>>>>>>>> orthogonal to the designs drafted here.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Please have a look and voice questions and concerns.
>> Since
>>> we
>>>>>>> should
>>>>>>>>> not
>>>>>>>>>>> break the streaming API more than once, we should make
>> sure
>>>>> this
>>>>>>>>>>> consolidation brings it into the shape we want it to be
>> in.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Greetings,
>>>>>>>>>>> Stephan
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Re: Design documents for consolidated DataStream API

[Gyula Fóra <gy...@gmail.com>]

+1
On Mon, Jul 13, 2015 at 6:23 PM Stephan Ewen <se...@apache.org> wrote:

> If naming is the only concern, then we should go ahead, because we can
> change names easily (before the release).
>
> In fact, I don't think it leaves a bad impression. Global windows are
> non-parallel windows. There are also parallel windows. Pick what you need
> and what works.
>
>
> On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <gy...@gmail.com> wrote:
>
> > I think we agree on everything its more of a naming issue :)
> >
> > I thought it might be misleading that global time windows are
> > "non-parallel" windows. We dont want to give a bad impression. (Also we
> > dont want them to think that every global window is parallel but thats
> not
> > a problem here)
> >
> > Gyula
> > On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <se...@apache.org> wrote:
> >
> > > Okay, what is missing about the windowing in your opinion?
> > >
> > > The core points of the document are:
> > >
> > >   - The parallel windows are per group only.
> > >
> > >   - The implementation of the parallel windows holds window data in the
> > > group buffers.
> > >
> > >   - The global windows are non-parallel. May have parallel
> > pre-aggregation,
> > > if they are time windows.
> > >
> > >   - Time may be operator time (timer thread), or watermark time.
> > Watermark
> > > time can refer to ingress or event time.
> > >
> > >   - Windows that do not pre-aggregate may require elements in order.
> Not
> > > part of the first prototype.
> > >
> > > Do we agree on those points?
> > >
> > >
> > > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <gy...@gmail.com>
> > wrote:
> > >
> > > > In general I like it, although the main difference between the
> current
> > > and
> > > > the new one is the windowing and that is still not very clear.
> > > >
> > > > Where do we have the full stream time windows for instance?(which is
> > > > parallel but not keyed)
> > > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <
> aljoscha@apache.org>
> > > > wrote:
> > > >
> > > > > +1 I like it as well.
> > > > >
> > > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <kt...@apache.org>
> > > wrote:
> > > > >
> > > > > > +1 from my side
> > > > > >
> > > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <se...@apache.org>
> > > > wrote:
> > > > > >
> > > > > > > Do we have consensus on these designs?
> > > > > > >
> > > > > > > If we have, we should get to implementing this soon, because
> > > > basically
> > > > > > all
> > > > > > > streaming patches will have to be revisited in light of this...
> > > > > > >
> > > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <
> gyula.fora@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > You are right thats an important issue.
> > > > > > > >
> > > > > > > > And I think we should also do some renaming with the
> > "iterations"
> > > > > > because
> > > > > > > > they are not really iterations like in the batch case and it
> > > might
> > > > > > > confuse
> > > > > > > > some users.
> > > > > > > > Maybe we can call them loops or cycles and rename the api
> calls
> > > to
> > > > > make
> > > > > > > it
> > > > > > > > more intuitive what happens. It is really just a cyclic
> > dataflow.
> > > > > > > >
> > > > > > > > Aljoscha Krettek <al...@apache.org> ezt írta (időpont:
> > 2015.
> > > > júl.
> > > > > > 7.,
> > > > > > > > K,
> > > > > > > > 15:35):
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > > I just noticed that we don't have anything about how
> > iterations
> > > > and
> > > > > > > > > timestamps/watermarks should interact.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Aljoscha
> > > > > > > > >
> > > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <sewen@apache.org
> >
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all!
> > > > > > > > > >
> > > > > > > > > > As many of you know, there are a ongoing efforts to
> > > consolidate
> > > > > the
> > > > > > > > > > streaming API for the next release, and then graduate it
> > > (from
> > > > > beta
> > > > > > > > > > status).
> > > > > > > > > >
> > > > > > > > > > In the process of this consolidation, we want to achieve
> > the
> > > > > > > following
> > > > > > > > > > goals.
> > > > > > > > > >
> > > > > > > > > >  - Make the code more robust and simplify it in parts
> > > > > > > > > >
> > > > > > > > > >  - Clearly define the semantics of the constructs.
> > > > > > > > > >
> > > > > > > > > >  - Prepare it for support of more advanced concepts, like
> > > > > > > partitionable
> > > > > > > > > > state, and event time.
> > > > > > > > > >
> > > > > > > > > >  - Cut support for certain corner cases that were
> > prototyped,
> > > > but
> > > > > > > > turned
> > > > > > > > > > out to be not efficiently doable
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Based on prior discussions on the mailing list, Aljoscha
> > and
> > > me
> > > > > > > drafted
> > > > > > > > > the
> > > > > > > > > > design documents below, which outline how the
> consolidated
> > > API
> > > > > > would
> > > > > > > > > like.
> > > > > > > > > > We focused in constructs, time, and window semantics.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Design document on how to restructure the Streaming API:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > > > > > > >
> > > > > > > > > > Design document on definitions of time, order, and the
> > > > resulting
> > > > > > > > > semantics:
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Note: The design of the interfaces and concepts for
> > advanced
> > > > > state
> > > > > > in
> > > > > > > > > > functions is not in here. That is part of a separate
> design
> > > > > > > discussion
> > > > > > > > > and
> > > > > > > > > > orthogonal to the designs drafted here.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Please have a look and voice questions and concerns.
> Since
> > we
> > > > > > should
> > > > > > > > not
> > > > > > > > > > break the streaming API more than once, we should make
> sure
> > > > this
> > > > > > > > > > consolidation brings it into the shape we want it to be
> in.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Greetings,
> > > > > > > > > > Stephan
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Stephan Ewen <se...@apache.org>]

If naming is the only concern, then we should go ahead, because we can
change names easily (before the release).

In fact, I don't think it leaves a bad impression. Global windows are
non-parallel windows. There are also parallel windows. Pick what you need
and what works.


On Mon, Jul 13, 2015 at 6:13 PM, Gyula Fóra <gy...@gmail.com> wrote:

> I think we agree on everything its more of a naming issue :)
>
> I thought it might be misleading that global time windows are
> "non-parallel" windows. We dont want to give a bad impression. (Also we
> dont want them to think that every global window is parallel but thats not
> a problem here)
>
> Gyula
> On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <se...@apache.org> wrote:
>
> > Okay, what is missing about the windowing in your opinion?
> >
> > The core points of the document are:
> >
> >   - The parallel windows are per group only.
> >
> >   - The implementation of the parallel windows holds window data in the
> > group buffers.
> >
> >   - The global windows are non-parallel. May have parallel
> pre-aggregation,
> > if they are time windows.
> >
> >   - Time may be operator time (timer thread), or watermark time.
> Watermark
> > time can refer to ingress or event time.
> >
> >   - Windows that do not pre-aggregate may require elements in order. Not
> > part of the first prototype.
> >
> > Do we agree on those points?
> >
> >
> > On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <gy...@gmail.com>
> wrote:
> >
> > > In general I like it, although the main difference between the current
> > and
> > > the new one is the windowing and that is still not very clear.
> > >
> > > Where do we have the full stream time windows for instance?(which is
> > > parallel but not keyed)
> > > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <al...@apache.org>
> > > wrote:
> > >
> > > > +1 I like it as well.
> > > >
> > > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <kt...@apache.org>
> > wrote:
> > > >
> > > > > +1 from my side
> > > > >
> > > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > >
> > > > > > Do we have consensus on these designs?
> > > > > >
> > > > > > If we have, we should get to implementing this soon, because
> > > basically
> > > > > all
> > > > > > streaming patches will have to be revisited in light of this...
> > > > > >
> > > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <gyula.fora@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > You are right thats an important issue.
> > > > > > >
> > > > > > > And I think we should also do some renaming with the
> "iterations"
> > > > > because
> > > > > > > they are not really iterations like in the batch case and it
> > might
> > > > > > confuse
> > > > > > > some users.
> > > > > > > Maybe we can call them loops or cycles and rename the api calls
> > to
> > > > make
> > > > > > it
> > > > > > > more intuitive what happens. It is really just a cyclic
> dataflow.
> > > > > > >
> > > > > > > Aljoscha Krettek <al...@apache.org> ezt írta (időpont:
> 2015.
> > > júl.
> > > > > 7.,
> > > > > > > K,
> > > > > > > 15:35):
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > I just noticed that we don't have anything about how
> iterations
> > > and
> > > > > > > > timestamps/watermarks should interact.
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Aljoscha
> > > > > > > >
> > > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <se...@apache.org>
> > > wrote:
> > > > > > > >
> > > > > > > > > Hi all!
> > > > > > > > >
> > > > > > > > > As many of you know, there are a ongoing efforts to
> > consolidate
> > > > the
> > > > > > > > > streaming API for the next release, and then graduate it
> > (from
> > > > beta
> > > > > > > > > status).
> > > > > > > > >
> > > > > > > > > In the process of this consolidation, we want to achieve
> the
> > > > > > following
> > > > > > > > > goals.
> > > > > > > > >
> > > > > > > > >  - Make the code more robust and simplify it in parts
> > > > > > > > >
> > > > > > > > >  - Clearly define the semantics of the constructs.
> > > > > > > > >
> > > > > > > > >  - Prepare it for support of more advanced concepts, like
> > > > > > partitionable
> > > > > > > > > state, and event time.
> > > > > > > > >
> > > > > > > > >  - Cut support for certain corner cases that were
> prototyped,
> > > but
> > > > > > > turned
> > > > > > > > > out to be not efficiently doable
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Based on prior discussions on the mailing list, Aljoscha
> and
> > me
> > > > > > drafted
> > > > > > > > the
> > > > > > > > > design documents below, which outline how the consolidated
> > API
> > > > > would
> > > > > > > > like.
> > > > > > > > > We focused in constructs, time, and window semantics.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Design document on how to restructure the Streaming API:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > > > > > >
> > > > > > > > > Design document on definitions of time, order, and the
> > > resulting
> > > > > > > > semantics:
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Note: The design of the interfaces and concepts for
> advanced
> > > > state
> > > > > in
> > > > > > > > > functions is not in here. That is part of a separate design
> > > > > > discussion
> > > > > > > > and
> > > > > > > > > orthogonal to the designs drafted here.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Please have a look and voice questions and concerns. Since
> we
> > > > > should
> > > > > > > not
> > > > > > > > > break the streaming API more than once, we should make sure
> > > this
> > > > > > > > > consolidation brings it into the shape we want it to be in.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Greetings,
> > > > > > > > > Stephan
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Gyula Fóra <gy...@gmail.com>]

I think we agree on everything its more of a naming issue :)

I thought it might be misleading that global time windows are
"non-parallel" windows. We dont want to give a bad impression. (Also we
dont want them to think that every global window is parallel but thats not
a problem here)

Gyula
On Mon, Jul 13, 2015 at 5:22 PM Stephan Ewen <se...@apache.org> wrote:

> Okay, what is missing about the windowing in your opinion?
>
> The core points of the document are:
>
>   - The parallel windows are per group only.
>
>   - The implementation of the parallel windows holds window data in the
> group buffers.
>
>   - The global windows are non-parallel. May have parallel pre-aggregation,
> if they are time windows.
>
>   - Time may be operator time (timer thread), or watermark time. Watermark
> time can refer to ingress or event time.
>
>   - Windows that do not pre-aggregate may require elements in order. Not
> part of the first prototype.
>
> Do we agree on those points?
>
>
> On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <gy...@gmail.com> wrote:
>
> > In general I like it, although the main difference between the current
> and
> > the new one is the windowing and that is still not very clear.
> >
> > Where do we have the full stream time windows for instance?(which is
> > parallel but not keyed)
> > On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> > > +1 I like it as well.
> > >
> > > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <kt...@apache.org>
> wrote:
> > >
> > > > +1 from my side
> > > >
> > > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <se...@apache.org>
> > wrote:
> > > >
> > > > > Do we have consensus on these designs?
> > > > >
> > > > > If we have, we should get to implementing this soon, because
> > basically
> > > > all
> > > > > streaming patches will have to be revisited in light of this...
> > > > >
> > > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <gy...@gmail.com>
> > > wrote:
> > > > >
> > > > > > You are right thats an important issue.
> > > > > >
> > > > > > And I think we should also do some renaming with the "iterations"
> > > > because
> > > > > > they are not really iterations like in the batch case and it
> might
> > > > > confuse
> > > > > > some users.
> > > > > > Maybe we can call them loops or cycles and rename the api calls
> to
> > > make
> > > > > it
> > > > > > more intuitive what happens. It is really just a cyclic dataflow.
> > > > > >
> > > > > > Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2015.
> > júl.
> > > > 7.,
> > > > > > K,
> > > > > > 15:35):
> > > > > >
> > > > > > > Hi,
> > > > > > > I just noticed that we don't have anything about how iterations
> > and
> > > > > > > timestamps/watermarks should interact.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Aljoscha
> > > > > > >
> > > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <se...@apache.org>
> > wrote:
> > > > > > >
> > > > > > > > Hi all!
> > > > > > > >
> > > > > > > > As many of you know, there are a ongoing efforts to
> consolidate
> > > the
> > > > > > > > streaming API for the next release, and then graduate it
> (from
> > > beta
> > > > > > > > status).
> > > > > > > >
> > > > > > > > In the process of this consolidation, we want to achieve the
> > > > > following
> > > > > > > > goals.
> > > > > > > >
> > > > > > > >  - Make the code more robust and simplify it in parts
> > > > > > > >
> > > > > > > >  - Clearly define the semantics of the constructs.
> > > > > > > >
> > > > > > > >  - Prepare it for support of more advanced concepts, like
> > > > > partitionable
> > > > > > > > state, and event time.
> > > > > > > >
> > > > > > > >  - Cut support for certain corner cases that were prototyped,
> > but
> > > > > > turned
> > > > > > > > out to be not efficiently doable
> > > > > > > >
> > > > > > > >
> > > > > > > > Based on prior discussions on the mailing list, Aljoscha and
> me
> > > > > drafted
> > > > > > > the
> > > > > > > > design documents below, which outline how the consolidated
> API
> > > > would
> > > > > > > like.
> > > > > > > > We focused in constructs, time, and window semantics.
> > > > > > > >
> > > > > > > >
> > > > > > > > Design document on how to restructure the Streaming API:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > > > > >
> > > > > > > > Design document on definitions of time, order, and the
> > resulting
> > > > > > > semantics:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Note: The design of the interfaces and concepts for advanced
> > > state
> > > > in
> > > > > > > > functions is not in here. That is part of a separate design
> > > > > discussion
> > > > > > > and
> > > > > > > > orthogonal to the designs drafted here.
> > > > > > > >
> > > > > > > >
> > > > > > > > Please have a look and voice questions and concerns. Since we
> > > > should
> > > > > > not
> > > > > > > > break the streaming API more than once, we should make sure
> > this
> > > > > > > > consolidation brings it into the shape we want it to be in.
> > > > > > > >
> > > > > > > >
> > > > > > > > Greetings,
> > > > > > > > Stephan
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Stephan Ewen <se...@apache.org>]

Okay, what is missing about the windowing in your opinion?

The core points of the document are:

  - The parallel windows are per group only.

  - The implementation of the parallel windows holds window data in the
group buffers.

  - The global windows are non-parallel. May have parallel pre-aggregation,
if they are time windows.

  - Time may be operator time (timer thread), or watermark time. Watermark
time can refer to ingress or event time.

  - Windows that do not pre-aggregate may require elements in order. Not
part of the first prototype.

Do we agree on those points?


On Mon, Jul 13, 2015 at 4:50 PM, Gyula Fóra <gy...@gmail.com> wrote:

> In general I like it, although the main difference between the current and
> the new one is the windowing and that is still not very clear.
>
> Where do we have the full stream time windows for instance?(which is
> parallel but not keyed)
> On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > +1 I like it as well.
> >
> > On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <kt...@apache.org> wrote:
> >
> > > +1 from my side
> > >
> > > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <se...@apache.org>
> wrote:
> > >
> > > > Do we have consensus on these designs?
> > > >
> > > > If we have, we should get to implementing this soon, because
> basically
> > > all
> > > > streaming patches will have to be revisited in light of this...
> > > >
> > > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <gy...@gmail.com>
> > wrote:
> > > >
> > > > > You are right thats an important issue.
> > > > >
> > > > > And I think we should also do some renaming with the "iterations"
> > > because
> > > > > they are not really iterations like in the batch case and it might
> > > > confuse
> > > > > some users.
> > > > > Maybe we can call them loops or cycles and rename the api calls to
> > make
> > > > it
> > > > > more intuitive what happens. It is really just a cyclic dataflow.
> > > > >
> > > > > Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2015.
> júl.
> > > 7.,
> > > > > K,
> > > > > 15:35):
> > > > >
> > > > > > Hi,
> > > > > > I just noticed that we don't have anything about how iterations
> and
> > > > > > timestamps/watermarks should interact.
> > > > > >
> > > > > > Cheers,
> > > > > > Aljoscha
> > > > > >
> > > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <se...@apache.org>
> wrote:
> > > > > >
> > > > > > > Hi all!
> > > > > > >
> > > > > > > As many of you know, there are a ongoing efforts to consolidate
> > the
> > > > > > > streaming API for the next release, and then graduate it (from
> > beta
> > > > > > > status).
> > > > > > >
> > > > > > > In the process of this consolidation, we want to achieve the
> > > > following
> > > > > > > goals.
> > > > > > >
> > > > > > >  - Make the code more robust and simplify it in parts
> > > > > > >
> > > > > > >  - Clearly define the semantics of the constructs.
> > > > > > >
> > > > > > >  - Prepare it for support of more advanced concepts, like
> > > > partitionable
> > > > > > > state, and event time.
> > > > > > >
> > > > > > >  - Cut support for certain corner cases that were prototyped,
> but
> > > > > turned
> > > > > > > out to be not efficiently doable
> > > > > > >
> > > > > > >
> > > > > > > Based on prior discussions on the mailing list, Aljoscha and me
> > > > drafted
> > > > > > the
> > > > > > > design documents below, which outline how the consolidated API
> > > would
> > > > > > like.
> > > > > > > We focused in constructs, time, and window semantics.
> > > > > > >
> > > > > > >
> > > > > > > Design document on how to restructure the Streaming API:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > > > >
> > > > > > > Design document on definitions of time, order, and the
> resulting
> > > > > > semantics:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Note: The design of the interfaces and concepts for advanced
> > state
> > > in
> > > > > > > functions is not in here. That is part of a separate design
> > > > discussion
> > > > > > and
> > > > > > > orthogonal to the designs drafted here.
> > > > > > >
> > > > > > >
> > > > > > > Please have a look and voice questions and concerns. Since we
> > > should
> > > > > not
> > > > > > > break the streaming API more than once, we should make sure
> this
> > > > > > > consolidation brings it into the shape we want it to be in.
> > > > > > >
> > > > > > >
> > > > > > > Greetings,
> > > > > > > Stephan
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Gyula Fóra <gy...@gmail.com>]

In general I like it, although the main difference between the current and
the new one is the windowing and that is still not very clear.

Where do we have the full stream time windows for instance?(which is
parallel but not keyed)
On Mon, Jul 13, 2015 at 4:28 PM Aljoscha Krettek <al...@apache.org>
wrote:

> +1 I like it as well.
>
> On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <kt...@apache.org> wrote:
>
> > +1 from my side
> >
> > On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <se...@apache.org> wrote:
> >
> > > Do we have consensus on these designs?
> > >
> > > If we have, we should get to implementing this soon, because basically
> > all
> > > streaming patches will have to be revisited in light of this...
> > >
> > > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <gy...@gmail.com>
> wrote:
> > >
> > > > You are right thats an important issue.
> > > >
> > > > And I think we should also do some renaming with the "iterations"
> > because
> > > > they are not really iterations like in the batch case and it might
> > > confuse
> > > > some users.
> > > > Maybe we can call them loops or cycles and rename the api calls to
> make
> > > it
> > > > more intuitive what happens. It is really just a cyclic dataflow.
> > > >
> > > > Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2015. júl.
> > 7.,
> > > > K,
> > > > 15:35):
> > > >
> > > > > Hi,
> > > > > I just noticed that we don't have anything about how iterations and
> > > > > timestamps/watermarks should interact.
> > > > >
> > > > > Cheers,
> > > > > Aljoscha
> > > > >
> > > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <se...@apache.org> wrote:
> > > > >
> > > > > > Hi all!
> > > > > >
> > > > > > As many of you know, there are a ongoing efforts to consolidate
> the
> > > > > > streaming API for the next release, and then graduate it (from
> beta
> > > > > > status).
> > > > > >
> > > > > > In the process of this consolidation, we want to achieve the
> > > following
> > > > > > goals.
> > > > > >
> > > > > >  - Make the code more robust and simplify it in parts
> > > > > >
> > > > > >  - Clearly define the semantics of the constructs.
> > > > > >
> > > > > >  - Prepare it for support of more advanced concepts, like
> > > partitionable
> > > > > > state, and event time.
> > > > > >
> > > > > >  - Cut support for certain corner cases that were prototyped, but
> > > > turned
> > > > > > out to be not efficiently doable
> > > > > >
> > > > > >
> > > > > > Based on prior discussions on the mailing list, Aljoscha and me
> > > drafted
> > > > > the
> > > > > > design documents below, which outline how the consolidated API
> > would
> > > > > like.
> > > > > > We focused in constructs, time, and window semantics.
> > > > > >
> > > > > >
> > > > > > Design document on how to restructure the Streaming API:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > > >
> > > > > > Design document on definitions of time, order, and the resulting
> > > > > semantics:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > > >
> > > > > >
> > > > > >
> > > > > > Note: The design of the interfaces and concepts for advanced
> state
> > in
> > > > > > functions is not in here. That is part of a separate design
> > > discussion
> > > > > and
> > > > > > orthogonal to the designs drafted here.
> > > > > >
> > > > > >
> > > > > > Please have a look and voice questions and concerns. Since we
> > should
> > > > not
> > > > > > break the streaming API more than once, we should make sure this
> > > > > > consolidation brings it into the shape we want it to be in.
> > > > > >
> > > > > >
> > > > > > Greetings,
> > > > > > Stephan
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Aljoscha Krettek <al...@apache.org>]

+1 I like it as well.

On Mon, 13 Jul 2015 at 16:17 Kostas Tzoumas <kt...@apache.org> wrote:

> +1 from my side
>
> On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <se...@apache.org> wrote:
>
> > Do we have consensus on these designs?
> >
> > If we have, we should get to implementing this soon, because basically
> all
> > streaming patches will have to be revisited in light of this...
> >
> > On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <gy...@gmail.com> wrote:
> >
> > > You are right thats an important issue.
> > >
> > > And I think we should also do some renaming with the "iterations"
> because
> > > they are not really iterations like in the batch case and it might
> > confuse
> > > some users.
> > > Maybe we can call them loops or cycles and rename the api calls to make
> > it
> > > more intuitive what happens. It is really just a cyclic dataflow.
> > >
> > > Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2015. júl.
> 7.,
> > > K,
> > > 15:35):
> > >
> > > > Hi,
> > > > I just noticed that we don't have anything about how iterations and
> > > > timestamps/watermarks should interact.
> > > >
> > > > Cheers,
> > > > Aljoscha
> > > >
> > > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <se...@apache.org> wrote:
> > > >
> > > > > Hi all!
> > > > >
> > > > > As many of you know, there are a ongoing efforts to consolidate the
> > > > > streaming API for the next release, and then graduate it (from beta
> > > > > status).
> > > > >
> > > > > In the process of this consolidation, we want to achieve the
> > following
> > > > > goals.
> > > > >
> > > > >  - Make the code more robust and simplify it in parts
> > > > >
> > > > >  - Clearly define the semantics of the constructs.
> > > > >
> > > > >  - Prepare it for support of more advanced concepts, like
> > partitionable
> > > > > state, and event time.
> > > > >
> > > > >  - Cut support for certain corner cases that were prototyped, but
> > > turned
> > > > > out to be not efficiently doable
> > > > >
> > > > >
> > > > > Based on prior discussions on the mailing list, Aljoscha and me
> > drafted
> > > > the
> > > > > design documents below, which outline how the consolidated API
> would
> > > > like.
> > > > > We focused in constructs, time, and window semantics.
> > > > >
> > > > >
> > > > > Design document on how to restructure the Streaming API:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > > >
> > > > > Design document on definitions of time, order, and the resulting
> > > > semantics:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > > >
> > > > >
> > > > >
> > > > > Note: The design of the interfaces and concepts for advanced state
> in
> > > > > functions is not in here. That is part of a separate design
> > discussion
> > > > and
> > > > > orthogonal to the designs drafted here.
> > > > >
> > > > >
> > > > > Please have a look and voice questions and concerns. Since we
> should
> > > not
> > > > > break the streaming API more than once, we should make sure this
> > > > > consolidation brings it into the shape we want it to be in.
> > > > >
> > > > >
> > > > > Greetings,
> > > > > Stephan
> > > > >
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Kostas Tzoumas <kt...@apache.org>]

+1 from my side

On Mon, Jul 13, 2015 at 4:15 PM, Stephan Ewen <se...@apache.org> wrote:

> Do we have consensus on these designs?
>
> If we have, we should get to implementing this soon, because basically all
> streaming patches will have to be revisited in light of this...
>
> On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <gy...@gmail.com> wrote:
>
> > You are right thats an important issue.
> >
> > And I think we should also do some renaming with the "iterations" because
> > they are not really iterations like in the batch case and it might
> confuse
> > some users.
> > Maybe we can call them loops or cycles and rename the api calls to make
> it
> > more intuitive what happens. It is really just a cyclic dataflow.
> >
> > Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2015. júl. 7.,
> > K,
> > 15:35):
> >
> > > Hi,
> > > I just noticed that we don't have anything about how iterations and
> > > timestamps/watermarks should interact.
> > >
> > > Cheers,
> > > Aljoscha
> > >
> > > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <se...@apache.org> wrote:
> > >
> > > > Hi all!
> > > >
> > > > As many of you know, there are a ongoing efforts to consolidate the
> > > > streaming API for the next release, and then graduate it (from beta
> > > > status).
> > > >
> > > > In the process of this consolidation, we want to achieve the
> following
> > > > goals.
> > > >
> > > >  - Make the code more robust and simplify it in parts
> > > >
> > > >  - Clearly define the semantics of the constructs.
> > > >
> > > >  - Prepare it for support of more advanced concepts, like
> partitionable
> > > > state, and event time.
> > > >
> > > >  - Cut support for certain corner cases that were prototyped, but
> > turned
> > > > out to be not efficiently doable
> > > >
> > > >
> > > > Based on prior discussions on the mailing list, Aljoscha and me
> drafted
> > > the
> > > > design documents below, which outline how the consolidated API would
> > > like.
> > > > We focused in constructs, time, and window semantics.
> > > >
> > > >
> > > > Design document on how to restructure the Streaming API:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > > >
> > > > Design document on definitions of time, order, and the resulting
> > > semantics:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > > >
> > > >
> > > >
> > > > Note: The design of the interfaces and concepts for advanced state in
> > > > functions is not in here. That is part of a separate design
> discussion
> > > and
> > > > orthogonal to the designs drafted here.
> > > >
> > > >
> > > > Please have a look and voice questions and concerns. Since we should
> > not
> > > > break the streaming API more than once, we should make sure this
> > > > consolidation brings it into the shape we want it to be in.
> > > >
> > > >
> > > > Greetings,
> > > > Stephan
> > > >
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Stephan Ewen <se...@apache.org>]

Do we have consensus on these designs?

If we have, we should get to implementing this soon, because basically all
streaming patches will have to be revisited in light of this...

On Tue, Jul 7, 2015 at 3:41 PM, Gyula Fóra <gy...@gmail.com> wrote:

> You are right thats an important issue.
>
> And I think we should also do some renaming with the "iterations" because
> they are not really iterations like in the batch case and it might confuse
> some users.
> Maybe we can call them loops or cycles and rename the api calls to make it
> more intuitive what happens. It is really just a cyclic dataflow.
>
> Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2015. júl. 7.,
> K,
> 15:35):
>
> > Hi,
> > I just noticed that we don't have anything about how iterations and
> > timestamps/watermarks should interact.
> >
> > Cheers,
> > Aljoscha
> >
> > On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <se...@apache.org> wrote:
> >
> > > Hi all!
> > >
> > > As many of you know, there are a ongoing efforts to consolidate the
> > > streaming API for the next release, and then graduate it (from beta
> > > status).
> > >
> > > In the process of this consolidation, we want to achieve the following
> > > goals.
> > >
> > >  - Make the code more robust and simplify it in parts
> > >
> > >  - Clearly define the semantics of the constructs.
> > >
> > >  - Prepare it for support of more advanced concepts, like partitionable
> > > state, and event time.
> > >
> > >  - Cut support for certain corner cases that were prototyped, but
> turned
> > > out to be not efficiently doable
> > >
> > >
> > > Based on prior discussions on the mailing list, Aljoscha and me drafted
> > the
> > > design documents below, which outline how the consolidated API would
> > like.
> > > We focused in constructs, time, and window semantics.
> > >
> > >
> > > Design document on how to restructure the Streaming API:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> > >
> > > Design document on definitions of time, order, and the resulting
> > semantics:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> > >
> > >
> > >
> > > Note: The design of the interfaces and concepts for advanced state in
> > > functions is not in here. That is part of a separate design discussion
> > and
> > > orthogonal to the designs drafted here.
> > >
> > >
> > > Please have a look and voice questions and concerns. Since we should
> not
> > > break the streaming API more than once, we should make sure this
> > > consolidation brings it into the shape we want it to be in.
> > >
> > >
> > > Greetings,
> > > Stephan
> > >
> >
>

Re: Design documents for consolidated DataStream API

[Gyula Fóra <gy...@gmail.com>]

You are right thats an important issue.

And I think we should also do some renaming with the "iterations" because
they are not really iterations like in the batch case and it might confuse
some users.
Maybe we can call them loops or cycles and rename the api calls to make it
more intuitive what happens. It is really just a cyclic dataflow.

Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2015. júl. 7., K,
15:35):

> Hi,
> I just noticed that we don't have anything about how iterations and
> timestamps/watermarks should interact.
>
> Cheers,
> Aljoscha
>
> On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <se...@apache.org> wrote:
>
> > Hi all!
> >
> > As many of you know, there are a ongoing efforts to consolidate the
> > streaming API for the next release, and then graduate it (from beta
> > status).
> >
> > In the process of this consolidation, we want to achieve the following
> > goals.
> >
> >  - Make the code more robust and simplify it in parts
> >
> >  - Clearly define the semantics of the constructs.
> >
> >  - Prepare it for support of more advanced concepts, like partitionable
> > state, and event time.
> >
> >  - Cut support for certain corner cases that were prototyped, but turned
> > out to be not efficiently doable
> >
> >
> > Based on prior discussions on the mailing list, Aljoscha and me drafted
> the
> > design documents below, which outline how the consolidated API would
> like.
> > We focused in constructs, time, and window semantics.
> >
> >
> > Design document on how to restructure the Streaming API:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
> >
> > Design document on definitions of time, order, and the resulting
> semantics:
> >
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
> >
> >
> >
> > Note: The design of the interfaces and concepts for advanced state in
> > functions is not in here. That is part of a separate design discussion
> and
> > orthogonal to the designs drafted here.
> >
> >
> > Please have a look and voice questions and concerns. Since we should not
> > break the streaming API more than once, we should make sure this
> > consolidation brings it into the shape we want it to be in.
> >
> >
> > Greetings,
> > Stephan
> >
>

Re: Design documents for consolidated DataStream API

[Aljoscha Krettek <al...@apache.org>]

Hi,
I just noticed that we don't have anything about how iterations and
timestamps/watermarks should interact.

Cheers,
Aljoscha

On Mon, 6 Jul 2015 at 23:56 Stephan Ewen <se...@apache.org> wrote:

> Hi all!
>
> As many of you know, there are a ongoing efforts to consolidate the
> streaming API for the next release, and then graduate it (from beta
> status).
>
> In the process of this consolidation, we want to achieve the following
> goals.
>
>  - Make the code more robust and simplify it in parts
>
>  - Clearly define the semantics of the constructs.
>
>  - Prepare it for support of more advanced concepts, like partitionable
> state, and event time.
>
>  - Cut support for certain corner cases that were prototyped, but turned
> out to be not efficiently doable
>
>
> Based on prior discussions on the mailing list, Aljoscha and me drafted the
> design documents below, which outline how the consolidated API would like.
> We focused in constructs, time, and window semantics.
>
>
> Design document on how to restructure the Streaming API:
>
> https://cwiki.apache.org/confluence/display/FLINK/Streams+and+Operations+on+Streams
>
> Design document on definitions of time, order, and the resulting semantics:
> https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
>
>
>
> Note: The design of the interfaces and concepts for advanced state in
> functions is not in here. That is part of a separate design discussion and
> orthogonal to the designs drafted here.
>
>
> Please have a look and voice questions and concerns. Since we should not
> break the streaming API more than once, we should make sure this
> consolidation brings it into the shape we want it to be in.
>
>
> Greetings,
> Stephan
>