You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Gyula Fóra <gy...@apache.org> on 2015/07/23 15:55:36 UTC
Guide/design doc for streaming operator states
Hey!
I started putting together a guide/design document for the streaming
operator state interfaces and implementations. The idea would be to create
a doc that contains all the details about the implementations so anyone can
use it as a reference later.
https://cwiki.apache.org/confluence/display/FLINK/Stateful+Stream+Processing
It will probably take me a couple of days to finish it, but in any case
feel free to comment.
Cheers,
Gyula
Re: Guide/design doc for streaming operator states
Posted by Aljoscha Krettek <al...@apache.org>.
Ah ok. I think that keyBy() can normally not be chained because we don't
know how the fields in the emitted object change.
On Thu, 30 Jul 2015 at 13:40 Gyula Fóra <gy...@gmail.com> wrote:
> Thanks for the feedback :)
>
> My idea when I wrote that was that you can chain keyBy statements to
> maintain order if your key does not change. Otherwise you are right, we
> need a sorting operator.
>
> Gyula
>
> Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2015. júl. 30.,
> Cs, 13:18):
>
> > Hi,
> > sorry for the long wait but I finally found the time to read it. It looks
> > good but the later parts of course still need to be fleshed out.
> >
> > I have one comments/questions:
> > In the description of partitioned state you have this sentence:
> "Operations
> > using partitioned state can also benefit from the partial ordering
> > guarantees that the flink runtime provides, to implement deterministic
> > behaviour." How do we provide the ordering guarantees. I would assume
> that
> > after a keyBy() the ordering in each partition is arbitrary, unless we
> add
> > some sorting operator.
> >
> > Cheers,
> > Aljoscha
> >
> > On Thu, 23 Jul 2015 at 15:55 Gyula Fóra <gy...@apache.org> wrote:
> >
> > > Hey!
> > > I started putting together a guide/design document for the streaming
> > > operator state interfaces and implementations. The idea would be to
> > create
> > > a doc that contains all the details about the implementations so anyone
> > can
> > > use it as a reference later.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Stateful+Stream+Processing
> > >
> > > It will probably take me a couple of days to finish it, but in any case
> > > feel free to comment.
> > >
> > > Cheers,
> > > Gyula
> > >
> >
>
Re: Guide/design doc for streaming operator states
Posted by Gyula Fóra <gy...@gmail.com>.
Thanks for the feedback :)
My idea when I wrote that was that you can chain keyBy statements to
maintain order if your key does not change. Otherwise you are right, we
need a sorting operator.
Gyula
Aljoscha Krettek <al...@apache.org> ezt írta (időpont: 2015. júl. 30.,
Cs, 13:18):
> Hi,
> sorry for the long wait but I finally found the time to read it. It looks
> good but the later parts of course still need to be fleshed out.
>
> I have one comments/questions:
> In the description of partitioned state you have this sentence: "Operations
> using partitioned state can also benefit from the partial ordering
> guarantees that the flink runtime provides, to implement deterministic
> behaviour." How do we provide the ordering guarantees. I would assume that
> after a keyBy() the ordering in each partition is arbitrary, unless we add
> some sorting operator.
>
> Cheers,
> Aljoscha
>
> On Thu, 23 Jul 2015 at 15:55 Gyula Fóra <gy...@apache.org> wrote:
>
> > Hey!
> > I started putting together a guide/design document for the streaming
> > operator state interfaces and implementations. The idea would be to
> create
> > a doc that contains all the details about the implementations so anyone
> can
> > use it as a reference later.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Stateful+Stream+Processing
> >
> > It will probably take me a couple of days to finish it, but in any case
> > feel free to comment.
> >
> > Cheers,
> > Gyula
> >
>
Re: Guide/design doc for streaming operator states
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
sorry for the long wait but I finally found the time to read it. It looks
good but the later parts of course still need to be fleshed out.
I have one comments/questions:
In the description of partitioned state you have this sentence: "Operations
using partitioned state can also benefit from the partial ordering
guarantees that the flink runtime provides, to implement deterministic
behaviour." How do we provide the ordering guarantees. I would assume that
after a keyBy() the ordering in each partition is arbitrary, unless we add
some sorting operator.
Cheers,
Aljoscha
On Thu, 23 Jul 2015 at 15:55 Gyula Fóra <gy...@apache.org> wrote:
> Hey!
> I started putting together a guide/design document for the streaming
> operator state interfaces and implementations. The idea would be to create
> a doc that contains all the details about the implementations so anyone can
> use it as a reference later.
>
>
> https://cwiki.apache.org/confluence/display/FLINK/Stateful+Stream+Processing
>
> It will probably take me a couple of days to finish it, but in any case
> feel free to comment.
>
> Cheers,
> Gyula
>