You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Shen Li <cs...@gmail.com> on 2016/06/24 17:56:42 UTC

What is the "Keyed State" in the capability matrix?

Hi,

There is a "Keyed State" row in the  "What is being computed" section of
the capability matrix. What does the "Keyed State" refer to? Is it a global
key-value store?

(
http://beam.incubator.apache.org/beam/capability/2016/03/17/capability-matrix.html
)

Thanks,

Shen

Re: What is the "Keyed State" in the capability matrix?

Posted by Shen Li <cs...@gmail.com>.

Hi Kenn,

Thanks for replying. I look forward to the technical docs.

Shen

On Fri, Jun 24, 2016 at 2:17 PM, Kenneth Knowles <kl...@google.com.invalid>
wrote:

> Hi Shen,
>
> The row refers to the ability for a DoFn in a ParDo to access per-key (and
> window) state cells that persist beyond the lifetime of an element or
> bundle. This is a feature that was in the later stages of design when the
> Beam code was donated. Hence it a row in the graph, but even the Beam Model
> column says "no", pending a public design proposal & consensus. Most
> runners already have a similar capability at a low level; this feature
> refers to exposing it in a nice way for users.
>
> I have a design doc that I'm busily revising to make sense for the whole
> community. I will send the doc to this list and add it to our technical
> docs folder as soon as I can get it ready. You can follow BEAM-25 [1] if
> you like, too.
>
> Kenn
>
> [1] https://issues.apache.org/jira/browse/BEAM-25
>
>
> On Fri, Jun 24, 2016 at 10:56 AM, Shen Li <cs...@gmail.com> wrote:
>
> > Hi,
> >
> > There is a "Keyed State" row in the  "What is being computed" section of
> > the capability matrix. What does the "Keyed State" refer to? Is it a
> global
> > key-value store?
> >
> > (
> >
> >
> http://beam.incubator.apache.org/beam/capability/2016/03/17/capability-matrix.html
> > )
> >
> > Thanks,
> >
> > Shen
> >
>

Re: What is the "Keyed State" in the capability matrix?

Posted by Kenneth Knowles <kl...@google.com.INVALID>.

Hi Shen,

The row refers to the ability for a DoFn in a ParDo to access per-key (and
window) state cells that persist beyond the lifetime of an element or
bundle. This is a feature that was in the later stages of design when the
Beam code was donated. Hence it a row in the graph, but even the Beam Model
column says "no", pending a public design proposal & consensus. Most
runners already have a similar capability at a low level; this feature
refers to exposing it in a nice way for users.

I have a design doc that I'm busily revising to make sense for the whole
community. I will send the doc to this list and add it to our technical
docs folder as soon as I can get it ready. You can follow BEAM-25 [1] if
you like, too.

Kenn

[1] https://issues.apache.org/jira/browse/BEAM-25

On Fri, Jun 24, 2016 at 10:56 AM, Shen Li <cs...@gmail.com> wrote:

> Hi,
>
> There is a "Keyed State" row in the  "What is being computed" section of
> the capability matrix. What does the "Keyed State" refer to? Is it a global
> key-value store?
>
> (
>
> http://beam.incubator.apache.org/beam/capability/2016/03/17/capability-matrix.html
> )
>
> Thanks,
>
> Shen
>