You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Kenneth Knowles <kl...@google.com.INVALID> on 2017/03/21 20:15:32 UTC

[PROPOSAL] "Requires deterministic input"

Problem:

I will drop all nuance and say that the `Write` transform as it exists in
the SDK is incorrect until we add some specification and APIs. We can't
keep shipping an SDK with an unsafe transform in it, and IMO this certainly
blocks a stable release.

Specifically, there is pseudorandom data generated and once it has been
observed and used to produce a side effect, it cannot be regenerated
without erroneous results.

This generalizes: For some side-effecting user-defined functions, it is
vital that even across retries/replays they have a consistent view of the
contents of their input PCollection, because their effect on the outside
world cannot be retracted if/when they fail and are retried. Once the
runner ensures a consistent view of the input, it is then their own
responsibility to be idempotent.

Ideally we should specify this requirement for the user-defined function
without imposing any particular implementation strategy on Beam runners.

Proposal:

1. Let a DoFn declare (mechanism not important right now) that it "requires
deterministic input".

2. Each runner will need a way to induce deterministic input - the obvious
choice being a materialization.

I want to keep the discussion focused, so I'm leaving out any possibilities
of taking this further.

Regarding performance: Today places that require this tend to be already
paying the cost via GroupByKey / Reshuffle operations, since that was a
simple way to induce determinism in batch Dataflow* (doesn't work for most
other runners nor for streaming Dataflow). This change will replace a
hard-coded implementation strategy with a requirement that may be fulfilled
in the most efficient way available.

Thoughts?

Kenn (w/ lots of consult from colleagues, especially Ben)

* There is some overlap with the reshuffle/redistribute discussion because
of this historical situation, but I would like to leave that broader
discussion out of this correctness issue.

Re: [PROPOSAL] "Requires deterministic input"

Posted by Thomas Groh <tg...@google.com.INVALID>.
As I said, a minor concern; we should be explicit in our documentation that
it is only the input _elements_ that are
deterministic/stable/replayable/etc, and not operational concerns
surrounding them (such as bundling). I'd generally avoid making the actual
annotation more verbose.

On Wed, Aug 9, 2017 at 1:49 PM, Kenneth Knowles <kl...@google.com.invalid>
wrote:

> On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh <tg...@google.com.invalid>
> wrote:
>
> > I have a minor concern that this may not work as expected for users that
> > try to batch remote calls in `FinishBundle` - we should make sure we
> > document that it is explicitly the input elements that will be replayed,
> > and bundles and other operational are still arbitrary.
> >
>
> I think it is safe since writing has to be idempotent per-element anyhow,
> and a DoFn must be discarded on failure and a new version used for any
> retries.
>
> Was your concern something else? Were you concerned that the phrasing may
> imply a deterministic ordering of calls to the DoFn's methods? Perhaps
> @RequiresStableInputContents to make clear it has nothing to do with
> replaying method calls?
>
> Kenn
>
>
> >
> >
> > On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax <re...@google.com.invalid>
> > wrote:
> >
> > > I think deterministic here means deterministically replayable. i.e. no
> > > matter how many times the element is retried, it will always be the
> same.
> > >
> > > I think we should also allow specifying this on processTimer. This
> would
> > > mean that any keyed state written in a previous processElement must be
> > > guaranteed durable before processTimer is called.
> > >
> > >
> > > On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <bc...@apache.org>
> > > wrote:
> > >
> > > > I strongly agree with this proposal. I think moving away from "just
> > > insert
> > > > a GroupByKey for one of the 3 different reasons you may want it"
> > towards
> > > > APIs that allow code to express the requirements they have and the
> > runner
> > > > to choose the best way to meet this is a major step forwards in terms
> > of
> > > > portability.
> > > >
> > > > I think "deterministic" may be misleading. The actual contents of the
> > > > collection aren't deterministic if upstream computations aren't. The
> > > > property we really need is that once an input may have been observed
> by
> > > the
> > > > side-effecting code it will never be observed with a different value.
> > > >
> > > > I would propose something RequiresStableInput, to indicate that the
> > input
> > > > must be stable as observed by the function. I could also see
> something
> > > > hinting at the fact we don't recompute the input, such as
> > > > RequiresMemoizedInput or RequiresNoRecomputation.
> > > >
> > > > -- Ben
> > > >
> > > > P.S For anyone interested other uses of GroupByKey that we may want
> to
> > > > discuss APIs for would be preventing retry across steps (eg.,
> > preventing
> > > > fusion) and redistributing inputs across workers.
> > > >
> > > > On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles
> <klk@google.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > This came up again, so I wanted to push it along by proposing a
> > > specific
> > > > > API for Java that could have a derived API in Python. I am writing
> > this
> > > > > quickly to get something out there, so I welcome suggestions for
> > > > revision.
> > > > >
> > > > > Today a DoFn has a @ProcessElement annotated method with various
> > > > automated
> > > > > parameters, but most fundamentally this:
> > > > >
> > > > > @ProcessElement
> > > > > public void process(ProcessContext ctx) {
> > > > >   ctx.element() // to access the current input element
> > > > >   ctx.output(something) // to write to default output collection
> > > > >   ctx.output(tag, something) // to write to other output
> collections
> > > > > }
> > > > >
> > > > > For some time, we have hoped to unpack the context - it is a
> > > > > backwards-compatibility pattern made obsolete by the new DoFn
> design.
> > > So
> > > > > instead it would look like this:
> > > > >
> > > > > @ProcessElement
> > > > > public void process(Element element, MainOutput mainOutput, ...) {
> > > > >   element.get() // to access the current input element
> > > > >   mainOutput.output(something) // to write to the default output
> > > > collection
> > > > >   other.output(something) // to write to other output collection
> > > > > }
> > > > >
> > > > > I've deliberately left out the undecided syntax for side outputs.
> But
> > > it
> > > > > would be nice for the tag to be built in to the parameter so it
> > doesn't
> > > > > have to be used when calling output().
> > > > >
> > > > > One way to enhance this to deterministic input would just be this:
> > > > >
> > > > > @ProcessElement
> > > > > @RequiresDeterministicInput
> > > > > public void process(Element element, MainOutput mainOutput, ...) {
> > > > >   element.get() // to access the current input element
> > > > >   mainOutput.output(something) // to write to the default output
> > > > collection
> > > > >   other.output(something) // to write to other output collection
> > > > > }
> > > > >
> > > > > There are really a lot of places where we could put an annotation
> or
> > > > change
> > > > > a type to indicate that the input PCollection should be
> > > > > well-defined/deterministically-replayable. I don't have a really
> > > strong
> > > > > opinion.
> > > > >
> > > > > Kenn
> > > > >
> > > > > On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > > <bchambers@google.com.invalid
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Allowing an annotation on DoFn's that produce deterministic
> output
> > > > could
> > > > > be
> > > > > > added in the future, but doesn't seem like a great option.
> > > > > >
> > > > > > 1. It is a correctness issue to assume a DoFn is deterministic
> and
> > be
> > > > > > wrong, so we would need to assume all transform outputs are
> > > > > > non-deterministic unless annotated. Getting this correct is
> > difficult
> > > > > (for
> > > > > > example, GBK is surprisingly non-deterministic except in specific
> > > > cases).
> > > > > >
> > > > > > 2. It is unlikely to be a major performance improvement, given
> that
> > > any
> > > > > > non-deterministic transform prior to a sink (which are most
> likely
> > to
> > > > > > require deterministic input) will cause additional work to be
> > needed.
> > > > > >
> > > > > > Based on this, it seems like the risk of allowing an annotation
> is
> > > high
> > > > > > while the potential for performance improvements is low. The
> > current
> > > > > > proposal (not allowing an annotation) makes sense for now, until
> we
> > > can
> > > > > > demonstrate that the impact on performance is high in cases that
> > > could
> > > > be
> > > > > > avoided with an annotation (in real-world use).
> > > > > >
> > > > > > -- Ben
> > > > > >
> > > > > > On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com>
> > wrote:
> > > > > >
> > > > > > +1 for the general idea of runners handling it over hard-coded
> > > > > > implementation strategy.
> > > > > >
> > > > > > For the Write transform I believe you are talking about
> > > > ApplyShardingKey
> > > > > > <
> > > > > > https://github.com/apache/beam/blob/
> d66029cafde152c0a46ebd276ddfa4
> > > > > > c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> > > > > > io/Write.java#L304
> > > > > > >
> > > > > > which
> > > > > > introduces non deterministic behavior when retried?
> > > > > >
> > > > > >
> > > > > > *Let a DoFn declare (mechanism not important right now) that it
> > > > > > "requiresdeterministic input"*
> > > > > >
> > > > > >
> > > > > >
> > > > > > *Each runner will need a way to induce deterministic input - the
> > > > > > obviouschoice being a materialization.*
> > > > > >
> > > > > > Does this mean that a runner will always materialize (or whatever
> > the
> > > > > > strategy is) an input PCollection to this DoFn even though the
> > > > > PCollection
> > > > > > might have been produced by deterministic transforms? Would it
> make
> > > > sense
> > > > > > to also let DoFns declare if they produce non-deterministic
> output?
> > > > > >
> > > > > > -Vikas
> > > > > >
> > > > > >
> > > > > > On 21 March 2017 at 13:52, Stephen Sisk <sisk@google.com.invalid
> >
> > > > wrote:
> > > > > >
> > > > > > > Hey Kenn-
> > > > > > >
> > > > > > > this seems important, but I don't have all the context on what
> > the
> > > > > > problem
> > > > > > > is.
> > > > > > >
> > > > > > > Can you explain this sentence "Specifically, there is
> > pseudorandom
> > > > data
> > > > > > > generated and once it has been observed and used to produce a
> > side
> > > > > > effect,
> > > > > > > it cannot be regenerated without erroneous results." ?
> > > > > > >
> > > > > > > Where is the pseudorandom data coming from? Perhaps a concrete
> > > > example
> > > > > > > would help?
> > > > > > >
> > > > > > > S
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > > <klk@google.com.invalid
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Problem:
> > > > > > > >
> > > > > > > > I will drop all nuance and say that the `Write` transform as
> it
> > > > > exists
> > > > > > in
> > > > > > > > the SDK is incorrect until we add some specification and
> APIs.
> > We
> > > > > can't
> > > > > > > > keep shipping an SDK with an unsafe transform in it, and IMO
> > this
> > > > > > > certainly
> > > > > > > > blocks a stable release.
> > > > > > > >
> > > > > > > > Specifically, there is pseudorandom data generated and once
> it
> > > has
> > > > > been
> > > > > > > > observed and used to produce a side effect, it cannot be
> > > > regenerated
> > > > > > > > without erroneous results.
> > > > > > > >
> > > > > > > > This generalizes: For some side-effecting user-defined
> > functions,
> > > > it
> > > > > is
> > > > > > > > vital that even across retries/replays they have a consistent
> > > view
> > > > of
> > > > > > the
> > > > > > > > contents of their input PCollection, because their effect on
> > the
> > > > > > outside
> > > > > > > > world cannot be retracted if/when they fail and are retried.
> > Once
> > > > the
> > > > > > > > runner ensures a consistent view of the input, it is then
> their
> > > own
> > > > > > > > responsibility to be idempotent.
> > > > > > > >
> > > > > > > > Ideally we should specify this requirement for the
> user-defined
> > > > > > function
> > > > > > > > without imposing any particular implementation strategy on
> Beam
> > > > > > runners.
> > > > > > > >
> > > > > > > > Proposal:
> > > > > > > >
> > > > > > > > 1. Let a DoFn declare (mechanism not important right now)
> that
> > it
> > > > > > > "requires
> > > > > > > > deterministic input".
> > > > > > > >
> > > > > > > > 2. Each runner will need a way to induce deterministic input
> -
> > > the
> > > > > > > obvious
> > > > > > > > choice being a materialization.
> > > > > > > >
> > > > > > > > I want to keep the discussion focused, so I'm leaving out any
> > > > > > > possibilities
> > > > > > > > of taking this further.
> > > > > > > >
> > > > > > > > Regarding performance: Today places that require this tend to
> > be
> > > > > > already
> > > > > > > > paying the cost via GroupByKey / Reshuffle operations, since
> > that
> > > > > was a
> > > > > > > > simple way to induce determinism in batch Dataflow* (doesn't
> > work
> > > > for
> > > > > > > most
> > > > > > > > other runners nor for streaming Dataflow). This change will
> > > > replace a
> > > > > > > > hard-coded implementation strategy with a requirement that
> may
> > be
> > > > > > > fulfilled
> > > > > > > > in the most efficient way available.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Kenn (w/ lots of consult from colleagues, especially Ben)
> > > > > > > >
> > > > > > > > * There is some overlap with the reshuffle/redistribute
> > > discussion
> > > > > > > because
> > > > > > > > of this historical situation, but I would like to leave that
> > > > broader
> > > > > > > > discussion out of this correctness issue.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Kenneth Knowles <kl...@google.com.INVALID>.
On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh <tg...@google.com.invalid>
wrote:

> I have a minor concern that this may not work as expected for users that
> try to batch remote calls in `FinishBundle` - we should make sure we
> document that it is explicitly the input elements that will be replayed,
> and bundles and other operational are still arbitrary.
>

I think it is safe since writing has to be idempotent per-element anyhow,
and a DoFn must be discarded on failure and a new version used for any
retries.

Was your concern something else? Were you concerned that the phrasing may
imply a deterministic ordering of calls to the DoFn's methods? Perhaps
@RequiresStableInputContents to make clear it has nothing to do with
replaying method calls?

Kenn


>
>
> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax <re...@google.com.invalid>
> wrote:
>
> > I think deterministic here means deterministically replayable. i.e. no
> > matter how many times the element is retried, it will always be the same.
> >
> > I think we should also allow specifying this on processTimer. This would
> > mean that any keyed state written in a previous processElement must be
> > guaranteed durable before processTimer is called.
> >
> >
> > On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <bc...@apache.org>
> > wrote:
> >
> > > I strongly agree with this proposal. I think moving away from "just
> > insert
> > > a GroupByKey for one of the 3 different reasons you may want it"
> towards
> > > APIs that allow code to express the requirements they have and the
> runner
> > > to choose the best way to meet this is a major step forwards in terms
> of
> > > portability.
> > >
> > > I think "deterministic" may be misleading. The actual contents of the
> > > collection aren't deterministic if upstream computations aren't. The
> > > property we really need is that once an input may have been observed by
> > the
> > > side-effecting code it will never be observed with a different value.
> > >
> > > I would propose something RequiresStableInput, to indicate that the
> input
> > > must be stable as observed by the function. I could also see something
> > > hinting at the fact we don't recompute the input, such as
> > > RequiresMemoizedInput or RequiresNoRecomputation.
> > >
> > > -- Ben
> > >
> > > P.S For anyone interested other uses of GroupByKey that we may want to
> > > discuss APIs for would be preventing retry across steps (eg.,
> preventing
> > > fusion) and redistributing inputs across workers.
> > >
> > > On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles <klk@google.com.invalid
> >
> > > wrote:
> > >
> > > > This came up again, so I wanted to push it along by proposing a
> > specific
> > > > API for Java that could have a derived API in Python. I am writing
> this
> > > > quickly to get something out there, so I welcome suggestions for
> > > revision.
> > > >
> > > > Today a DoFn has a @ProcessElement annotated method with various
> > > automated
> > > > parameters, but most fundamentally this:
> > > >
> > > > @ProcessElement
> > > > public void process(ProcessContext ctx) {
> > > >   ctx.element() // to access the current input element
> > > >   ctx.output(something) // to write to default output collection
> > > >   ctx.output(tag, something) // to write to other output collections
> > > > }
> > > >
> > > > For some time, we have hoped to unpack the context - it is a
> > > > backwards-compatibility pattern made obsolete by the new DoFn design.
> > So
> > > > instead it would look like this:
> > > >
> > > > @ProcessElement
> > > > public void process(Element element, MainOutput mainOutput, ...) {
> > > >   element.get() // to access the current input element
> > > >   mainOutput.output(something) // to write to the default output
> > > collection
> > > >   other.output(something) // to write to other output collection
> > > > }
> > > >
> > > > I've deliberately left out the undecided syntax for side outputs. But
> > it
> > > > would be nice for the tag to be built in to the parameter so it
> doesn't
> > > > have to be used when calling output().
> > > >
> > > > One way to enhance this to deterministic input would just be this:
> > > >
> > > > @ProcessElement
> > > > @RequiresDeterministicInput
> > > > public void process(Element element, MainOutput mainOutput, ...) {
> > > >   element.get() // to access the current input element
> > > >   mainOutput.output(something) // to write to the default output
> > > collection
> > > >   other.output(something) // to write to other output collection
> > > > }
> > > >
> > > > There are really a lot of places where we could put an annotation or
> > > change
> > > > a type to indicate that the input PCollection should be
> > > > well-defined/deterministically-replayable. I don't have a really
> > strong
> > > > opinion.
> > > >
> > > > Kenn
> > > >
> > > > On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > <bchambers@google.com.invalid
> > > > >
> > > > wrote:
> > > >
> > > > > Allowing an annotation on DoFn's that produce deterministic output
> > > could
> > > > be
> > > > > added in the future, but doesn't seem like a great option.
> > > > >
> > > > > 1. It is a correctness issue to assume a DoFn is deterministic and
> be
> > > > > wrong, so we would need to assume all transform outputs are
> > > > > non-deterministic unless annotated. Getting this correct is
> difficult
> > > > (for
> > > > > example, GBK is surprisingly non-deterministic except in specific
> > > cases).
> > > > >
> > > > > 2. It is unlikely to be a major performance improvement, given that
> > any
> > > > > non-deterministic transform prior to a sink (which are most likely
> to
> > > > > require deterministic input) will cause additional work to be
> needed.
> > > > >
> > > > > Based on this, it seems like the risk of allowing an annotation is
> > high
> > > > > while the potential for performance improvements is low. The
> current
> > > > > proposal (not allowing an annotation) makes sense for now, until we
> > can
> > > > > demonstrate that the impact on performance is high in cases that
> > could
> > > be
> > > > > avoided with an annotation (in real-world use).
> > > > >
> > > > > -- Ben
> > > > >
> > > > > On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com>
> wrote:
> > > > >
> > > > > +1 for the general idea of runners handling it over hard-coded
> > > > > implementation strategy.
> > > > >
> > > > > For the Write transform I believe you are talking about
> > > ApplyShardingKey
> > > > > <
> > > > > https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > > > > c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> > > > > io/Write.java#L304
> > > > > >
> > > > > which
> > > > > introduces non deterministic behavior when retried?
> > > > >
> > > > >
> > > > > *Let a DoFn declare (mechanism not important right now) that it
> > > > > "requiresdeterministic input"*
> > > > >
> > > > >
> > > > >
> > > > > *Each runner will need a way to induce deterministic input - the
> > > > > obviouschoice being a materialization.*
> > > > >
> > > > > Does this mean that a runner will always materialize (or whatever
> the
> > > > > strategy is) an input PCollection to this DoFn even though the
> > > > PCollection
> > > > > might have been produced by deterministic transforms? Would it make
> > > sense
> > > > > to also let DoFns declare if they produce non-deterministic output?
> > > > >
> > > > > -Vikas
> > > > >
> > > > >
> > > > > On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid>
> > > wrote:
> > > > >
> > > > > > Hey Kenn-
> > > > > >
> > > > > > this seems important, but I don't have all the context on what
> the
> > > > > problem
> > > > > > is.
> > > > > >
> > > > > > Can you explain this sentence "Specifically, there is
> pseudorandom
> > > data
> > > > > > generated and once it has been observed and used to produce a
> side
> > > > > effect,
> > > > > > it cannot be regenerated without erroneous results." ?
> > > > > >
> > > > > > Where is the pseudorandom data coming from? Perhaps a concrete
> > > example
> > > > > > would help?
> > > > > >
> > > > > > S
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > <klk@google.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Problem:
> > > > > > >
> > > > > > > I will drop all nuance and say that the `Write` transform as it
> > > > exists
> > > > > in
> > > > > > > the SDK is incorrect until we add some specification and APIs.
> We
> > > > can't
> > > > > > > keep shipping an SDK with an unsafe transform in it, and IMO
> this
> > > > > > certainly
> > > > > > > blocks a stable release.
> > > > > > >
> > > > > > > Specifically, there is pseudorandom data generated and once it
> > has
> > > > been
> > > > > > > observed and used to produce a side effect, it cannot be
> > > regenerated
> > > > > > > without erroneous results.
> > > > > > >
> > > > > > > This generalizes: For some side-effecting user-defined
> functions,
> > > it
> > > > is
> > > > > > > vital that even across retries/replays they have a consistent
> > view
> > > of
> > > > > the
> > > > > > > contents of their input PCollection, because their effect on
> the
> > > > > outside
> > > > > > > world cannot be retracted if/when they fail and are retried.
> Once
> > > the
> > > > > > > runner ensures a consistent view of the input, it is then their
> > own
> > > > > > > responsibility to be idempotent.
> > > > > > >
> > > > > > > Ideally we should specify this requirement for the user-defined
> > > > > function
> > > > > > > without imposing any particular implementation strategy on Beam
> > > > > runners.
> > > > > > >
> > > > > > > Proposal:
> > > > > > >
> > > > > > > 1. Let a DoFn declare (mechanism not important right now) that
> it
> > > > > > "requires
> > > > > > > deterministic input".
> > > > > > >
> > > > > > > 2. Each runner will need a way to induce deterministic input -
> > the
> > > > > > obvious
> > > > > > > choice being a materialization.
> > > > > > >
> > > > > > > I want to keep the discussion focused, so I'm leaving out any
> > > > > > possibilities
> > > > > > > of taking this further.
> > > > > > >
> > > > > > > Regarding performance: Today places that require this tend to
> be
> > > > > already
> > > > > > > paying the cost via GroupByKey / Reshuffle operations, since
> that
> > > > was a
> > > > > > > simple way to induce determinism in batch Dataflow* (doesn't
> work
> > > for
> > > > > > most
> > > > > > > other runners nor for streaming Dataflow). This change will
> > > replace a
> > > > > > > hard-coded implementation strategy with a requirement that may
> be
> > > > > > fulfilled
> > > > > > > in the most efficient way available.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Kenn (w/ lots of consult from colleagues, especially Ben)
> > > > > > >
> > > > > > > * There is some overlap with the reshuffle/redistribute
> > discussion
> > > > > > because
> > > > > > > of this historical situation, but I would like to leave that
> > > broader
> > > > > > > discussion out of this correctness issue.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Reuven Lax <re...@google.com>.
On Mon, Dec 11, 2017 at 8:47 PM, Kenneth Knowles <kl...@google.com> wrote:

> In discussion on https://github.com/apache/beam/pull/4135 and offline it
> came up how this should interact with stateful/timely DoFns.
>
> Highlights:
>
>  - It was suggested to support it for @OnTimer which would mean that the
> state is stable on retry
>

This sounds good.


>  - But an unfortunate corollary is that for stateful DoFn it should be
> stable after each element
>  - And also if the state is expected to be stable on replay, then ordering
> must be stable, including interleaving of timers and elements
>

I'm not sure why this follows? I think rather we simply need the timer not
to fire _until_ state is stable. Elements (and therefore state writes)
might retry in different orders, but the timer does not fire until the
result is committed.


>
> And since stateful DoFn are very useful for sinks, and
> @RequiresStableInput is very useful for sinks, I wonder if someone has
> thoughts about how these should interact?
>
> In the special case of @OnWindowExpiry (not yet implemented), the most
> important timer for flushing state, it wouldn't actually be so bad.
>
> Kenn
>
> On Wed, Nov 15, 2017 at 7:28 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
>
>> +1
>>
>> > On 15. Nov 2017, at 14:07, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>> >
>> > Agree !
>> >
>> > Thanks Kenn,
>> > Regards
>> > JB
>> >
>> > On 11/15/2017 02:05 PM, Kenneth Knowles wrote:
>> >> Reviving this again, since it came up again today in yet another
>> context. I
>> >> think it is time to add this as an experimental annotation. I think we
>> know
>> >> that we need it, and roughly how it should work, while there are still
>> >> finer points to discuss about what it means for input to be stable.
>> >> So I filed https://issues.apache.org/jira/browse/BEAM-3194 and
>> whipped up
>> >> https://github.com/apache/beam/pull/4135 to move it along a smidge.
>> It will
>> >> need to be incorporated into the Beam model's ParDoPayload and the
>> Python
>> >> SDK as well.
>> >> Kenn
>> >> On Tue, Aug 15, 2017 at 2:40 PM, Reuven Lax <re...@google.com.invalid>
>> >> wrote:
>> >>> Well the Fn API is still being designed, so this is something we'd
>> have to
>> >>> think about.
>> >>>
>> >>> On Tue, Aug 15, 2017 at 2:19 PM, Robert Bradshaw <
>> >>> robertwb@google.com.invalid> wrote:
>> >>>
>> >>>> On Tue, Aug 15, 2017 at 2:14 PM, Reuven Lax <relax@google.com.invalid
>> >
>> >>>> wrote:
>> >>>>> On Tue, Aug 15, 2017 at 1:59 PM, Robert Bradshaw <
>> >>>>> robertwb@google.com.invalid> wrote:
>> >>>>>
>> >>>>>> On Sat, Aug 12, 2017 at 1:13 AM, Reuven Lax
>> <relax@google.com.invalid
>> >>>>
>> >>>>>> wrote:
>> >>>>>>> On Fri, Aug 11, 2017 at 10:52 PM, Robert Bradshaw <
>> >>>>>>>> The question here is whether the ordering is part of the
>> "content"
>> >>> of
>> >>>>>>>> an iterable.
>> >>>>>>>
>> >>>>>>> My initial instinct was to say yes - but maybe it should not be
>> >>> until
>> >>>>>> Beam
>> >>>>>>> has a first-class notion of sorted values after a GBK?
>> >>>>>>
>> >>>>>> Yeah, I'm not sure on this either. Interestingly, if we consider
>> >>>>>> ordering to be important, than the composite gbk + ungroup will be
>> >>>>>> stable despite its components not being so.
>> >>>>>>
>> >>>>>>>>>> As I mention above, the iterable is semantically [part of] a
>> >>>> single
>> >>>>>>>>>> element. So just to unpack this, to make sure we are talking
>> >>> about
>> >>>>>> the
>> >>>>>>>> same
>> >>>>>>>>>> thing, I think you are talking about GBK as implemented via
>> >>> GBKO +
>> >>>>>> GABW.
>> >>>>>>>>>>
>> >>>>>>>>>> When the output of GABW is required to be stable but the output
>> >>> of
>> >>>>>> GBKO
>> >>>>>>>> is
>> >>>>>>>>>> not stable, we don't have stability for free in all cases by
>> >>>>>> inserting a
>> >>>>>>>>>> GBK, but require something more to make the output of GABW
>> >>>> stable, in
>> >>>>>>>> the
>> >>>>>>>>>> worst case a full materialization.
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> Correct. My point is that there are alternate, cheaper ways of
>> >>>> doing
>> >>>>>>>> this.
>> >>>>>>>>> If GABW stores state in an ordered list, it can simply
>> >>> checkpoint a
>> >>>>>>>> market
>> >>>>>>>>> into that list to ensure that the output is stabl.
>> >>>>>>>>
>> >>>>>>>> In the presence of non-trivial triggering and/or late data, I'm
>> not
>> >>>> so
>> >>>>>>>> sure this is "easy." E.g. A bundle may fail, and more data may
>> come
>> >>>> in
>> >>>>>>>> from upstream (and get appended to the buffer) before it is
>> >>> retried.
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>> That will still work. If the subsequent ParDo has processed the
>> >>>> Iterable,
>> >>>>>>> that means we'll have successfully checkpointed a marker to the
>> list
>> >>>>>> (using
>> >>>>>>> whatever technique the runner supports). More data coming in will
>> >>> get
>> >>>>>>> appended after the marker, so we can ensure that the retry still
>> >>> sees
>> >>>> the
>> >>>>>>> same elements in the Iterable.
>> >>>>>>
>> >>>>>> I'm thinking of the following.
>> >>>>>>
>> >>>>>> 1. (k, v1) and (k, v2) come into the GABW and [v1, v2] gets stored
>> in
>> >>>>>> the state. A trigger gets set.
>> >>>>>> 2. The trigger is fired and (k, [v1, v2]) gets sent downstream, but
>> >>>>>> for some reason fails.
>> >>>>>> 3. (k, v3) comes into the GABW and [v3] gets appended to the state.
>> >>>>>> 4. The trigger is again fired, and this time (k, [v1, v2, v3]) is
>> sent
>> >>>>>> downstream.
>> >>>>>>
>> >>>>>>
>> >>>>> If you add the annotation specifying stableinput, then we will not
>> do
>> >>>> this.
>> >>>>> Before we send anything downstream, we will add a marker to the
>> list,
>> >>> and
>> >>>>> only forward data downstream once the marker has been checkpointed.
>> >>> This
>> >>>>> adds a bit of cost and latency of course, but the assumption is that
>> >>>> adding
>> >>>>> this annotation will always add some cost.
>> >>>>
>> >>>> I don't think you can checkpoint anything "before sending data
>> >>>> downstream" if its being executed as part of a fused graph, unless we
>> >>>> add special support for this in the Fn API. I suppose the runner
>> could
>> >>>> pre-emptively modify the state of any GABW operations before firing
>> >>>> triggers...
>> >>>>
>> >>>>>> It is unclear when a marker would be added to the list. Is this in
>> >>>>>> step 2 which, despite failing, still result in modified state [v1,
>> v2,
>> >>>>>> marker]? (And this state modification would have to be committed
>> >>>>>> before attempting the bundle, in case the "failure" was something
>> like
>> >>>>>> a VM shutdown.) And only on success the state is modified to be
>> (say
>> >>>>>> this is accumulating mode) [v1, v2]?
>> >>>>>>
>> >>>>>> I think it could be done, but it may significantly complicate
>> things.
>> >>>>>>
>> >>>>
>> >>>
>> >
>> > --
>> > Jean-Baptiste Onofré
>> > jbonofre@apache.org
>> > http://blog.nanthrax.net
>> > Talend - http://www.talend.com
>>
>>
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Kenneth Knowles <kl...@google.com>.
In discussion on https://github.com/apache/beam/pull/4135 and offline it
came up how this should interact with stateful/timely DoFns.

Highlights:

 - It was suggested to support it for @OnTimer which would mean that the
state is stable on retry
 - But an unfortunate corollary is that for stateful DoFn it should be
stable after each element
 - And also if the state is expected to be stable on replay, then ordering
must be stable, including interleaving of timers and elements

And since stateful DoFn are very useful for sinks, and @RequiresStableInput
is very useful for sinks, I wonder if someone has thoughts about how these
should interact?

In the special case of @OnWindowExpiry (not yet implemented), the most
important timer for flushing state, it wouldn't actually be so bad.

Kenn

On Wed, Nov 15, 2017 at 7:28 AM, Aljoscha Krettek <al...@apache.org>
wrote:

> +1
>
> > On 15. Nov 2017, at 14:07, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> >
> > Agree !
> >
> > Thanks Kenn,
> > Regards
> > JB
> >
> > On 11/15/2017 02:05 PM, Kenneth Knowles wrote:
> >> Reviving this again, since it came up again today in yet another
> context. I
> >> think it is time to add this as an experimental annotation. I think we
> know
> >> that we need it, and roughly how it should work, while there are still
> >> finer points to discuss about what it means for input to be stable.
> >> So I filed https://issues.apache.org/jira/browse/BEAM-3194 and whipped
> up
> >> https://github.com/apache/beam/pull/4135 to move it along a smidge. It
> will
> >> need to be incorporated into the Beam model's ParDoPayload and the
> Python
> >> SDK as well.
> >> Kenn
> >> On Tue, Aug 15, 2017 at 2:40 PM, Reuven Lax <re...@google.com.invalid>
> >> wrote:
> >>> Well the Fn API is still being designed, so this is something we'd
> have to
> >>> think about.
> >>>
> >>> On Tue, Aug 15, 2017 at 2:19 PM, Robert Bradshaw <
> >>> robertwb@google.com.invalid> wrote:
> >>>
> >>>> On Tue, Aug 15, 2017 at 2:14 PM, Reuven Lax <relax@google.com.invalid
> >
> >>>> wrote:
> >>>>> On Tue, Aug 15, 2017 at 1:59 PM, Robert Bradshaw <
> >>>>> robertwb@google.com.invalid> wrote:
> >>>>>
> >>>>>> On Sat, Aug 12, 2017 at 1:13 AM, Reuven Lax
> <relax@google.com.invalid
> >>>>
> >>>>>> wrote:
> >>>>>>> On Fri, Aug 11, 2017 at 10:52 PM, Robert Bradshaw <
> >>>>>>>> The question here is whether the ordering is part of the "content"
> >>> of
> >>>>>>>> an iterable.
> >>>>>>>
> >>>>>>> My initial instinct was to say yes - but maybe it should not be
> >>> until
> >>>>>> Beam
> >>>>>>> has a first-class notion of sorted values after a GBK?
> >>>>>>
> >>>>>> Yeah, I'm not sure on this either. Interestingly, if we consider
> >>>>>> ordering to be important, than the composite gbk + ungroup will be
> >>>>>> stable despite its components not being so.
> >>>>>>
> >>>>>>>>>> As I mention above, the iterable is semantically [part of] a
> >>>> single
> >>>>>>>>>> element. So just to unpack this, to make sure we are talking
> >>> about
> >>>>>> the
> >>>>>>>> same
> >>>>>>>>>> thing, I think you are talking about GBK as implemented via
> >>> GBKO +
> >>>>>> GABW.
> >>>>>>>>>>
> >>>>>>>>>> When the output of GABW is required to be stable but the output
> >>> of
> >>>>>> GBKO
> >>>>>>>> is
> >>>>>>>>>> not stable, we don't have stability for free in all cases by
> >>>>>> inserting a
> >>>>>>>>>> GBK, but require something more to make the output of GABW
> >>>> stable, in
> >>>>>>>> the
> >>>>>>>>>> worst case a full materialization.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Correct. My point is that there are alternate, cheaper ways of
> >>>> doing
> >>>>>>>> this.
> >>>>>>>>> If GABW stores state in an ordered list, it can simply
> >>> checkpoint a
> >>>>>>>> market
> >>>>>>>>> into that list to ensure that the output is stabl.
> >>>>>>>>
> >>>>>>>> In the presence of non-trivial triggering and/or late data, I'm
> not
> >>>> so
> >>>>>>>> sure this is "easy." E.g. A bundle may fail, and more data may
> come
> >>>> in
> >>>>>>>> from upstream (and get appended to the buffer) before it is
> >>> retried.
> >>>>>>>>
> >>>>>>>
> >>>>>>> That will still work. If the subsequent ParDo has processed the
> >>>> Iterable,
> >>>>>>> that means we'll have successfully checkpointed a marker to the
> list
> >>>>>> (using
> >>>>>>> whatever technique the runner supports). More data coming in will
> >>> get
> >>>>>>> appended after the marker, so we can ensure that the retry still
> >>> sees
> >>>> the
> >>>>>>> same elements in the Iterable.
> >>>>>>
> >>>>>> I'm thinking of the following.
> >>>>>>
> >>>>>> 1. (k, v1) and (k, v2) come into the GABW and [v1, v2] gets stored
> in
> >>>>>> the state. A trigger gets set.
> >>>>>> 2. The trigger is fired and (k, [v1, v2]) gets sent downstream, but
> >>>>>> for some reason fails.
> >>>>>> 3. (k, v3) comes into the GABW and [v3] gets appended to the state.
> >>>>>> 4. The trigger is again fired, and this time (k, [v1, v2, v3]) is
> sent
> >>>>>> downstream.
> >>>>>>
> >>>>>>
> >>>>> If you add the annotation specifying stableinput, then we will not do
> >>>> this.
> >>>>> Before we send anything downstream, we will add a marker to the list,
> >>> and
> >>>>> only forward data downstream once the marker has been checkpointed.
> >>> This
> >>>>> adds a bit of cost and latency of course, but the assumption is that
> >>>> adding
> >>>>> this annotation will always add some cost.
> >>>>
> >>>> I don't think you can checkpoint anything "before sending data
> >>>> downstream" if its being executed as part of a fused graph, unless we
> >>>> add special support for this in the Fn API. I suppose the runner could
> >>>> pre-emptively modify the state of any GABW operations before firing
> >>>> triggers...
> >>>>
> >>>>>> It is unclear when a marker would be added to the list. Is this in
> >>>>>> step 2 which, despite failing, still result in modified state [v1,
> v2,
> >>>>>> marker]? (And this state modification would have to be committed
> >>>>>> before attempting the bundle, in case the "failure" was something
> like
> >>>>>> a VM shutdown.) And only on success the state is modified to be (say
> >>>>>> this is accumulating mode) [v1, v2]?
> >>>>>>
> >>>>>> I think it could be done, but it may significantly complicate
> things.
> >>>>>>
> >>>>
> >>>
> >
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
>
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Aljoscha Krettek <al...@apache.org>.
+1

> On 15. Nov 2017, at 14:07, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> 
> Agree !
> 
> Thanks Kenn,
> Regards
> JB
> 
> On 11/15/2017 02:05 PM, Kenneth Knowles wrote:
>> Reviving this again, since it came up again today in yet another context. I
>> think it is time to add this as an experimental annotation. I think we know
>> that we need it, and roughly how it should work, while there are still
>> finer points to discuss about what it means for input to be stable.
>> So I filed https://issues.apache.org/jira/browse/BEAM-3194 and whipped up
>> https://github.com/apache/beam/pull/4135 to move it along a smidge. It will
>> need to be incorporated into the Beam model's ParDoPayload and the Python
>> SDK as well.
>> Kenn
>> On Tue, Aug 15, 2017 at 2:40 PM, Reuven Lax <re...@google.com.invalid>
>> wrote:
>>> Well the Fn API is still being designed, so this is something we'd have to
>>> think about.
>>> 
>>> On Tue, Aug 15, 2017 at 2:19 PM, Robert Bradshaw <
>>> robertwb@google.com.invalid> wrote:
>>> 
>>>> On Tue, Aug 15, 2017 at 2:14 PM, Reuven Lax <re...@google.com.invalid>
>>>> wrote:
>>>>> On Tue, Aug 15, 2017 at 1:59 PM, Robert Bradshaw <
>>>>> robertwb@google.com.invalid> wrote:
>>>>> 
>>>>>> On Sat, Aug 12, 2017 at 1:13 AM, Reuven Lax <relax@google.com.invalid
>>>> 
>>>>>> wrote:
>>>>>>> On Fri, Aug 11, 2017 at 10:52 PM, Robert Bradshaw <
>>>>>>>> The question here is whether the ordering is part of the "content"
>>> of
>>>>>>>> an iterable.
>>>>>>> 
>>>>>>> My initial instinct was to say yes - but maybe it should not be
>>> until
>>>>>> Beam
>>>>>>> has a first-class notion of sorted values after a GBK?
>>>>>> 
>>>>>> Yeah, I'm not sure on this either. Interestingly, if we consider
>>>>>> ordering to be important, than the composite gbk + ungroup will be
>>>>>> stable despite its components not being so.
>>>>>> 
>>>>>>>>>> As I mention above, the iterable is semantically [part of] a
>>>> single
>>>>>>>>>> element. So just to unpack this, to make sure we are talking
>>> about
>>>>>> the
>>>>>>>> same
>>>>>>>>>> thing, I think you are talking about GBK as implemented via
>>> GBKO +
>>>>>> GABW.
>>>>>>>>>> 
>>>>>>>>>> When the output of GABW is required to be stable but the output
>>> of
>>>>>> GBKO
>>>>>>>> is
>>>>>>>>>> not stable, we don't have stability for free in all cases by
>>>>>> inserting a
>>>>>>>>>> GBK, but require something more to make the output of GABW
>>>> stable, in
>>>>>>>> the
>>>>>>>>>> worst case a full materialization.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Correct. My point is that there are alternate, cheaper ways of
>>>> doing
>>>>>>>> this.
>>>>>>>>> If GABW stores state in an ordered list, it can simply
>>> checkpoint a
>>>>>>>> market
>>>>>>>>> into that list to ensure that the output is stabl.
>>>>>>>> 
>>>>>>>> In the presence of non-trivial triggering and/or late data, I'm not
>>>> so
>>>>>>>> sure this is "easy." E.g. A bundle may fail, and more data may come
>>>> in
>>>>>>>> from upstream (and get appended to the buffer) before it is
>>> retried.
>>>>>>>> 
>>>>>>> 
>>>>>>> That will still work. If the subsequent ParDo has processed the
>>>> Iterable,
>>>>>>> that means we'll have successfully checkpointed a marker to the list
>>>>>> (using
>>>>>>> whatever technique the runner supports). More data coming in will
>>> get
>>>>>>> appended after the marker, so we can ensure that the retry still
>>> sees
>>>> the
>>>>>>> same elements in the Iterable.
>>>>>> 
>>>>>> I'm thinking of the following.
>>>>>> 
>>>>>> 1. (k, v1) and (k, v2) come into the GABW and [v1, v2] gets stored in
>>>>>> the state. A trigger gets set.
>>>>>> 2. The trigger is fired and (k, [v1, v2]) gets sent downstream, but
>>>>>> for some reason fails.
>>>>>> 3. (k, v3) comes into the GABW and [v3] gets appended to the state.
>>>>>> 4. The trigger is again fired, and this time (k, [v1, v2, v3]) is sent
>>>>>> downstream.
>>>>>> 
>>>>>> 
>>>>> If you add the annotation specifying stableinput, then we will not do
>>>> this.
>>>>> Before we send anything downstream, we will add a marker to the list,
>>> and
>>>>> only forward data downstream once the marker has been checkpointed.
>>> This
>>>>> adds a bit of cost and latency of course, but the assumption is that
>>>> adding
>>>>> this annotation will always add some cost.
>>>> 
>>>> I don't think you can checkpoint anything "before sending data
>>>> downstream" if its being executed as part of a fused graph, unless we
>>>> add special support for this in the Fn API. I suppose the runner could
>>>> pre-emptively modify the state of any GABW operations before firing
>>>> triggers...
>>>> 
>>>>>> It is unclear when a marker would be added to the list. Is this in
>>>>>> step 2 which, despite failing, still result in modified state [v1, v2,
>>>>>> marker]? (And this state modification would have to be committed
>>>>>> before attempting the bundle, in case the "failure" was something like
>>>>>> a VM shutdown.) And only on success the state is modified to be (say
>>>>>> this is accumulating mode) [v1, v2]?
>>>>>> 
>>>>>> I think it could be done, but it may significantly complicate things.
>>>>>> 
>>>> 
>>> 
> 
> -- 
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com


Re: [PROPOSAL] "Requires deterministic input"

Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Agree !

Thanks Kenn,
Regards
JB

On 11/15/2017 02:05 PM, Kenneth Knowles wrote:
> Reviving this again, since it came up again today in yet another context. I
> think it is time to add this as an experimental annotation. I think we know
> that we need it, and roughly how it should work, while there are still
> finer points to discuss about what it means for input to be stable.
> 
> So I filed https://issues.apache.org/jira/browse/BEAM-3194 and whipped up
> https://github.com/apache/beam/pull/4135 to move it along a smidge. It will
> need to be incorporated into the Beam model's ParDoPayload and the Python
> SDK as well.
> 
> Kenn
> 
> On Tue, Aug 15, 2017 at 2:40 PM, Reuven Lax <re...@google.com.invalid>
> wrote:
> 
>> Well the Fn API is still being designed, so this is something we'd have to
>> think about.
>>
>> On Tue, Aug 15, 2017 at 2:19 PM, Robert Bradshaw <
>> robertwb@google.com.invalid> wrote:
>>
>>> On Tue, Aug 15, 2017 at 2:14 PM, Reuven Lax <re...@google.com.invalid>
>>> wrote:
>>>> On Tue, Aug 15, 2017 at 1:59 PM, Robert Bradshaw <
>>>> robertwb@google.com.invalid> wrote:
>>>>
>>>>> On Sat, Aug 12, 2017 at 1:13 AM, Reuven Lax <relax@google.com.invalid
>>>
>>>>> wrote:
>>>>>> On Fri, Aug 11, 2017 at 10:52 PM, Robert Bradshaw <
>>>>>>> The question here is whether the ordering is part of the "content"
>> of
>>>>>>> an iterable.
>>>>>>
>>>>>> My initial instinct was to say yes - but maybe it should not be
>> until
>>>>> Beam
>>>>>> has a first-class notion of sorted values after a GBK?
>>>>>
>>>>> Yeah, I'm not sure on this either. Interestingly, if we consider
>>>>> ordering to be important, than the composite gbk + ungroup will be
>>>>> stable despite its components not being so.
>>>>>
>>>>>>>>> As I mention above, the iterable is semantically [part of] a
>>> single
>>>>>>>>> element. So just to unpack this, to make sure we are talking
>> about
>>>>> the
>>>>>>> same
>>>>>>>>> thing, I think you are talking about GBK as implemented via
>> GBKO +
>>>>> GABW.
>>>>>>>>>
>>>>>>>>> When the output of GABW is required to be stable but the output
>> of
>>>>> GBKO
>>>>>>> is
>>>>>>>>> not stable, we don't have stability for free in all cases by
>>>>> inserting a
>>>>>>>>> GBK, but require something more to make the output of GABW
>>> stable, in
>>>>>>> the
>>>>>>>>> worst case a full materialization.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Correct. My point is that there are alternate, cheaper ways of
>>> doing
>>>>>>> this.
>>>>>>>> If GABW stores state in an ordered list, it can simply
>> checkpoint a
>>>>>>> market
>>>>>>>> into that list to ensure that the output is stabl.
>>>>>>>
>>>>>>> In the presence of non-trivial triggering and/or late data, I'm not
>>> so
>>>>>>> sure this is "easy." E.g. A bundle may fail, and more data may come
>>> in
>>>>>>> from upstream (and get appended to the buffer) before it is
>> retried.
>>>>>>>
>>>>>>
>>>>>> That will still work. If the subsequent ParDo has processed the
>>> Iterable,
>>>>>> that means we'll have successfully checkpointed a marker to the list
>>>>> (using
>>>>>> whatever technique the runner supports). More data coming in will
>> get
>>>>>> appended after the marker, so we can ensure that the retry still
>> sees
>>> the
>>>>>> same elements in the Iterable.
>>>>>
>>>>> I'm thinking of the following.
>>>>>
>>>>> 1. (k, v1) and (k, v2) come into the GABW and [v1, v2] gets stored in
>>>>> the state. A trigger gets set.
>>>>> 2. The trigger is fired and (k, [v1, v2]) gets sent downstream, but
>>>>> for some reason fails.
>>>>> 3. (k, v3) comes into the GABW and [v3] gets appended to the state.
>>>>> 4. The trigger is again fired, and this time (k, [v1, v2, v3]) is sent
>>>>> downstream.
>>>>>
>>>>>
>>>> If you add the annotation specifying stableinput, then we will not do
>>> this.
>>>> Before we send anything downstream, we will add a marker to the list,
>> and
>>>> only forward data downstream once the marker has been checkpointed.
>> This
>>>> adds a bit of cost and latency of course, but the assumption is that
>>> adding
>>>> this annotation will always add some cost.
>>>
>>> I don't think you can checkpoint anything "before sending data
>>> downstream" if its being executed as part of a fused graph, unless we
>>> add special support for this in the Fn API. I suppose the runner could
>>> pre-emptively modify the state of any GABW operations before firing
>>> triggers...
>>>
>>>>> It is unclear when a marker would be added to the list. Is this in
>>>>> step 2 which, despite failing, still result in modified state [v1, v2,
>>>>> marker]? (And this state modification would have to be committed
>>>>> before attempting the bundle, in case the "failure" was something like
>>>>> a VM shutdown.) And only on success the state is modified to be (say
>>>>> this is accumulating mode) [v1, v2]?
>>>>>
>>>>> I think it could be done, but it may significantly complicate things.
>>>>>
>>>
>>
> 

-- 
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com

Re: [PROPOSAL] "Requires deterministic input"

Posted by Kenneth Knowles <kl...@google.com.INVALID>.
Reviving this again, since it came up again today in yet another context. I
think it is time to add this as an experimental annotation. I think we know
that we need it, and roughly how it should work, while there are still
finer points to discuss about what it means for input to be stable.

So I filed https://issues.apache.org/jira/browse/BEAM-3194 and whipped up
https://github.com/apache/beam/pull/4135 to move it along a smidge. It will
need to be incorporated into the Beam model's ParDoPayload and the Python
SDK as well.

Kenn

On Tue, Aug 15, 2017 at 2:40 PM, Reuven Lax <re...@google.com.invalid>
wrote:

> Well the Fn API is still being designed, so this is something we'd have to
> think about.
>
> On Tue, Aug 15, 2017 at 2:19 PM, Robert Bradshaw <
> robertwb@google.com.invalid> wrote:
>
> > On Tue, Aug 15, 2017 at 2:14 PM, Reuven Lax <re...@google.com.invalid>
> > wrote:
> > > On Tue, Aug 15, 2017 at 1:59 PM, Robert Bradshaw <
> > > robertwb@google.com.invalid> wrote:
> > >
> > >> On Sat, Aug 12, 2017 at 1:13 AM, Reuven Lax <relax@google.com.invalid
> >
> > >> wrote:
> > >> > On Fri, Aug 11, 2017 at 10:52 PM, Robert Bradshaw <
> > >> >> The question here is whether the ordering is part of the "content"
> of
> > >> >> an iterable.
> > >> >
> > >> > My initial instinct was to say yes - but maybe it should not be
> until
> > >> Beam
> > >> > has a first-class notion of sorted values after a GBK?
> > >>
> > >> Yeah, I'm not sure on this either. Interestingly, if we consider
> > >> ordering to be important, than the composite gbk + ungroup will be
> > >> stable despite its components not being so.
> > >>
> > >> >> >> As I mention above, the iterable is semantically [part of] a
> > single
> > >> >> >> element. So just to unpack this, to make sure we are talking
> about
> > >> the
> > >> >> same
> > >> >> >> thing, I think you are talking about GBK as implemented via
> GBKO +
> > >> GABW.
> > >> >> >>
> > >> >> >> When the output of GABW is required to be stable but the output
> of
> > >> GBKO
> > >> >> is
> > >> >> >> not stable, we don't have stability for free in all cases by
> > >> inserting a
> > >> >> >> GBK, but require something more to make the output of GABW
> > stable, in
> > >> >> the
> > >> >> >> worst case a full materialization.
> > >> >> >>
> > >> >> >
> > >> >> > Correct. My point is that there are alternate, cheaper ways of
> > doing
> > >> >> this.
> > >> >> > If GABW stores state in an ordered list, it can simply
> checkpoint a
> > >> >> market
> > >> >> > into that list to ensure that the output is stabl.
> > >> >>
> > >> >> In the presence of non-trivial triggering and/or late data, I'm not
> > so
> > >> >> sure this is "easy." E.g. A bundle may fail, and more data may come
> > in
> > >> >> from upstream (and get appended to the buffer) before it is
> retried.
> > >> >>
> > >> >
> > >> > That will still work. If the subsequent ParDo has processed the
> > Iterable,
> > >> > that means we'll have successfully checkpointed a marker to the list
> > >> (using
> > >> > whatever technique the runner supports). More data coming in will
> get
> > >> > appended after the marker, so we can ensure that the retry still
> sees
> > the
> > >> > same elements in the Iterable.
> > >>
> > >> I'm thinking of the following.
> > >>
> > >> 1. (k, v1) and (k, v2) come into the GABW and [v1, v2] gets stored in
> > >> the state. A trigger gets set.
> > >> 2. The trigger is fired and (k, [v1, v2]) gets sent downstream, but
> > >> for some reason fails.
> > >> 3. (k, v3) comes into the GABW and [v3] gets appended to the state.
> > >> 4. The trigger is again fired, and this time (k, [v1, v2, v3]) is sent
> > >> downstream.
> > >>
> > >>
> > > If you add the annotation specifying stableinput, then we will not do
> > this.
> > > Before we send anything downstream, we will add a marker to the list,
> and
> > > only forward data downstream once the marker has been checkpointed.
> This
> > > adds a bit of cost and latency of course, but the assumption is that
> > adding
> > > this annotation will always add some cost.
> >
> > I don't think you can checkpoint anything "before sending data
> > downstream" if its being executed as part of a fused graph, unless we
> > add special support for this in the Fn API. I suppose the runner could
> > pre-emptively modify the state of any GABW operations before firing
> > triggers...
> >
> > >> It is unclear when a marker would be added to the list. Is this in
> > >> step 2 which, despite failing, still result in modified state [v1, v2,
> > >> marker]? (And this state modification would have to be committed
> > >> before attempting the bundle, in case the "failure" was something like
> > >> a VM shutdown.) And only on success the state is modified to be (say
> > >> this is accumulating mode) [v1, v2]?
> > >>
> > >> I think it could be done, but it may significantly complicate things.
> > >>
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Reuven Lax <re...@google.com.INVALID>.
Well the Fn API is still being designed, so this is something we'd have to
think about.

On Tue, Aug 15, 2017 at 2:19 PM, Robert Bradshaw <
robertwb@google.com.invalid> wrote:

> On Tue, Aug 15, 2017 at 2:14 PM, Reuven Lax <re...@google.com.invalid>
> wrote:
> > On Tue, Aug 15, 2017 at 1:59 PM, Robert Bradshaw <
> > robertwb@google.com.invalid> wrote:
> >
> >> On Sat, Aug 12, 2017 at 1:13 AM, Reuven Lax <re...@google.com.invalid>
> >> wrote:
> >> > On Fri, Aug 11, 2017 at 10:52 PM, Robert Bradshaw <
> >> >> The question here is whether the ordering is part of the "content" of
> >> >> an iterable.
> >> >
> >> > My initial instinct was to say yes - but maybe it should not be until
> >> Beam
> >> > has a first-class notion of sorted values after a GBK?
> >>
> >> Yeah, I'm not sure on this either. Interestingly, if we consider
> >> ordering to be important, than the composite gbk + ungroup will be
> >> stable despite its components not being so.
> >>
> >> >> >> As I mention above, the iterable is semantically [part of] a
> single
> >> >> >> element. So just to unpack this, to make sure we are talking about
> >> the
> >> >> same
> >> >> >> thing, I think you are talking about GBK as implemented via GBKO +
> >> GABW.
> >> >> >>
> >> >> >> When the output of GABW is required to be stable but the output of
> >> GBKO
> >> >> is
> >> >> >> not stable, we don't have stability for free in all cases by
> >> inserting a
> >> >> >> GBK, but require something more to make the output of GABW
> stable, in
> >> >> the
> >> >> >> worst case a full materialization.
> >> >> >>
> >> >> >
> >> >> > Correct. My point is that there are alternate, cheaper ways of
> doing
> >> >> this.
> >> >> > If GABW stores state in an ordered list, it can simply checkpoint a
> >> >> market
> >> >> > into that list to ensure that the output is stabl.
> >> >>
> >> >> In the presence of non-trivial triggering and/or late data, I'm not
> so
> >> >> sure this is "easy." E.g. A bundle may fail, and more data may come
> in
> >> >> from upstream (and get appended to the buffer) before it is retried.
> >> >>
> >> >
> >> > That will still work. If the subsequent ParDo has processed the
> Iterable,
> >> > that means we'll have successfully checkpointed a marker to the list
> >> (using
> >> > whatever technique the runner supports). More data coming in will get
> >> > appended after the marker, so we can ensure that the retry still sees
> the
> >> > same elements in the Iterable.
> >>
> >> I'm thinking of the following.
> >>
> >> 1. (k, v1) and (k, v2) come into the GABW and [v1, v2] gets stored in
> >> the state. A trigger gets set.
> >> 2. The trigger is fired and (k, [v1, v2]) gets sent downstream, but
> >> for some reason fails.
> >> 3. (k, v3) comes into the GABW and [v3] gets appended to the state.
> >> 4. The trigger is again fired, and this time (k, [v1, v2, v3]) is sent
> >> downstream.
> >>
> >>
> > If you add the annotation specifying stableinput, then we will not do
> this.
> > Before we send anything downstream, we will add a marker to the list, and
> > only forward data downstream once the marker has been checkpointed. This
> > adds a bit of cost and latency of course, but the assumption is that
> adding
> > this annotation will always add some cost.
>
> I don't think you can checkpoint anything "before sending data
> downstream" if its being executed as part of a fused graph, unless we
> add special support for this in the Fn API. I suppose the runner could
> pre-emptively modify the state of any GABW operations before firing
> triggers...
>
> >> It is unclear when a marker would be added to the list. Is this in
> >> step 2 which, despite failing, still result in modified state [v1, v2,
> >> marker]? (And this state modification would have to be committed
> >> before attempting the bundle, in case the "failure" was something like
> >> a VM shutdown.) And only on success the state is modified to be (say
> >> this is accumulating mode) [v1, v2]?
> >>
> >> I think it could be done, but it may significantly complicate things.
> >>
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Robert Bradshaw <ro...@google.com.INVALID>.
On Tue, Aug 15, 2017 at 2:14 PM, Reuven Lax <re...@google.com.invalid> wrote:
> On Tue, Aug 15, 2017 at 1:59 PM, Robert Bradshaw <
> robertwb@google.com.invalid> wrote:
>
>> On Sat, Aug 12, 2017 at 1:13 AM, Reuven Lax <re...@google.com.invalid>
>> wrote:
>> > On Fri, Aug 11, 2017 at 10:52 PM, Robert Bradshaw <
>> >> The question here is whether the ordering is part of the "content" of
>> >> an iterable.
>> >
>> > My initial instinct was to say yes - but maybe it should not be until
>> Beam
>> > has a first-class notion of sorted values after a GBK?
>>
>> Yeah, I'm not sure on this either. Interestingly, if we consider
>> ordering to be important, than the composite gbk + ungroup will be
>> stable despite its components not being so.
>>
>> >> >> As I mention above, the iterable is semantically [part of] a single
>> >> >> element. So just to unpack this, to make sure we are talking about
>> the
>> >> same
>> >> >> thing, I think you are talking about GBK as implemented via GBKO +
>> GABW.
>> >> >>
>> >> >> When the output of GABW is required to be stable but the output of
>> GBKO
>> >> is
>> >> >> not stable, we don't have stability for free in all cases by
>> inserting a
>> >> >> GBK, but require something more to make the output of GABW stable, in
>> >> the
>> >> >> worst case a full materialization.
>> >> >>
>> >> >
>> >> > Correct. My point is that there are alternate, cheaper ways of doing
>> >> this.
>> >> > If GABW stores state in an ordered list, it can simply checkpoint a
>> >> market
>> >> > into that list to ensure that the output is stabl.
>> >>
>> >> In the presence of non-trivial triggering and/or late data, I'm not so
>> >> sure this is "easy." E.g. A bundle may fail, and more data may come in
>> >> from upstream (and get appended to the buffer) before it is retried.
>> >>
>> >
>> > That will still work. If the subsequent ParDo has processed the Iterable,
>> > that means we'll have successfully checkpointed a marker to the list
>> (using
>> > whatever technique the runner supports). More data coming in will get
>> > appended after the marker, so we can ensure that the retry still sees the
>> > same elements in the Iterable.
>>
>> I'm thinking of the following.
>>
>> 1. (k, v1) and (k, v2) come into the GABW and [v1, v2] gets stored in
>> the state. A trigger gets set.
>> 2. The trigger is fired and (k, [v1, v2]) gets sent downstream, but
>> for some reason fails.
>> 3. (k, v3) comes into the GABW and [v3] gets appended to the state.
>> 4. The trigger is again fired, and this time (k, [v1, v2, v3]) is sent
>> downstream.
>>
>>
> If you add the annotation specifying stableinput, then we will not do this.
> Before we send anything downstream, we will add a marker to the list, and
> only forward data downstream once the marker has been checkpointed. This
> adds a bit of cost and latency of course, but the assumption is that adding
> this annotation will always add some cost.

I don't think you can checkpoint anything "before sending data
downstream" if its being executed as part of a fused graph, unless we
add special support for this in the Fn API. I suppose the runner could
pre-emptively modify the state of any GABW operations before firing
triggers...

>> It is unclear when a marker would be added to the list. Is this in
>> step 2 which, despite failing, still result in modified state [v1, v2,
>> marker]? (And this state modification would have to be committed
>> before attempting the bundle, in case the "failure" was something like
>> a VM shutdown.) And only on success the state is modified to be (say
>> this is accumulating mode) [v1, v2]?
>>
>> I think it could be done, but it may significantly complicate things.
>>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Reuven Lax <re...@google.com.INVALID>.
On Tue, Aug 15, 2017 at 1:59 PM, Robert Bradshaw <
robertwb@google.com.invalid> wrote:

> On Sat, Aug 12, 2017 at 1:13 AM, Reuven Lax <re...@google.com.invalid>
> wrote:
> > On Fri, Aug 11, 2017 at 10:52 PM, Robert Bradshaw <
> >> The question here is whether the ordering is part of the "content" of
> >> an iterable.
> >
> > My initial instinct was to say yes - but maybe it should not be until
> Beam
> > has a first-class notion of sorted values after a GBK?
>
> Yeah, I'm not sure on this either. Interestingly, if we consider
> ordering to be important, than the composite gbk + ungroup will be
> stable despite its components not being so.
>
> >> >> As I mention above, the iterable is semantically [part of] a single
> >> >> element. So just to unpack this, to make sure we are talking about
> the
> >> same
> >> >> thing, I think you are talking about GBK as implemented via GBKO +
> GABW.
> >> >>
> >> >> When the output of GABW is required to be stable but the output of
> GBKO
> >> is
> >> >> not stable, we don't have stability for free in all cases by
> inserting a
> >> >> GBK, but require something more to make the output of GABW stable, in
> >> the
> >> >> worst case a full materialization.
> >> >>
> >> >
> >> > Correct. My point is that there are alternate, cheaper ways of doing
> >> this.
> >> > If GABW stores state in an ordered list, it can simply checkpoint a
> >> market
> >> > into that list to ensure that the output is stabl.
> >>
> >> In the presence of non-trivial triggering and/or late data, I'm not so
> >> sure this is "easy." E.g. A bundle may fail, and more data may come in
> >> from upstream (and get appended to the buffer) before it is retried.
> >>
> >
> > That will still work. If the subsequent ParDo has processed the Iterable,
> > that means we'll have successfully checkpointed a marker to the list
> (using
> > whatever technique the runner supports). More data coming in will get
> > appended after the marker, so we can ensure that the retry still sees the
> > same elements in the Iterable.
>
> I'm thinking of the following.
>
> 1. (k, v1) and (k, v2) come into the GABW and [v1, v2] gets stored in
> the state. A trigger gets set.
> 2. The trigger is fired and (k, [v1, v2]) gets sent downstream, but
> for some reason fails.
> 3. (k, v3) comes into the GABW and [v3] gets appended to the state.
> 4. The trigger is again fired, and this time (k, [v1, v2, v3]) is sent
> downstream.
>
>
If you add the annotation specifying stableinput, then we will not do this.
Before we send anything downstream, we will add a marker to the list, and
only forward data downstream once the marker has been checkpointed. This
adds a bit of cost and latency of course, but the assumption is that adding
this annotation will always add some cost.



> It is unclear when a marker would be added to the list. Is this in
> step 2 which, despite failing, still result in modified state [v1, v2,
> marker]? (And this state modification would have to be committed
> before attempting the bundle, in case the "failure" was something like
> a VM shutdown.) And only on success the state is modified to be (say
> this is accumulating mode) [v1, v2]?
>
> I think it could be done, but it may significantly complicate things.
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Robert Bradshaw <ro...@google.com.INVALID>.
On Sat, Aug 12, 2017 at 1:13 AM, Reuven Lax <re...@google.com.invalid> wrote:
> On Fri, Aug 11, 2017 at 10:52 PM, Robert Bradshaw <
>> The question here is whether the ordering is part of the "content" of
>> an iterable.
>
> My initial instinct was to say yes - but maybe it should not be until Beam
> has a first-class notion of sorted values after a GBK?

Yeah, I'm not sure on this either. Interestingly, if we consider
ordering to be important, than the composite gbk + ungroup will be
stable despite its components not being so.

>> >> As I mention above, the iterable is semantically [part of] a single
>> >> element. So just to unpack this, to make sure we are talking about the
>> same
>> >> thing, I think you are talking about GBK as implemented via GBKO + GABW.
>> >>
>> >> When the output of GABW is required to be stable but the output of GBKO
>> is
>> >> not stable, we don't have stability for free in all cases by inserting a
>> >> GBK, but require something more to make the output of GABW stable, in
>> the
>> >> worst case a full materialization.
>> >>
>> >
>> > Correct. My point is that there are alternate, cheaper ways of doing
>> this.
>> > If GABW stores state in an ordered list, it can simply checkpoint a
>> market
>> > into that list to ensure that the output is stabl.
>>
>> In the presence of non-trivial triggering and/or late data, I'm not so
>> sure this is "easy." E.g. A bundle may fail, and more data may come in
>> from upstream (and get appended to the buffer) before it is retried.
>>
>
> That will still work. If the subsequent ParDo has processed the Iterable,
> that means we'll have successfully checkpointed a marker to the list (using
> whatever technique the runner supports). More data coming in will get
> appended after the marker, so we can ensure that the retry still sees the
> same elements in the Iterable.

I'm thinking of the following.

1. (k, v1) and (k, v2) come into the GABW and [v1, v2] gets stored in
the state. A trigger gets set.
2. The trigger is fired and (k, [v1, v2]) gets sent downstream, but
for some reason fails.
3. (k, v3) comes into the GABW and [v3] gets appended to the state.
4. The trigger is again fired, and this time (k, [v1, v2, v3]) is sent
downstream.

It is unclear when a marker would be added to the list. Is this in
step 2 which, despite failing, still result in modified state [v1, v2,
marker]? (And this state modification would have to be committed
before attempting the bundle, in case the "failure" was something like
a VM shutdown.) And only on success the state is modified to be (say
this is accumulating mode) [v1, v2]?

I think it could be done, but it may significantly complicate things.

Re: [PROPOSAL] "Requires deterministic input"

Posted by Reuven Lax <re...@google.com.INVALID>.
On Fri, Aug 11, 2017 at 10:52 PM, Robert Bradshaw <
robertwb@google.com.invalid> wrote:

> On Thu, Aug 10, 2017 at 1:53 PM, Reuven Lax <re...@google.com.invalid>
> wrote:
> > On Thu, Aug 10, 2017 at 1:07 PM, Kenneth Knowles <klk@google.com.invalid
> >
> > wrote:
> >
> >>  > > >    - Does it also imply fixed length and content for value
> >> iterators?
> >> > > >
> >>
> >> The concept of "value iterator" brings up a nit.
> >>
> >> First, there is no such concept in the Beam model, and I don't think
> there
> >> should be. I don't think we should special case GBK if we can avoid it.
> If
> >> a PCollection contains elements of type KV<K, Iterable<V>> we should
> have
> >> the same definition of "stable" whether or not it came from GBK. So I
> >> think, for some definition of "fixed content", it should imply fixed
> length
> >> and content.
> >
> > I agree. Of course it might take longer to implement this well. It's
> > perfectly acceptable for this to temporarily not work in this scenario,
> as
> > long as we know how to make it work.
>
> The question here is whether the ordering is part of the "content" of
> an iterable.
>

My initial instinct was to say yes - but maybe it should not be until Beam
has a first-class notion of sorted values after a GBK?


>
> >> But actually what is the definition of whether two Iterable<V> values
> have
> >> the same content, since we don't require a deterministic coder for the
> >> values? I think this question applies to this proposal in general.
> >> Obviously we expect that if the runner is replaying from persisted
> >> serialized data, then it suffices, but that is too operational for a
> good
> >> definition IMO.
> >
> > I don't quite understand what you mean here. The user has a definition of
> > equality which is independent of coders (e.g. equals() in Java).
>
> I agree with this definition; the difficulty here arises in the fact
> that the Iterable interface does not specify what equals() should
> return for collections that differ only in order. (E.g. Set and List
> both implement Iterable). Beam places no constraint on the ordering of
> values; is this part of the equivalence class/is equivalent according
> to the user. In other words, does that mean on replay if the ordering
> differs it's considered "stable."
>
> > Even
> > non-deterministic coders are expected to deserialize to objects that the
> > user finds equivalent. Or if you want to word things a bit more formally
> -
> > the set of encoded objects can be partitioned into equivalence classes,
> > based on the users' notion of equality. A deterministic coder is simply
> one
> > in which each equivalence class has cardinality one.
> >
> > the KV<K,
> >> > > > Iterable<V>> has no good way of being deterministic if there is
> late
> >> > > data.
> >> > > > We could do so by forcing the Iterable to be materialized into a
> >> single
> >> > > > element, but that would also mean that the entire Iterable must
> fit
> >> in
> >> > > > memory (which at least the Dataflow runner does not require).
> >> >
> >>
> >> As I mention above, the iterable is semantically [part of] a single
> >> element. So just to unpack this, to make sure we are talking about the
> same
> >> thing, I think you are talking about GBK as implemented via GBKO + GABW.
> >>
> >> When the output of GABW is required to be stable but the output of GBKO
> is
> >> not stable, we don't have stability for free in all cases by inserting a
> >> GBK, but require something more to make the output of GABW stable, in
> the
> >> worst case a full materialization.
> >>
> >
> > Correct. My point is that there are alternate, cheaper ways of doing
> this.
> > If GABW stores state in an ordered list, it can simply checkpoint a
> market
> > into that list to ensure that the output is stabl.
>
> In the presence of non-trivial triggering and/or late data, I'm not so
> sure this is "easy." E.g. A bundle may fail, and more data may come in
> from upstream (and get appended to the buffer) before it is retried.
>

That will still work. If the subsequent ParDo has processed the Iterable,
that means we'll have successfully checkpointed a marker to the list (using
whatever technique the runner supports). More data coming in will get
appended after the marker, so we can ensure that the retry still sees the
same elements in the Iterable.

Re: [PROPOSAL] "Requires deterministic input"

Posted by Robert Bradshaw <ro...@google.com.INVALID>.
On Thu, Aug 10, 2017 at 1:53 PM, Reuven Lax <re...@google.com.invalid> wrote:
> On Thu, Aug 10, 2017 at 1:07 PM, Kenneth Knowles <kl...@google.com.invalid>
> wrote:
>
>>  > > >    - Does it also imply fixed length and content for value
>> iterators?
>> > > >
>>
>> The concept of "value iterator" brings up a nit.
>>
>> First, there is no such concept in the Beam model, and I don't think there
>> should be. I don't think we should special case GBK if we can avoid it. If
>> a PCollection contains elements of type KV<K, Iterable<V>> we should have
>> the same definition of "stable" whether or not it came from GBK. So I
>> think, for some definition of "fixed content", it should imply fixed length
>> and content.
>
> I agree. Of course it might take longer to implement this well. It's
> perfectly acceptable for this to temporarily not work in this scenario, as
> long as we know how to make it work.

The question here is whether the ordering is part of the "content" of
an iterable.

>> But actually what is the definition of whether two Iterable<V> values have
>> the same content, since we don't require a deterministic coder for the
>> values? I think this question applies to this proposal in general.
>> Obviously we expect that if the runner is replaying from persisted
>> serialized data, then it suffices, but that is too operational for a good
>> definition IMO.
>
> I don't quite understand what you mean here. The user has a definition of
> equality which is independent of coders (e.g. equals() in Java).

I agree with this definition; the difficulty here arises in the fact
that the Iterable interface does not specify what equals() should
return for collections that differ only in order. (E.g. Set and List
both implement Iterable). Beam places no constraint on the ordering of
values; is this part of the equivalence class/is equivalent according
to the user. In other words, does that mean on replay if the ordering
differs it's considered "stable."

> Even
> non-deterministic coders are expected to deserialize to objects that the
> user finds equivalent. Or if you want to word things a bit more formally -
> the set of encoded objects can be partitioned into equivalence classes,
> based on the users' notion of equality. A deterministic coder is simply one
> in which each equivalence class has cardinality one.
>
> the KV<K,
>> > > > Iterable<V>> has no good way of being deterministic if there is late
>> > > data.
>> > > > We could do so by forcing the Iterable to be materialized into a
>> single
>> > > > element, but that would also mean that the entire Iterable must fit
>> in
>> > > > memory (which at least the Dataflow runner does not require).
>> >
>>
>> As I mention above, the iterable is semantically [part of] a single
>> element. So just to unpack this, to make sure we are talking about the same
>> thing, I think you are talking about GBK as implemented via GBKO + GABW.
>>
>> When the output of GABW is required to be stable but the output of GBKO is
>> not stable, we don't have stability for free in all cases by inserting a
>> GBK, but require something more to make the output of GABW stable, in the
>> worst case a full materialization.
>>
>
> Correct. My point is that there are alternate, cheaper ways of doing this.
> If GABW stores state in an ordered list, it can simply checkpoint a market
> into that list to ensure that the output is stabl.

In the presence of non-trivial triggering and/or late data, I'm not so
sure this is "easy." E.g. A bundle may fail, and more data may come in
from upstream (and get appended to the buffer) before it is retried.

Re: [PROPOSAL] "Requires deterministic input"

Posted by Reuven Lax <re...@google.com.INVALID>.
On Thu, Aug 10, 2017 at 1:07 PM, Kenneth Knowles <kl...@google.com.invalid>
wrote:

>  > > >    - Does it also imply fixed length and content for value
> iterators?
> > > >
>
> The concept of "value iterator" brings up a nit.
>
> First, there is no such concept in the Beam model, and I don't think there
> should be. I don't think we should special case GBK if we can avoid it. If
> a PCollection contains elements of type KV<K, Iterable<V>> we should have
> the same definition of "stable" whether or not it came from GBK. So I
> think, for some definition of "fixed content", it should imply fixed length
> and content.
>

I agree. Of course it might take longer to implement this well. It's
perfectly acceptable for this to temporarily not work in this scenario, as
long as we know how to make it work.


>
> But actually what is the definition of whether two Iterable<V> values have
> the same content, since we don't require a deterministic coder for the
> values? I think this question applies to this proposal in general.
> Obviously we expect that if the runner is replaying from persisted
> serialized data, then it suffices, but that is too operational for a good
> definition IMO.
>

I don't quite understand what you mean here. The user has a definition of
equality which is independent of coders (e.g. equals() in Java). Even
non-deterministic coders are expected to deserialize to objects that the
user finds equivalent. Or if you want to word things a bit more formally -
the set of encoded objects can be partitioned into equivalence classes,
based on the users' notion of equality. A deterministic coder is simply one
in which each equivalence class has cardinality one.

the KV<K,
> > > > Iterable<V>> has no good way of being deterministic if there is late
> > > data.
> > > > We could do so by forcing the Iterable to be materialized into a
> single
> > > > element, but that would also mean that the entire Iterable must fit
> in
> > > > memory (which at least the Dataflow runner does not require).
> >
>
> As I mention above, the iterable is semantically [part of] a single
> element. So just to unpack this, to make sure we are talking about the same
> thing, I think you are talking about GBK as implemented via GBKO + GABW.
>
> When the output of GABW is required to be stable but the output of GBKO is
> not stable, we don't have stability for free in all cases by inserting a
> GBK, but require something more to make the output of GABW stable, in the
> worst case a full materialization.
>

Correct. My point is that there are alternate, cheaper ways of doing this.
If GABW stores state in an ordered list, it can simply checkpoint a market
into that list to ensure that the output is stabl.


>
> In a perfect (future) world, if we were in accumulating mode (aka each new
> element is a full replacement value) then we would know that the sink will
> just overwrite prior values so it is safe to be unstable w.r.t. to
> triggerings (but not other side effects). We could materialize at most the
> pane index (aka sequence number for the write).
>
> For discarding mode (aka each new element is incremental output) I think
> you need full stability of the pane index / sequence number and its
> associated element. There might be different ways to implement, such as the
> ones suggested so far, depending on the runner.
>
> Kenn
>
> > >
> > > >
> > > > >    - Some examples to clarify nuances would be very useful.
> > > > >
> > > > > State durability semantics for timers that Reuven proposed seem to
> be
> > > > > unrelated to stable input (at model level). It might be simpler to
> > add
> > > > > these semantics first. A lot of deterministic side-effects issues
> can
> > > be
> > > > > handled by durable state in timers. One thing I like about timers
> > > > approach
> > > > > is that it makes the cost more transparent to the user since the
> > state
> > > is
> > > > > explicitly stored.
> > > > >
> > > > >
> > > > > On Thu, Aug 10, 2017 at 10:02 AM, Ben Chambers
> > > > > <bchambers@google.com.invalid
> > > > > > wrote:
> > > > >
> > > > > > I think it only makes sense in places where a user might
> reasonable
> > > > > require
> > > > > > stable input to ensure idempotency of side-effects. It also only
> > > makes
> > > > > > sense in places where a runner could reasonably provide such a
> > > > guarantee.
> > > > > >
> > > > > > A given Combine is unlikely to have side effects so it is less
> > likely
> > > > to
> > > > > > benefit from stability of the input. Further, the reason writing
> a
> > > > > Combine
> > > > > > is desirable is because its execution can be split up and moved
> to
> > > the
> > > > > > mapper-side (before the GroupByKey). But this division is
> > inherently
> > > > > > non-deterministic, and so it seems unlikely to benefit from
> > > stability.
> > > > > And
> > > > > > many cases where I could see wanting side-effects would end up in
> > > > > > extractOutput, for which there is an easy (arguably better)
> > solution
> > > --
> > > > > > have extractOutput return the accumulators and do the
> side-effects
> > > in a
> > > > > > DoFn afterwards.
> > > > > >
> > > > > > For composites, it is a bit trickier. I could see a case for
> > > supporting
> > > > > it
> > > > > > on composites, but it would need to make it very clear that it
> only
> > > > > > affected the input to the composite. If any of the operations
> > within
> > > > the
> > > > > > composite were non-deterministic, then the outputs of that could
> be
> > > > > > unstable, leading to instability in later parts of the composite.
> > > > > Further,
> > > > > > it doesn't seem to offer much. The composite itself doesn't
> perform
> > > > > > side-effects, so there is no benefit to having the annotation
> there
> > > --
> > > > > > instead, we allow the annotation to be put where it is relevant
> and
> > > > > > important -- on the DoFn's that actually have side-effects that
> > > require
> > > > > > stability.
> > > > > >
> > > > > > On Thu, Aug 10, 2017 at 9:23 AM Reuven Lax
> > <relax@google.com.invalid
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I don't think it really makes sense to to do this on Combine.
> > And I
> > > > > agree
> > > > > > > with you, it doesn't make sense on composites either.
> > > > > > >
> > > > > > > On Thu, Aug 10, 2017 at 9:19 AM, Scott Wegner
> > > > > <swegner@google.com.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Does requires-stable-input only apply to ParDo transforms?
> > > > > > > >
> > > > > > > > I don't think it would make sense to annotate to composite,
> > > because
> > > > > > > > checkpointing should happen as close to the side-effecting
> > > > operation
> > > > > as
> > > > > > > > possible, since upstream transforms within a composite could
> > > > > introduce
> > > > > > > > non-determinism. So it's the primitive transform that should
> > own
> > > > the
> > > > > > > > requirement.
> > > > > > > >
> > > > > > > > Are there other primitives that should be annotated?
> 'Combine'
> > is
> > > > > > > > interesting because it optimized in Dataflow (and perhaps
> other
> > > > > > runners)
> > > > > > > to
> > > > > > > > partially apply before a GroupByKey.
> > > > > > > >
> > > > > > > > On Thu, Aug 10, 2017 at 9:01 AM Tyler Akidau
> > > > > > <takidau@google.com.invalid
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 to the annotation idea, and to having it on
> processTimer.
> > > > > > > > >
> > > > > > > > > -Tyler
> > > > > > > > >
> > > > > > > > > On Thu, Aug 10, 2017 at 2:15 AM Aljoscha Krettek <
> > > > > > aljoscha@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 to the annotation approach. I outlined how
> implementing
> > > this
> > > > > > would
> > > > > > > > > work
> > > > > > > > > > in the Flink runner in the Thread about the exactly-once
> > > Kafka
> > > > > > Sink.
> > > > > > > > > >
> > > > > > > > > > > On 9. Aug 2017, at 23:03, Reuven Lax
> > > > <relax@google.com.INVALID
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Yes - I don't think we should try and make any
> > > deterministic
> > > > > > > > guarantees
> > > > > > > > > > > about what is in a bundle. Stability guarantees are per
> > > > element
> > > > > > > only.
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh
> > > > > > > <tgroh@google.com.invalid
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> +1 to the annotation-on-ProcessElement approach.
> > > > > ProcessElement
> > > > > > is
> > > > > > > > the
> > > > > > > > > > >> minimum implementation requirement of a DoFn, and
> should
> > > be
> > > > > > where
> > > > > > > > the
> > > > > > > > > > >> processing logic which depends on characteristics of
> the
> > > > > inputs
> > > > > > > lie.
> > > > > > > > > > It's a
> > > > > > > > > > >> good way of signalling the requirements of the Fn, and
> > > > letting
> > > > > > the
> > > > > > > > > > runner
> > > > > > > > > > >> decide.
> > > > > > > > > > >>
> > > > > > > > > > >> I have a minor concern that this may not work as
> > expected
> > > > for
> > > > > > > users
> > > > > > > > > that
> > > > > > > > > > >> try to batch remote calls in `FinishBundle` - we
> should
> > > make
> > > > > > sure
> > > > > > > we
> > > > > > > > > > >> document that it is explicitly the input elements that
> > > will
> > > > be
> > > > > > > > > replayed,
> > > > > > > > > > >> and bundles and other operational are still arbitrary.
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax
> > > > > > > > <relax@google.com.invalid
> > > > > > > > > >
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>> I think deterministic here means deterministically
> > > > > replayable.
> > > > > > > i.e.
> > > > > > > > > no
> > > > > > > > > > >>> matter how many times the element is retried, it will
> > > > always
> > > > > be
> > > > > > > the
> > > > > > > > > > same.
> > > > > > > > > > >>>
> > > > > > > > > > >>> I think we should also allow specifying this on
> > > > processTimer.
> > > > > > > This
> > > > > > > > > > would
> > > > > > > > > > >>> mean that any keyed state written in a previous
> > > > > processElement
> > > > > > > must
> > > > > > > > > be
> > > > > > > > > > >>> guaranteed durable before processTimer is called.
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <
> > > > > > > > bchambers@apache.org>
> > > > > > > > > > >>> wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> I strongly agree with this proposal. I think moving
> > away
> > > > > from
> > > > > > > > "just
> > > > > > > > > > >>> insert
> > > > > > > > > > >>>> a GroupByKey for one of the 3 different reasons you
> > may
> > > > want
> > > > > > it"
> > > > > > > > > > >> towards
> > > > > > > > > > >>>> APIs that allow code to express the requirements
> they
> > > have
> > > > > and
> > > > > > > the
> > > > > > > > > > >> runner
> > > > > > > > > > >>>> to choose the best way to meet this is a major step
> > > > forwards
> > > > > > in
> > > > > > > > > terms
> > > > > > > > > > >> of
> > > > > > > > > > >>>> portability.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> I think "deterministic" may be misleading. The
> actual
> > > > > contents
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > >>>> collection aren't deterministic if upstream
> > computations
> > > > > > aren't.
> > > > > > > > The
> > > > > > > > > > >>>> property we really need is that once an input may
> have
> > > > been
> > > > > > > > observed
> > > > > > > > > > by
> > > > > > > > > > >>> the
> > > > > > > > > > >>>> side-effecting code it will never be observed with a
> > > > > different
> > > > > > > > > value.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> I would propose something RequiresStableInput, to
> > > indicate
> > > > > > that
> > > > > > > > the
> > > > > > > > > > >> input
> > > > > > > > > > >>>> must be stable as observed by the function. I could
> > also
> > > > see
> > > > > > > > > something
> > > > > > > > > > >>>> hinting at the fact we don't recompute the input,
> such
> > > as
> > > > > > > > > > >>>> RequiresMemoizedInput or RequiresNoRecomputation.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> -- Ben
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> P.S For anyone interested other uses of GroupByKey
> > that
> > > we
> > > > > may
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > > >>>> discuss APIs for would be preventing retry across
> > steps
> > > > > (eg.,
> > > > > > > > > > >> preventing
> > > > > > > > > > >>>> fusion) and redistributing inputs across workers.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles
> > > > > > > > > <klk@google.com.invalid
> > > > > > > > > > >>>
> > > > > > > > > > >>>> wrote:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> This came up again, so I wanted to push it along by
> > > > > > proposing a
> > > > > > > > > > >>> specific
> > > > > > > > > > >>>>> API for Java that could have a derived API in
> > Python. I
> > > > am
> > > > > > > > writing
> > > > > > > > > > >> this
> > > > > > > > > > >>>>> quickly to get something out there, so I welcome
> > > > > suggestions
> > > > > > > for
> > > > > > > > > > >>>> revision.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Today a DoFn has a @ProcessElement annotated method
> > > with
> > > > > > > various
> > > > > > > > > > >>>> automated
> > > > > > > > > > >>>>> parameters, but most fundamentally this:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> @ProcessElement
> > > > > > > > > > >>>>> public void process(ProcessContext ctx) {
> > > > > > > > > > >>>>>  ctx.element() // to access the current input
> element
> > > > > > > > > > >>>>>  ctx.output(something) // to write to default
> output
> > > > > > collection
> > > > > > > > > > >>>>>  ctx.output(tag, something) // to write to other
> > output
> > > > > > > > collections
> > > > > > > > > > >>>>> }
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> For some time, we have hoped to unpack the context
> -
> > it
> > > > is
> > > > > a
> > > > > > > > > > >>>>> backwards-compatibility pattern made obsolete by
> the
> > > new
> > > > > DoFn
> > > > > > > > > design.
> > > > > > > > > > >>> So
> > > > > > > > > > >>>>> instead it would look like this:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> @ProcessElement
> > > > > > > > > > >>>>> public void process(Element element, MainOutput
> > > > mainOutput,
> > > > > > > ...)
> > > > > > > > {
> > > > > > > > > > >>>>>  element.get() // to access the current input
> element
> > > > > > > > > > >>>>>  mainOutput.output(something) // to write to the
> > > default
> > > > > > output
> > > > > > > > > > >>>> collection
> > > > > > > > > > >>>>>  other.output(something) // to write to other
> output
> > > > > > collection
> > > > > > > > > > >>>>> }
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> I've deliberately left out the undecided syntax for
> > > side
> > > > > > > outputs.
> > > > > > > > > But
> > > > > > > > > > >>> it
> > > > > > > > > > >>>>> would be nice for the tag to be built in to the
> > > parameter
> > > > > so
> > > > > > it
> > > > > > > > > > >> doesn't
> > > > > > > > > > >>>>> have to be used when calling output().
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> One way to enhance this to deterministic input
> would
> > > just
> > > > > be
> > > > > > > > this:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> @ProcessElement
> > > > > > > > > > >>>>> @RequiresDeterministicInput
> > > > > > > > > > >>>>> public void process(Element element, MainOutput
> > > > mainOutput,
> > > > > > > ...)
> > > > > > > > {
> > > > > > > > > > >>>>>  element.get() // to access the current input
> element
> > > > > > > > > > >>>>>  mainOutput.output(something) // to write to the
> > > default
> > > > > > output
> > > > > > > > > > >>>> collection
> > > > > > > > > > >>>>>  other.output(something) // to write to other
> output
> > > > > > collection
> > > > > > > > > > >>>>> }
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> There are really a lot of places where we could put
> > an
> > > > > > > annotation
> > > > > > > > > or
> > > > > > > > > > >>>> change
> > > > > > > > > > >>>>> a type to indicate that the input PCollection
> should
> > be
> > > > > > > > > > >>>>> well-defined/deterministically-replayable. I don't
> > > have
> > > > a
> > > > > > > really
> > > > > > > > > > >>> strong
> > > > > > > > > > >>>>> opinion.
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Kenn
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > > > > > > > > >>>> <bchambers@google.com.invalid
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>> wrote:
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>>> Allowing an annotation on DoFn's that produce
> > > > > deterministic
> > > > > > > > output
> > > > > > > > > > >>>> could
> > > > > > > > > > >>>>> be
> > > > > > > > > > >>>>>> added in the future, but doesn't seem like a great
> > > > option.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> 1. It is a correctness issue to assume a DoFn is
> > > > > > deterministic
> > > > > > > > and
> > > > > > > > > > >> be
> > > > > > > > > > >>>>>> wrong, so we would need to assume all transform
> > > outputs
> > > > > are
> > > > > > > > > > >>>>>> non-deterministic unless annotated. Getting this
> > > correct
> > > > > is
> > > > > > > > > > >> difficult
> > > > > > > > > > >>>>> (for
> > > > > > > > > > >>>>>> example, GBK is surprisingly non-deterministic
> > except
> > > in
> > > > > > > > specific
> > > > > > > > > > >>>> cases).
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> 2. It is unlikely to be a major performance
> > > improvement,
> > > > > > given
> > > > > > > > > that
> > > > > > > > > > >>> any
> > > > > > > > > > >>>>>> non-deterministic transform prior to a sink (which
> > are
> > > > > most
> > > > > > > > likely
> > > > > > > > > > >> to
> > > > > > > > > > >>>>>> require deterministic input) will cause additional
> > > work
> > > > to
> > > > > > be
> > > > > > > > > > >> needed.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> Based on this, it seems like the risk of allowing
> an
> > > > > > > annotation
> > > > > > > > is
> > > > > > > > > > >>> high
> > > > > > > > > > >>>>>> while the potential for performance improvements
> is
> > > low.
> > > > > The
> > > > > > > > > > >> current
> > > > > > > > > > >>>>>> proposal (not allowing an annotation) makes sense
> > for
> > > > now,
> > > > > > > until
> > > > > > > > > we
> > > > > > > > > > >>> can
> > > > > > > > > > >>>>>> demonstrate that the impact on performance is high
> > in
> > > > > cases
> > > > > > > that
> > > > > > > > > > >>> could
> > > > > > > > > > >>>> be
> > > > > > > > > > >>>>>> avoided with an annotation (in real-world use).
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> -- Ben
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <
> > > > > > vikky.rk@gmail.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> +1 for the general idea of runners handling it
> over
> > > > > > hard-coded
> > > > > > > > > > >>>>>> implementation strategy.
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> For the Write transform I believe you are talking
> > > about
> > > > > > > > > > >>>> ApplyShardingKey
> > > > > > > > > > >>>>>> <
> > > > > > > > > > >>>>>>
> > > > > > > > > https://github.com/apache/beam/blob/
> > > > d66029cafde152c0a46ebd276ddfa4
> > > > > > > > > > >>>>>> c3e7fd3433/sdks/java/core/src/
> > > > > > main/java/org/apache/beam/sdk/
> > > > > > > > > > >>>>>> io/Write.java#L304
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>> which
> > > > > > > > > > >>>>>> introduces non deterministic behavior when
> retried?
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> *Let a DoFn declare (mechanism not important right
> > > now)
> > > > > that
> > > > > > > it
> > > > > > > > > > >>>>>> "requiresdeterministic input"*
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> *Each runner will need a way to induce
> deterministic
> > > > > input -
> > > > > > > the
> > > > > > > > > > >>>>>> obviouschoice being a materialization.*
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> Does this mean that a runner will always
> materialize
> > > (or
> > > > > > > > whatever
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>> strategy is) an input PCollection to this DoFn
> even
> > > > though
> > > > > > the
> > > > > > > > > > >>>>> PCollection
> > > > > > > > > > >>>>>> might have been produced by deterministic
> > transforms?
> > > > > Would
> > > > > > it
> > > > > > > > > make
> > > > > > > > > > >>>> sense
> > > > > > > > > > >>>>>> to also let DoFns declare if they produce
> > > > > non-deterministic
> > > > > > > > > output?
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> -Vikas
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>> On 21 March 2017 at 13:52, Stephen Sisk
> > > > > > > <sisk@google.com.invalid
> > > > > > > > >
> > > > > > > > > > >>>> wrote:
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> Hey Kenn-
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> this seems important, but I don't have all the
> > > context
> > > > on
> > > > > > > what
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>> problem
> > > > > > > > > > >>>>>>> is.
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Can you explain this sentence "Specifically,
> there
> > is
> > > > > > > > > > >> pseudorandom
> > > > > > > > > > >>>> data
> > > > > > > > > > >>>>>>> generated and once it has been observed and used
> to
> > > > > > produce a
> > > > > > > > > > >> side
> > > > > > > > > > >>>>>> effect,
> > > > > > > > > > >>>>>>> it cannot be regenerated without erroneous
> > results."
> > > ?
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> Where is the pseudorandom data coming from?
> > Perhaps a
> > > > > > > concrete
> > > > > > > > > > >>>> example
> > > > > > > > > > >>>>>>> would help?
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> S
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > > > > > > > > >>>> <klk@google.com.invalid
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>>> wrote:
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>>> Problem:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> I will drop all nuance and say that the `Write`
> > > > > transform
> > > > > > as
> > > > > > > > it
> > > > > > > > > > >>>>> exists
> > > > > > > > > > >>>>>> in
> > > > > > > > > > >>>>>>>> the SDK is incorrect until we add some
> > specification
> > > > and
> > > > > > > APIs.
> > > > > > > > > > >> We
> > > > > > > > > > >>>>> can't
> > > > > > > > > > >>>>>>>> keep shipping an SDK with an unsafe transform in
> > it,
> > > > and
> > > > > > IMO
> > > > > > > > > > >> this
> > > > > > > > > > >>>>>>> certainly
> > > > > > > > > > >>>>>>>> blocks a stable release.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Specifically, there is pseudorandom data
> generated
> > > and
> > > > > > once
> > > > > > > it
> > > > > > > > > > >>> has
> > > > > > > > > > >>>>> been
> > > > > > > > > > >>>>>>>> observed and used to produce a side effect, it
> > > cannot
> > > > be
> > > > > > > > > > >>>> regenerated
> > > > > > > > > > >>>>>>>> without erroneous results.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> This generalizes: For some side-effecting
> > > user-defined
> > > > > > > > > > >> functions,
> > > > > > > > > > >>>> it
> > > > > > > > > > >>>>> is
> > > > > > > > > > >>>>>>>> vital that even across retries/replays they
> have a
> > > > > > > consistent
> > > > > > > > > > >>> view
> > > > > > > > > > >>>> of
> > > > > > > > > > >>>>>> the
> > > > > > > > > > >>>>>>>> contents of their input PCollection, because
> their
> > > > > effect
> > > > > > on
> > > > > > > > > > >> the
> > > > > > > > > > >>>>>> outside
> > > > > > > > > > >>>>>>>> world cannot be retracted if/when they fail and
> > are
> > > > > > retried.
> > > > > > > > > > >> Once
> > > > > > > > > > >>>> the
> > > > > > > > > > >>>>>>>> runner ensures a consistent view of the input,
> it
> > is
> > > > > then
> > > > > > > > their
> > > > > > > > > > >>> own
> > > > > > > > > > >>>>>>>> responsibility to be idempotent.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Ideally we should specify this requirement for
> the
> > > > > > > > user-defined
> > > > > > > > > > >>>>>> function
> > > > > > > > > > >>>>>>>> without imposing any particular implementation
> > > > strategy
> > > > > on
> > > > > > > > Beam
> > > > > > > > > > >>>>>> runners.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Proposal:
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> 1. Let a DoFn declare (mechanism not important
> > right
> > > > > now)
> > > > > > > that
> > > > > > > > > > >> it
> > > > > > > > > > >>>>>>> "requires
> > > > > > > > > > >>>>>>>> deterministic input".
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> 2. Each runner will need a way to induce
> > > deterministic
> > > > > > > input -
> > > > > > > > > > >>> the
> > > > > > > > > > >>>>>>> obvious
> > > > > > > > > > >>>>>>>> choice being a materialization.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> I want to keep the discussion focused, so I'm
> > > leaving
> > > > > out
> > > > > > > any
> > > > > > > > > > >>>>>>> possibilities
> > > > > > > > > > >>>>>>>> of taking this further.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Regarding performance: Today places that require
> > > this
> > > > > tend
> > > > > > > to
> > > > > > > > > > >> be
> > > > > > > > > > >>>>>> already
> > > > > > > > > > >>>>>>>> paying the cost via GroupByKey / Reshuffle
> > > operations,
> > > > > > since
> > > > > > > > > > >> that
> > > > > > > > > > >>>>> was a
> > > > > > > > > > >>>>>>>> simple way to induce determinism in batch
> > Dataflow*
> > > > > > (doesn't
> > > > > > > > > > >> work
> > > > > > > > > > >>>> for
> > > > > > > > > > >>>>>>> most
> > > > > > > > > > >>>>>>>> other runners nor for streaming Dataflow). This
> > > change
> > > > > > will
> > > > > > > > > > >>>> replace a
> > > > > > > > > > >>>>>>>> hard-coded implementation strategy with a
> > > requirement
> > > > > that
> > > > > > > may
> > > > > > > > > > >> be
> > > > > > > > > > >>>>>>> fulfilled
> > > > > > > > > > >>>>>>>> in the most efficient way available.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Thoughts?
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> Kenn (w/ lots of consult from colleagues,
> > especially
> > > > > Ben)
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>> * There is some overlap with the
> > > > reshuffle/redistribute
> > > > > > > > > > >>> discussion
> > > > > > > > > > >>>>>>> because
> > > > > > > > > > >>>>>>>> of this historical situation, but I would like
> to
> > > > leave
> > > > > > that
> > > > > > > > > > >>>> broader
> > > > > > > > > > >>>>>>>> discussion out of this correctness issue.
> > > > > > > > > > >>>>>>>>
> > > > > > > > > > >>>>>>>
> > > > > > > > > > >>>>>>
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Kenneth Knowles <kl...@google.com.INVALID>.
 > > >    - Does it also imply fixed length and content for value
iterators?
> > >

The concept of "value iterator" brings up a nit.

First, there is no such concept in the Beam model, and I don't think there
should be. I don't think we should special case GBK if we can avoid it. If
a PCollection contains elements of type KV<K, Iterable<V>> we should have
the same definition of "stable" whether or not it came from GBK. So I
think, for some definition of "fixed content", it should imply fixed length
and content.

But actually what is the definition of whether two Iterable<V> values have
the same content, since we don't require a deterministic coder for the
values? I think this question applies to this proposal in general.
Obviously we expect that if the runner is replaying from persisted
serialized data, then it suffices, but that is too operational for a good
definition IMO.

the KV<K,
> > > Iterable<V>> has no good way of being deterministic if there is late
> > data.
> > > We could do so by forcing the Iterable to be materialized into a single
> > > element, but that would also mean that the entire Iterable must fit in
> > > memory (which at least the Dataflow runner does not require).
>

As I mention above, the iterable is semantically [part of] a single
element. So just to unpack this, to make sure we are talking about the same
thing, I think you are talking about GBK as implemented via GBKO + GABW.

When the output of GABW is required to be stable but the output of GBKO is
not stable, we don't have stability for free in all cases by inserting a
GBK, but require something more to make the output of GABW stable, in the
worst case a full materialization.

In a perfect (future) world, if we were in accumulating mode (aka each new
element is a full replacement value) then we would know that the sink will
just overwrite prior values so it is safe to be unstable w.r.t. to
triggerings (but not other side effects). We could materialize at most the
pane index (aka sequence number for the write).

For discarding mode (aka each new element is incremental output) I think
you need full stability of the pane index / sequence number and its
associated element. There might be different ways to implement, such as the
ones suggested so far, depending on the runner.

Kenn

> >
> > >
> > > >    - Some examples to clarify nuances would be very useful.
> > > >
> > > > State durability semantics for timers that Reuven proposed seem to be
> > > > unrelated to stable input (at model level). It might be simpler to
> add
> > > > these semantics first. A lot of deterministic side-effects issues can
> > be
> > > > handled by durable state in timers. One thing I like about timers
> > > approach
> > > > is that it makes the cost more transparent to the user since the
> state
> > is
> > > > explicitly stored.
> > > >
> > > >
> > > > On Thu, Aug 10, 2017 at 10:02 AM, Ben Chambers
> > > > <bchambers@google.com.invalid
> > > > > wrote:
> > > >
> > > > > I think it only makes sense in places where a user might reasonable
> > > > require
> > > > > stable input to ensure idempotency of side-effects. It also only
> > makes
> > > > > sense in places where a runner could reasonably provide such a
> > > guarantee.
> > > > >
> > > > > A given Combine is unlikely to have side effects so it is less
> likely
> > > to
> > > > > benefit from stability of the input. Further, the reason writing a
> > > > Combine
> > > > > is desirable is because its execution can be split up and moved to
> > the
> > > > > mapper-side (before the GroupByKey). But this division is
> inherently
> > > > > non-deterministic, and so it seems unlikely to benefit from
> > stability.
> > > > And
> > > > > many cases where I could see wanting side-effects would end up in
> > > > > extractOutput, for which there is an easy (arguably better)
> solution
> > --
> > > > > have extractOutput return the accumulators and do the side-effects
> > in a
> > > > > DoFn afterwards.
> > > > >
> > > > > For composites, it is a bit trickier. I could see a case for
> > supporting
> > > > it
> > > > > on composites, but it would need to make it very clear that it only
> > > > > affected the input to the composite. If any of the operations
> within
> > > the
> > > > > composite were non-deterministic, then the outputs of that could be
> > > > > unstable, leading to instability in later parts of the composite.
> > > > Further,
> > > > > it doesn't seem to offer much. The composite itself doesn't perform
> > > > > side-effects, so there is no benefit to having the annotation there
> > --
> > > > > instead, we allow the annotation to be put where it is relevant and
> > > > > important -- on the DoFn's that actually have side-effects that
> > require
> > > > > stability.
> > > > >
> > > > > On Thu, Aug 10, 2017 at 9:23 AM Reuven Lax
> <relax@google.com.invalid
> > >
> > > > > wrote:
> > > > >
> > > > > > I don't think it really makes sense to to do this on Combine.
> And I
> > > > agree
> > > > > > with you, it doesn't make sense on composites either.
> > > > > >
> > > > > > On Thu, Aug 10, 2017 at 9:19 AM, Scott Wegner
> > > > <swegner@google.com.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Does requires-stable-input only apply to ParDo transforms?
> > > > > > >
> > > > > > > I don't think it would make sense to annotate to composite,
> > because
> > > > > > > checkpointing should happen as close to the side-effecting
> > > operation
> > > > as
> > > > > > > possible, since upstream transforms within a composite could
> > > > introduce
> > > > > > > non-determinism. So it's the primitive transform that should
> own
> > > the
> > > > > > > requirement.
> > > > > > >
> > > > > > > Are there other primitives that should be annotated? 'Combine'
> is
> > > > > > > interesting because it optimized in Dataflow (and perhaps other
> > > > > runners)
> > > > > > to
> > > > > > > partially apply before a GroupByKey.
> > > > > > >
> > > > > > > On Thu, Aug 10, 2017 at 9:01 AM Tyler Akidau
> > > > > <takidau@google.com.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 to the annotation idea, and to having it on processTimer.
> > > > > > > >
> > > > > > > > -Tyler
> > > > > > > >
> > > > > > > > On Thu, Aug 10, 2017 at 2:15 AM Aljoscha Krettek <
> > > > > aljoscha@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 to the annotation approach. I outlined how implementing
> > this
> > > > > would
> > > > > > > > work
> > > > > > > > > in the Flink runner in the Thread about the exactly-once
> > Kafka
> > > > > Sink.
> > > > > > > > >
> > > > > > > > > > On 9. Aug 2017, at 23:03, Reuven Lax
> > > <relax@google.com.INVALID
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Yes - I don't think we should try and make any
> > deterministic
> > > > > > > guarantees
> > > > > > > > > > about what is in a bundle. Stability guarantees are per
> > > element
> > > > > > only.
> > > > > > > > > >
> > > > > > > > > > On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh
> > > > > > <tgroh@google.com.invalid
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> +1 to the annotation-on-ProcessElement approach.
> > > > ProcessElement
> > > > > is
> > > > > > > the
> > > > > > > > > >> minimum implementation requirement of a DoFn, and should
> > be
> > > > > where
> > > > > > > the
> > > > > > > > > >> processing logic which depends on characteristics of the
> > > > inputs
> > > > > > lie.
> > > > > > > > > It's a
> > > > > > > > > >> good way of signalling the requirements of the Fn, and
> > > letting
> > > > > the
> > > > > > > > > runner
> > > > > > > > > >> decide.
> > > > > > > > > >>
> > > > > > > > > >> I have a minor concern that this may not work as
> expected
> > > for
> > > > > > users
> > > > > > > > that
> > > > > > > > > >> try to batch remote calls in `FinishBundle` - we should
> > make
> > > > > sure
> > > > > > we
> > > > > > > > > >> document that it is explicitly the input elements that
> > will
> > > be
> > > > > > > > replayed,
> > > > > > > > > >> and bundles and other operational are still arbitrary.
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax
> > > > > > > <relax@google.com.invalid
> > > > > > > > >
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >>> I think deterministic here means deterministically
> > > > replayable.
> > > > > > i.e.
> > > > > > > > no
> > > > > > > > > >>> matter how many times the element is retried, it will
> > > always
> > > > be
> > > > > > the
> > > > > > > > > same.
> > > > > > > > > >>>
> > > > > > > > > >>> I think we should also allow specifying this on
> > > processTimer.
> > > > > > This
> > > > > > > > > would
> > > > > > > > > >>> mean that any keyed state written in a previous
> > > > processElement
> > > > > > must
> > > > > > > > be
> > > > > > > > > >>> guaranteed durable before processTimer is called.
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <
> > > > > > > bchambers@apache.org>
> > > > > > > > > >>> wrote:
> > > > > > > > > >>>
> > > > > > > > > >>>> I strongly agree with this proposal. I think moving
> away
> > > > from
> > > > > > > "just
> > > > > > > > > >>> insert
> > > > > > > > > >>>> a GroupByKey for one of the 3 different reasons you
> may
> > > want
> > > > > it"
> > > > > > > > > >> towards
> > > > > > > > > >>>> APIs that allow code to express the requirements they
> > have
> > > > and
> > > > > > the
> > > > > > > > > >> runner
> > > > > > > > > >>>> to choose the best way to meet this is a major step
> > > forwards
> > > > > in
> > > > > > > > terms
> > > > > > > > > >> of
> > > > > > > > > >>>> portability.
> > > > > > > > > >>>>
> > > > > > > > > >>>> I think "deterministic" may be misleading. The actual
> > > > contents
> > > > > > of
> > > > > > > > the
> > > > > > > > > >>>> collection aren't deterministic if upstream
> computations
> > > > > aren't.
> > > > > > > The
> > > > > > > > > >>>> property we really need is that once an input may have
> > > been
> > > > > > > observed
> > > > > > > > > by
> > > > > > > > > >>> the
> > > > > > > > > >>>> side-effecting code it will never be observed with a
> > > > different
> > > > > > > > value.
> > > > > > > > > >>>>
> > > > > > > > > >>>> I would propose something RequiresStableInput, to
> > indicate
> > > > > that
> > > > > > > the
> > > > > > > > > >> input
> > > > > > > > > >>>> must be stable as observed by the function. I could
> also
> > > see
> > > > > > > > something
> > > > > > > > > >>>> hinting at the fact we don't recompute the input, such
> > as
> > > > > > > > > >>>> RequiresMemoizedInput or RequiresNoRecomputation.
> > > > > > > > > >>>>
> > > > > > > > > >>>> -- Ben
> > > > > > > > > >>>>
> > > > > > > > > >>>> P.S For anyone interested other uses of GroupByKey
> that
> > we
> > > > may
> > > > > > > want
> > > > > > > > to
> > > > > > > > > >>>> discuss APIs for would be preventing retry across
> steps
> > > > (eg.,
> > > > > > > > > >> preventing
> > > > > > > > > >>>> fusion) and redistributing inputs across workers.
> > > > > > > > > >>>>
> > > > > > > > > >>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles
> > > > > > > > <klk@google.com.invalid
> > > > > > > > > >>>
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> This came up again, so I wanted to push it along by
> > > > > proposing a
> > > > > > > > > >>> specific
> > > > > > > > > >>>>> API for Java that could have a derived API in
> Python. I
> > > am
> > > > > > > writing
> > > > > > > > > >> this
> > > > > > > > > >>>>> quickly to get something out there, so I welcome
> > > > suggestions
> > > > > > for
> > > > > > > > > >>>> revision.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Today a DoFn has a @ProcessElement annotated method
> > with
> > > > > > various
> > > > > > > > > >>>> automated
> > > > > > > > > >>>>> parameters, but most fundamentally this:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> @ProcessElement
> > > > > > > > > >>>>> public void process(ProcessContext ctx) {
> > > > > > > > > >>>>>  ctx.element() // to access the current input element
> > > > > > > > > >>>>>  ctx.output(something) // to write to default output
> > > > > collection
> > > > > > > > > >>>>>  ctx.output(tag, something) // to write to other
> output
> > > > > > > collections
> > > > > > > > > >>>>> }
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> For some time, we have hoped to unpack the context -
> it
> > > is
> > > > a
> > > > > > > > > >>>>> backwards-compatibility pattern made obsolete by the
> > new
> > > > DoFn
> > > > > > > > design.
> > > > > > > > > >>> So
> > > > > > > > > >>>>> instead it would look like this:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> @ProcessElement
> > > > > > > > > >>>>> public void process(Element element, MainOutput
> > > mainOutput,
> > > > > > ...)
> > > > > > > {
> > > > > > > > > >>>>>  element.get() // to access the current input element
> > > > > > > > > >>>>>  mainOutput.output(something) // to write to the
> > default
> > > > > output
> > > > > > > > > >>>> collection
> > > > > > > > > >>>>>  other.output(something) // to write to other output
> > > > > collection
> > > > > > > > > >>>>> }
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> I've deliberately left out the undecided syntax for
> > side
> > > > > > outputs.
> > > > > > > > But
> > > > > > > > > >>> it
> > > > > > > > > >>>>> would be nice for the tag to be built in to the
> > parameter
> > > > so
> > > > > it
> > > > > > > > > >> doesn't
> > > > > > > > > >>>>> have to be used when calling output().
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> One way to enhance this to deterministic input would
> > just
> > > > be
> > > > > > > this:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> @ProcessElement
> > > > > > > > > >>>>> @RequiresDeterministicInput
> > > > > > > > > >>>>> public void process(Element element, MainOutput
> > > mainOutput,
> > > > > > ...)
> > > > > > > {
> > > > > > > > > >>>>>  element.get() // to access the current input element
> > > > > > > > > >>>>>  mainOutput.output(something) // to write to the
> > default
> > > > > output
> > > > > > > > > >>>> collection
> > > > > > > > > >>>>>  other.output(something) // to write to other output
> > > > > collection
> > > > > > > > > >>>>> }
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> There are really a lot of places where we could put
> an
> > > > > > annotation
> > > > > > > > or
> > > > > > > > > >>>> change
> > > > > > > > > >>>>> a type to indicate that the input PCollection should
> be
> > > > > > > > > >>>>> well-defined/deterministically-replayable. I don't
> > have
> > > a
> > > > > > really
> > > > > > > > > >>> strong
> > > > > > > > > >>>>> opinion.
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Kenn
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > > > > > > > >>>> <bchambers@google.com.invalid
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>> wrote:
> > > > > > > > > >>>>>
> > > > > > > > > >>>>>> Allowing an annotation on DoFn's that produce
> > > > deterministic
> > > > > > > output
> > > > > > > > > >>>> could
> > > > > > > > > >>>>> be
> > > > > > > > > >>>>>> added in the future, but doesn't seem like a great
> > > option.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> 1. It is a correctness issue to assume a DoFn is
> > > > > deterministic
> > > > > > > and
> > > > > > > > > >> be
> > > > > > > > > >>>>>> wrong, so we would need to assume all transform
> > outputs
> > > > are
> > > > > > > > > >>>>>> non-deterministic unless annotated. Getting this
> > correct
> > > > is
> > > > > > > > > >> difficult
> > > > > > > > > >>>>> (for
> > > > > > > > > >>>>>> example, GBK is surprisingly non-deterministic
> except
> > in
> > > > > > > specific
> > > > > > > > > >>>> cases).
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> 2. It is unlikely to be a major performance
> > improvement,
> > > > > given
> > > > > > > > that
> > > > > > > > > >>> any
> > > > > > > > > >>>>>> non-deterministic transform prior to a sink (which
> are
> > > > most
> > > > > > > likely
> > > > > > > > > >> to
> > > > > > > > > >>>>>> require deterministic input) will cause additional
> > work
> > > to
> > > > > be
> > > > > > > > > >> needed.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Based on this, it seems like the risk of allowing an
> > > > > > annotation
> > > > > > > is
> > > > > > > > > >>> high
> > > > > > > > > >>>>>> while the potential for performance improvements is
> > low.
> > > > The
> > > > > > > > > >> current
> > > > > > > > > >>>>>> proposal (not allowing an annotation) makes sense
> for
> > > now,
> > > > > > until
> > > > > > > > we
> > > > > > > > > >>> can
> > > > > > > > > >>>>>> demonstrate that the impact on performance is high
> in
> > > > cases
> > > > > > that
> > > > > > > > > >>> could
> > > > > > > > > >>>> be
> > > > > > > > > >>>>>> avoided with an annotation (in real-world use).
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> -- Ben
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <
> > > > > vikky.rk@gmail.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> +1 for the general idea of runners handling it over
> > > > > hard-coded
> > > > > > > > > >>>>>> implementation strategy.
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> For the Write transform I believe you are talking
> > about
> > > > > > > > > >>>> ApplyShardingKey
> > > > > > > > > >>>>>> <
> > > > > > > > > >>>>>>
> > > > > > > > https://github.com/apache/beam/blob/
> > > d66029cafde152c0a46ebd276ddfa4
> > > > > > > > > >>>>>> c3e7fd3433/sdks/java/core/src/
> > > > > main/java/org/apache/beam/sdk/
> > > > > > > > > >>>>>> io/Write.java#L304
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>> which
> > > > > > > > > >>>>>> introduces non deterministic behavior when retried?
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> *Let a DoFn declare (mechanism not important right
> > now)
> > > > that
> > > > > > it
> > > > > > > > > >>>>>> "requiresdeterministic input"*
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> *Each runner will need a way to induce deterministic
> > > > input -
> > > > > > the
> > > > > > > > > >>>>>> obviouschoice being a materialization.*
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> Does this mean that a runner will always materialize
> > (or
> > > > > > > whatever
> > > > > > > > > >> the
> > > > > > > > > >>>>>> strategy is) an input PCollection to this DoFn even
> > > though
> > > > > the
> > > > > > > > > >>>>> PCollection
> > > > > > > > > >>>>>> might have been produced by deterministic
> transforms?
> > > > Would
> > > > > it
> > > > > > > > make
> > > > > > > > > >>>> sense
> > > > > > > > > >>>>>> to also let DoFns declare if they produce
> > > > non-deterministic
> > > > > > > > output?
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> -Vikas
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>> On 21 March 2017 at 13:52, Stephen Sisk
> > > > > > <sisk@google.com.invalid
> > > > > > > >
> > > > > > > > > >>>> wrote:
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> Hey Kenn-
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> this seems important, but I don't have all the
> > context
> > > on
> > > > > > what
> > > > > > > > > >> the
> > > > > > > > > >>>>>> problem
> > > > > > > > > >>>>>>> is.
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Can you explain this sentence "Specifically, there
> is
> > > > > > > > > >> pseudorandom
> > > > > > > > > >>>> data
> > > > > > > > > >>>>>>> generated and once it has been observed and used to
> > > > > produce a
> > > > > > > > > >> side
> > > > > > > > > >>>>>> effect,
> > > > > > > > > >>>>>>> it cannot be regenerated without erroneous
> results."
> > ?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> Where is the pseudorandom data coming from?
> Perhaps a
> > > > > > concrete
> > > > > > > > > >>>> example
> > > > > > > > > >>>>>>> would help?
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> S
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > > > > > > > >>>> <klk@google.com.invalid
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>>> wrote:
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>>> Problem:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I will drop all nuance and say that the `Write`
> > > > transform
> > > > > as
> > > > > > > it
> > > > > > > > > >>>>> exists
> > > > > > > > > >>>>>> in
> > > > > > > > > >>>>>>>> the SDK is incorrect until we add some
> specification
> > > and
> > > > > > APIs.
> > > > > > > > > >> We
> > > > > > > > > >>>>> can't
> > > > > > > > > >>>>>>>> keep shipping an SDK with an unsafe transform in
> it,
> > > and
> > > > > IMO
> > > > > > > > > >> this
> > > > > > > > > >>>>>>> certainly
> > > > > > > > > >>>>>>>> blocks a stable release.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Specifically, there is pseudorandom data generated
> > and
> > > > > once
> > > > > > it
> > > > > > > > > >>> has
> > > > > > > > > >>>>> been
> > > > > > > > > >>>>>>>> observed and used to produce a side effect, it
> > cannot
> > > be
> > > > > > > > > >>>> regenerated
> > > > > > > > > >>>>>>>> without erroneous results.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> This generalizes: For some side-effecting
> > user-defined
> > > > > > > > > >> functions,
> > > > > > > > > >>>> it
> > > > > > > > > >>>>> is
> > > > > > > > > >>>>>>>> vital that even across retries/replays they have a
> > > > > > consistent
> > > > > > > > > >>> view
> > > > > > > > > >>>> of
> > > > > > > > > >>>>>> the
> > > > > > > > > >>>>>>>> contents of their input PCollection, because their
> > > > effect
> > > > > on
> > > > > > > > > >> the
> > > > > > > > > >>>>>> outside
> > > > > > > > > >>>>>>>> world cannot be retracted if/when they fail and
> are
> > > > > retried.
> > > > > > > > > >> Once
> > > > > > > > > >>>> the
> > > > > > > > > >>>>>>>> runner ensures a consistent view of the input, it
> is
> > > > then
> > > > > > > their
> > > > > > > > > >>> own
> > > > > > > > > >>>>>>>> responsibility to be idempotent.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Ideally we should specify this requirement for the
> > > > > > > user-defined
> > > > > > > > > >>>>>> function
> > > > > > > > > >>>>>>>> without imposing any particular implementation
> > > strategy
> > > > on
> > > > > > > Beam
> > > > > > > > > >>>>>> runners.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Proposal:
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> 1. Let a DoFn declare (mechanism not important
> right
> > > > now)
> > > > > > that
> > > > > > > > > >> it
> > > > > > > > > >>>>>>> "requires
> > > > > > > > > >>>>>>>> deterministic input".
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> 2. Each runner will need a way to induce
> > deterministic
> > > > > > input -
> > > > > > > > > >>> the
> > > > > > > > > >>>>>>> obvious
> > > > > > > > > >>>>>>>> choice being a materialization.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> I want to keep the discussion focused, so I'm
> > leaving
> > > > out
> > > > > > any
> > > > > > > > > >>>>>>> possibilities
> > > > > > > > > >>>>>>>> of taking this further.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Regarding performance: Today places that require
> > this
> > > > tend
> > > > > > to
> > > > > > > > > >> be
> > > > > > > > > >>>>>> already
> > > > > > > > > >>>>>>>> paying the cost via GroupByKey / Reshuffle
> > operations,
> > > > > since
> > > > > > > > > >> that
> > > > > > > > > >>>>> was a
> > > > > > > > > >>>>>>>> simple way to induce determinism in batch
> Dataflow*
> > > > > (doesn't
> > > > > > > > > >> work
> > > > > > > > > >>>> for
> > > > > > > > > >>>>>>> most
> > > > > > > > > >>>>>>>> other runners nor for streaming Dataflow). This
> > change
> > > > > will
> > > > > > > > > >>>> replace a
> > > > > > > > > >>>>>>>> hard-coded implementation strategy with a
> > requirement
> > > > that
> > > > > > may
> > > > > > > > > >> be
> > > > > > > > > >>>>>>> fulfilled
> > > > > > > > > >>>>>>>> in the most efficient way available.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Thoughts?
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> Kenn (w/ lots of consult from colleagues,
> especially
> > > > Ben)
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>> * There is some overlap with the
> > > reshuffle/redistribute
> > > > > > > > > >>> discussion
> > > > > > > > > >>>>>>> because
> > > > > > > > > >>>>>>>> of this historical situation, but I would like to
> > > leave
> > > > > that
> > > > > > > > > >>>> broader
> > > > > > > > > >>>>>>>> discussion out of this correctness issue.
> > > > > > > > > >>>>>>>>
> > > > > > > > > >>>>>>>
> > > > > > > > > >>>>>>
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Reuven Lax <re...@google.com.INVALID>.
On Thu, Aug 10, 2017 at 11:18 AM, Thomas Groh <tg...@google.com.invalid>
wrote:

> I think it must imply fixed content <K, Iter<V>>s - making a decision based
> on the contents of an iterable assuming the Iterable is deterministic seems
> an acceptable use of the API, and that requires the contents to be
> identical through failures. This does imply that (assuming this is reading
> directly from the output of a GroupByKey) the elements will be
> materialized, grouped, materialized again, and then read back to ensure
> that elements are not added on to the end.
>

I think this is doable, but a bit more involved to implement. We could
expose a list-state primitive for the runner to use (as opposed to the bag
state that is used today). Then before we deliver the Iterable, we would
add and checkpoint a boundary marker to the list. This will allow us to
ensure that redeliveries always use the same Iterable, and would not
require us to materialize the entire Iterable. This sounds harder to
implement than the other uses though.


> I agree that there is no ordering guarantee for the sequence in which
> elements are processed.
>
> On Thu, Aug 10, 2017 at 11:03 AM, Reuven Lax <re...@google.com.invalid>
> wrote:
>
> > It means that single element replay is stable.
> >
> > On Thu, Aug 10, 2017 at 10:56 AM, Raghu Angadi
> <rangadi@google.com.invalid
> > >
> > wrote:
> >
> > > Can we define what exactly is meant by deterministic/stable/replayable
> > > etc?
> > >
> > >    - Does it imply a fixed order? If yes, it implies fixed order of
> > >    processElement() invocations, right? Are there any qualifiers
> (within
> > a
> > >    window+key etc)?
> > >
> >
> > No, no ordering guarantee.
> >
> >
> > >    - Does it also imply fixed length and content for value iterators?
> > >
> >
> > Good point. With our current runner api, it does not. the KV<K,
> > Iterable<V>> has no good way of being deterministic if there is late
> data.
> > We could do so by forcing the Iterable to be materialized into a single
> > element, but that would also mean that the entire Iterable must fit in
> > memory (which at least the Dataflow runner does not require).
> >
> >
> > >    - Some examples to clarify nuances would be very useful.
> > >
> > > State durability semantics for timers that Reuven proposed seem to be
> > > unrelated to stable input (at model level). It might be simpler to add
> > > these semantics first. A lot of deterministic side-effects issues can
> be
> > > handled by durable state in timers. One thing I like about timers
> > approach
> > > is that it makes the cost more transparent to the user since the state
> is
> > > explicitly stored.
> > >
> > >
> > > On Thu, Aug 10, 2017 at 10:02 AM, Ben Chambers
> > > <bchambers@google.com.invalid
> > > > wrote:
> > >
> > > > I think it only makes sense in places where a user might reasonable
> > > require
> > > > stable input to ensure idempotency of side-effects. It also only
> makes
> > > > sense in places where a runner could reasonably provide such a
> > guarantee.
> > > >
> > > > A given Combine is unlikely to have side effects so it is less likely
> > to
> > > > benefit from stability of the input. Further, the reason writing a
> > > Combine
> > > > is desirable is because its execution can be split up and moved to
> the
> > > > mapper-side (before the GroupByKey). But this division is inherently
> > > > non-deterministic, and so it seems unlikely to benefit from
> stability.
> > > And
> > > > many cases where I could see wanting side-effects would end up in
> > > > extractOutput, for which there is an easy (arguably better) solution
> --
> > > > have extractOutput return the accumulators and do the side-effects
> in a
> > > > DoFn afterwards.
> > > >
> > > > For composites, it is a bit trickier. I could see a case for
> supporting
> > > it
> > > > on composites, but it would need to make it very clear that it only
> > > > affected the input to the composite. If any of the operations within
> > the
> > > > composite were non-deterministic, then the outputs of that could be
> > > > unstable, leading to instability in later parts of the composite.
> > > Further,
> > > > it doesn't seem to offer much. The composite itself doesn't perform
> > > > side-effects, so there is no benefit to having the annotation there
> --
> > > > instead, we allow the annotation to be put where it is relevant and
> > > > important -- on the DoFn's that actually have side-effects that
> require
> > > > stability.
> > > >
> > > > On Thu, Aug 10, 2017 at 9:23 AM Reuven Lax <relax@google.com.invalid
> >
> > > > wrote:
> > > >
> > > > > I don't think it really makes sense to to do this on Combine. And I
> > > agree
> > > > > with you, it doesn't make sense on composites either.
> > > > >
> > > > > On Thu, Aug 10, 2017 at 9:19 AM, Scott Wegner
> > > <swegner@google.com.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Does requires-stable-input only apply to ParDo transforms?
> > > > > >
> > > > > > I don't think it would make sense to annotate to composite,
> because
> > > > > > checkpointing should happen as close to the side-effecting
> > operation
> > > as
> > > > > > possible, since upstream transforms within a composite could
> > > introduce
> > > > > > non-determinism. So it's the primitive transform that should own
> > the
> > > > > > requirement.
> > > > > >
> > > > > > Are there other primitives that should be annotated? 'Combine' is
> > > > > > interesting because it optimized in Dataflow (and perhaps other
> > > > runners)
> > > > > to
> > > > > > partially apply before a GroupByKey.
> > > > > >
> > > > > > On Thu, Aug 10, 2017 at 9:01 AM Tyler Akidau
> > > > <takidau@google.com.invalid
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1 to the annotation idea, and to having it on processTimer.
> > > > > > >
> > > > > > > -Tyler
> > > > > > >
> > > > > > > On Thu, Aug 10, 2017 at 2:15 AM Aljoscha Krettek <
> > > > aljoscha@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 to the annotation approach. I outlined how implementing
> this
> > > > would
> > > > > > > work
> > > > > > > > in the Flink runner in the Thread about the exactly-once
> Kafka
> > > > Sink.
> > > > > > > >
> > > > > > > > > On 9. Aug 2017, at 23:03, Reuven Lax
> > <relax@google.com.INVALID
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Yes - I don't think we should try and make any
> deterministic
> > > > > > guarantees
> > > > > > > > > about what is in a bundle. Stability guarantees are per
> > element
> > > > > only.
> > > > > > > > >
> > > > > > > > > On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh
> > > > > <tgroh@google.com.invalid
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> +1 to the annotation-on-ProcessElement approach.
> > > ProcessElement
> > > > is
> > > > > > the
> > > > > > > > >> minimum implementation requirement of a DoFn, and should
> be
> > > > where
> > > > > > the
> > > > > > > > >> processing logic which depends on characteristics of the
> > > inputs
> > > > > lie.
> > > > > > > > It's a
> > > > > > > > >> good way of signalling the requirements of the Fn, and
> > letting
> > > > the
> > > > > > > > runner
> > > > > > > > >> decide.
> > > > > > > > >>
> > > > > > > > >> I have a minor concern that this may not work as expected
> > for
> > > > > users
> > > > > > > that
> > > > > > > > >> try to batch remote calls in `FinishBundle` - we should
> make
> > > > sure
> > > > > we
> > > > > > > > >> document that it is explicitly the input elements that
> will
> > be
> > > > > > > replayed,
> > > > > > > > >> and bundles and other operational are still arbitrary.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax
> > > > > > <relax@google.com.invalid
> > > > > > > >
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >>> I think deterministic here means deterministically
> > > replayable.
> > > > > i.e.
> > > > > > > no
> > > > > > > > >>> matter how many times the element is retried, it will
> > always
> > > be
> > > > > the
> > > > > > > > same.
> > > > > > > > >>>
> > > > > > > > >>> I think we should also allow specifying this on
> > processTimer.
> > > > > This
> > > > > > > > would
> > > > > > > > >>> mean that any keyed state written in a previous
> > > processElement
> > > > > must
> > > > > > > be
> > > > > > > > >>> guaranteed durable before processTimer is called.
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <
> > > > > > bchambers@apache.org>
> > > > > > > > >>> wrote:
> > > > > > > > >>>
> > > > > > > > >>>> I strongly agree with this proposal. I think moving away
> > > from
> > > > > > "just
> > > > > > > > >>> insert
> > > > > > > > >>>> a GroupByKey for one of the 3 different reasons you may
> > want
> > > > it"
> > > > > > > > >> towards
> > > > > > > > >>>> APIs that allow code to express the requirements they
> have
> > > and
> > > > > the
> > > > > > > > >> runner
> > > > > > > > >>>> to choose the best way to meet this is a major step
> > forwards
> > > > in
> > > > > > > terms
> > > > > > > > >> of
> > > > > > > > >>>> portability.
> > > > > > > > >>>>
> > > > > > > > >>>> I think "deterministic" may be misleading. The actual
> > > contents
> > > > > of
> > > > > > > the
> > > > > > > > >>>> collection aren't deterministic if upstream computations
> > > > aren't.
> > > > > > The
> > > > > > > > >>>> property we really need is that once an input may have
> > been
> > > > > > observed
> > > > > > > > by
> > > > > > > > >>> the
> > > > > > > > >>>> side-effecting code it will never be observed with a
> > > different
> > > > > > > value.
> > > > > > > > >>>>
> > > > > > > > >>>> I would propose something RequiresStableInput, to
> indicate
> > > > that
> > > > > > the
> > > > > > > > >> input
> > > > > > > > >>>> must be stable as observed by the function. I could also
> > see
> > > > > > > something
> > > > > > > > >>>> hinting at the fact we don't recompute the input, such
> as
> > > > > > > > >>>> RequiresMemoizedInput or RequiresNoRecomputation.
> > > > > > > > >>>>
> > > > > > > > >>>> -- Ben
> > > > > > > > >>>>
> > > > > > > > >>>> P.S For anyone interested other uses of GroupByKey that
> we
> > > may
> > > > > > want
> > > > > > > to
> > > > > > > > >>>> discuss APIs for would be preventing retry across steps
> > > (eg.,
> > > > > > > > >> preventing
> > > > > > > > >>>> fusion) and redistributing inputs across workers.
> > > > > > > > >>>>
> > > > > > > > >>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles
> > > > > > > <klk@google.com.invalid
> > > > > > > > >>>
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> This came up again, so I wanted to push it along by
> > > > proposing a
> > > > > > > > >>> specific
> > > > > > > > >>>>> API for Java that could have a derived API in Python. I
> > am
> > > > > > writing
> > > > > > > > >> this
> > > > > > > > >>>>> quickly to get something out there, so I welcome
> > > suggestions
> > > > > for
> > > > > > > > >>>> revision.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Today a DoFn has a @ProcessElement annotated method
> with
> > > > > various
> > > > > > > > >>>> automated
> > > > > > > > >>>>> parameters, but most fundamentally this:
> > > > > > > > >>>>>
> > > > > > > > >>>>> @ProcessElement
> > > > > > > > >>>>> public void process(ProcessContext ctx) {
> > > > > > > > >>>>>  ctx.element() // to access the current input element
> > > > > > > > >>>>>  ctx.output(something) // to write to default output
> > > > collection
> > > > > > > > >>>>>  ctx.output(tag, something) // to write to other output
> > > > > > collections
> > > > > > > > >>>>> }
> > > > > > > > >>>>>
> > > > > > > > >>>>> For some time, we have hoped to unpack the context - it
> > is
> > > a
> > > > > > > > >>>>> backwards-compatibility pattern made obsolete by the
> new
> > > DoFn
> > > > > > > design.
> > > > > > > > >>> So
> > > > > > > > >>>>> instead it would look like this:
> > > > > > > > >>>>>
> > > > > > > > >>>>> @ProcessElement
> > > > > > > > >>>>> public void process(Element element, MainOutput
> > mainOutput,
> > > > > ...)
> > > > > > {
> > > > > > > > >>>>>  element.get() // to access the current input element
> > > > > > > > >>>>>  mainOutput.output(something) // to write to the
> default
> > > > output
> > > > > > > > >>>> collection
> > > > > > > > >>>>>  other.output(something) // to write to other output
> > > > collection
> > > > > > > > >>>>> }
> > > > > > > > >>>>>
> > > > > > > > >>>>> I've deliberately left out the undecided syntax for
> side
> > > > > outputs.
> > > > > > > But
> > > > > > > > >>> it
> > > > > > > > >>>>> would be nice for the tag to be built in to the
> parameter
> > > so
> > > > it
> > > > > > > > >> doesn't
> > > > > > > > >>>>> have to be used when calling output().
> > > > > > > > >>>>>
> > > > > > > > >>>>> One way to enhance this to deterministic input would
> just
> > > be
> > > > > > this:
> > > > > > > > >>>>>
> > > > > > > > >>>>> @ProcessElement
> > > > > > > > >>>>> @RequiresDeterministicInput
> > > > > > > > >>>>> public void process(Element element, MainOutput
> > mainOutput,
> > > > > ...)
> > > > > > {
> > > > > > > > >>>>>  element.get() // to access the current input element
> > > > > > > > >>>>>  mainOutput.output(something) // to write to the
> default
> > > > output
> > > > > > > > >>>> collection
> > > > > > > > >>>>>  other.output(something) // to write to other output
> > > > collection
> > > > > > > > >>>>> }
> > > > > > > > >>>>>
> > > > > > > > >>>>> There are really a lot of places where we could put an
> > > > > annotation
> > > > > > > or
> > > > > > > > >>>> change
> > > > > > > > >>>>> a type to indicate that the input PCollection should be
> > > > > > > > >>>>> well-defined/deterministically-replayable. I don't
> have
> > a
> > > > > really
> > > > > > > > >>> strong
> > > > > > > > >>>>> opinion.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Kenn
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > > > > > > >>>> <bchambers@google.com.invalid
> > > > > > > > >>>>>>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>> Allowing an annotation on DoFn's that produce
> > > deterministic
> > > > > > output
> > > > > > > > >>>> could
> > > > > > > > >>>>> be
> > > > > > > > >>>>>> added in the future, but doesn't seem like a great
> > option.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> 1. It is a correctness issue to assume a DoFn is
> > > > deterministic
> > > > > > and
> > > > > > > > >> be
> > > > > > > > >>>>>> wrong, so we would need to assume all transform
> outputs
> > > are
> > > > > > > > >>>>>> non-deterministic unless annotated. Getting this
> correct
> > > is
> > > > > > > > >> difficult
> > > > > > > > >>>>> (for
> > > > > > > > >>>>>> example, GBK is surprisingly non-deterministic except
> in
> > > > > > specific
> > > > > > > > >>>> cases).
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> 2. It is unlikely to be a major performance
> improvement,
> > > > given
> > > > > > > that
> > > > > > > > >>> any
> > > > > > > > >>>>>> non-deterministic transform prior to a sink (which are
> > > most
> > > > > > likely
> > > > > > > > >> to
> > > > > > > > >>>>>> require deterministic input) will cause additional
> work
> > to
> > > > be
> > > > > > > > >> needed.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Based on this, it seems like the risk of allowing an
> > > > > annotation
> > > > > > is
> > > > > > > > >>> high
> > > > > > > > >>>>>> while the potential for performance improvements is
> low.
> > > The
> > > > > > > > >> current
> > > > > > > > >>>>>> proposal (not allowing an annotation) makes sense for
> > now,
> > > > > until
> > > > > > > we
> > > > > > > > >>> can
> > > > > > > > >>>>>> demonstrate that the impact on performance is high in
> > > cases
> > > > > that
> > > > > > > > >>> could
> > > > > > > > >>>> be
> > > > > > > > >>>>>> avoided with an annotation (in real-world use).
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> -- Ben
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <
> > > > vikky.rk@gmail.com>
> > > > > > > > >> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> +1 for the general idea of runners handling it over
> > > > hard-coded
> > > > > > > > >>>>>> implementation strategy.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> For the Write transform I believe you are talking
> about
> > > > > > > > >>>> ApplyShardingKey
> > > > > > > > >>>>>> <
> > > > > > > > >>>>>>
> > > > > > > https://github.com/apache/beam/blob/
> > d66029cafde152c0a46ebd276ddfa4
> > > > > > > > >>>>>> c3e7fd3433/sdks/java/core/src/
> > > > main/java/org/apache/beam/sdk/
> > > > > > > > >>>>>> io/Write.java#L304
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>> which
> > > > > > > > >>>>>> introduces non deterministic behavior when retried?
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> *Let a DoFn declare (mechanism not important right
> now)
> > > that
> > > > > it
> > > > > > > > >>>>>> "requiresdeterministic input"*
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> *Each runner will need a way to induce deterministic
> > > input -
> > > > > the
> > > > > > > > >>>>>> obviouschoice being a materialization.*
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Does this mean that a runner will always materialize
> (or
> > > > > > whatever
> > > > > > > > >> the
> > > > > > > > >>>>>> strategy is) an input PCollection to this DoFn even
> > though
> > > > the
> > > > > > > > >>>>> PCollection
> > > > > > > > >>>>>> might have been produced by deterministic transforms?
> > > Would
> > > > it
> > > > > > > make
> > > > > > > > >>>> sense
> > > > > > > > >>>>>> to also let DoFns declare if they produce
> > > non-deterministic
> > > > > > > output?
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> -Vikas
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On 21 March 2017 at 13:52, Stephen Sisk
> > > > > <sisk@google.com.invalid
> > > > > > >
> > > > > > > > >>>> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Hey Kenn-
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> this seems important, but I don't have all the
> context
> > on
> > > > > what
> > > > > > > > >> the
> > > > > > > > >>>>>> problem
> > > > > > > > >>>>>>> is.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Can you explain this sentence "Specifically, there is
> > > > > > > > >> pseudorandom
> > > > > > > > >>>> data
> > > > > > > > >>>>>>> generated and once it has been observed and used to
> > > > produce a
> > > > > > > > >> side
> > > > > > > > >>>>>> effect,
> > > > > > > > >>>>>>> it cannot be regenerated without erroneous results."
> ?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Where is the pseudorandom data coming from? Perhaps a
> > > > > concrete
> > > > > > > > >>>> example
> > > > > > > > >>>>>>> would help?
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> S
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > > > > > > >>>> <klk@google.com.invalid
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>>> Problem:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> I will drop all nuance and say that the `Write`
> > > transform
> > > > as
> > > > > > it
> > > > > > > > >>>>> exists
> > > > > > > > >>>>>> in
> > > > > > > > >>>>>>>> the SDK is incorrect until we add some specification
> > and
> > > > > APIs.
> > > > > > > > >> We
> > > > > > > > >>>>> can't
> > > > > > > > >>>>>>>> keep shipping an SDK with an unsafe transform in it,
> > and
> > > > IMO
> > > > > > > > >> this
> > > > > > > > >>>>>>> certainly
> > > > > > > > >>>>>>>> blocks a stable release.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Specifically, there is pseudorandom data generated
> and
> > > > once
> > > > > it
> > > > > > > > >>> has
> > > > > > > > >>>>> been
> > > > > > > > >>>>>>>> observed and used to produce a side effect, it
> cannot
> > be
> > > > > > > > >>>> regenerated
> > > > > > > > >>>>>>>> without erroneous results.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> This generalizes: For some side-effecting
> user-defined
> > > > > > > > >> functions,
> > > > > > > > >>>> it
> > > > > > > > >>>>> is
> > > > > > > > >>>>>>>> vital that even across retries/replays they have a
> > > > > consistent
> > > > > > > > >>> view
> > > > > > > > >>>> of
> > > > > > > > >>>>>> the
> > > > > > > > >>>>>>>> contents of their input PCollection, because their
> > > effect
> > > > on
> > > > > > > > >> the
> > > > > > > > >>>>>> outside
> > > > > > > > >>>>>>>> world cannot be retracted if/when they fail and are
> > > > retried.
> > > > > > > > >> Once
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>> runner ensures a consistent view of the input, it is
> > > then
> > > > > > their
> > > > > > > > >>> own
> > > > > > > > >>>>>>>> responsibility to be idempotent.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Ideally we should specify this requirement for the
> > > > > > user-defined
> > > > > > > > >>>>>> function
> > > > > > > > >>>>>>>> without imposing any particular implementation
> > strategy
> > > on
> > > > > > Beam
> > > > > > > > >>>>>> runners.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Proposal:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> 1. Let a DoFn declare (mechanism not important right
> > > now)
> > > > > that
> > > > > > > > >> it
> > > > > > > > >>>>>>> "requires
> > > > > > > > >>>>>>>> deterministic input".
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> 2. Each runner will need a way to induce
> deterministic
> > > > > input -
> > > > > > > > >>> the
> > > > > > > > >>>>>>> obvious
> > > > > > > > >>>>>>>> choice being a materialization.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> I want to keep the discussion focused, so I'm
> leaving
> > > out
> > > > > any
> > > > > > > > >>>>>>> possibilities
> > > > > > > > >>>>>>>> of taking this further.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Regarding performance: Today places that require
> this
> > > tend
> > > > > to
> > > > > > > > >> be
> > > > > > > > >>>>>> already
> > > > > > > > >>>>>>>> paying the cost via GroupByKey / Reshuffle
> operations,
> > > > since
> > > > > > > > >> that
> > > > > > > > >>>>> was a
> > > > > > > > >>>>>>>> simple way to induce determinism in batch Dataflow*
> > > > (doesn't
> > > > > > > > >> work
> > > > > > > > >>>> for
> > > > > > > > >>>>>>> most
> > > > > > > > >>>>>>>> other runners nor for streaming Dataflow). This
> change
> > > > will
> > > > > > > > >>>> replace a
> > > > > > > > >>>>>>>> hard-coded implementation strategy with a
> requirement
> > > that
> > > > > may
> > > > > > > > >> be
> > > > > > > > >>>>>>> fulfilled
> > > > > > > > >>>>>>>> in the most efficient way available.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Thoughts?
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Kenn (w/ lots of consult from colleagues, especially
> > > Ben)
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> * There is some overlap with the
> > reshuffle/redistribute
> > > > > > > > >>> discussion
> > > > > > > > >>>>>>> because
> > > > > > > > >>>>>>>> of this historical situation, but I would like to
> > leave
> > > > that
> > > > > > > > >>>> broader
> > > > > > > > >>>>>>>> discussion out of this correctness issue.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Thomas Groh <tg...@google.com.INVALID>.
I think it must imply fixed content <K, Iter<V>>s - making a decision based
on the contents of an iterable assuming the Iterable is deterministic seems
an acceptable use of the API, and that requires the contents to be
identical through failures. This does imply that (assuming this is reading
directly from the output of a GroupByKey) the elements will be
materialized, grouped, materialized again, and then read back to ensure
that elements are not added on to the end.

I agree that there is no ordering guarantee for the sequence in which
elements are processed.

On Thu, Aug 10, 2017 at 11:03 AM, Reuven Lax <re...@google.com.invalid>
wrote:

> It means that single element replay is stable.
>
> On Thu, Aug 10, 2017 at 10:56 AM, Raghu Angadi <rangadi@google.com.invalid
> >
> wrote:
>
> > Can we define what exactly is meant by deterministic/stable/replayable
> > etc?
> >
> >    - Does it imply a fixed order? If yes, it implies fixed order of
> >    processElement() invocations, right? Are there any qualifiers (within
> a
> >    window+key etc)?
> >
>
> No, no ordering guarantee.
>
>
> >    - Does it also imply fixed length and content for value iterators?
> >
>
> Good point. With our current runner api, it does not. the KV<K,
> Iterable<V>> has no good way of being deterministic if there is late data.
> We could do so by forcing the Iterable to be materialized into a single
> element, but that would also mean that the entire Iterable must fit in
> memory (which at least the Dataflow runner does not require).
>
>
> >    - Some examples to clarify nuances would be very useful.
> >
> > State durability semantics for timers that Reuven proposed seem to be
> > unrelated to stable input (at model level). It might be simpler to add
> > these semantics first. A lot of deterministic side-effects issues can be
> > handled by durable state in timers. One thing I like about timers
> approach
> > is that it makes the cost more transparent to the user since the state is
> > explicitly stored.
> >
> >
> > On Thu, Aug 10, 2017 at 10:02 AM, Ben Chambers
> > <bchambers@google.com.invalid
> > > wrote:
> >
> > > I think it only makes sense in places where a user might reasonable
> > require
> > > stable input to ensure idempotency of side-effects. It also only makes
> > > sense in places where a runner could reasonably provide such a
> guarantee.
> > >
> > > A given Combine is unlikely to have side effects so it is less likely
> to
> > > benefit from stability of the input. Further, the reason writing a
> > Combine
> > > is desirable is because its execution can be split up and moved to the
> > > mapper-side (before the GroupByKey). But this division is inherently
> > > non-deterministic, and so it seems unlikely to benefit from stability.
> > And
> > > many cases where I could see wanting side-effects would end up in
> > > extractOutput, for which there is an easy (arguably better) solution --
> > > have extractOutput return the accumulators and do the side-effects in a
> > > DoFn afterwards.
> > >
> > > For composites, it is a bit trickier. I could see a case for supporting
> > it
> > > on composites, but it would need to make it very clear that it only
> > > affected the input to the composite. If any of the operations within
> the
> > > composite were non-deterministic, then the outputs of that could be
> > > unstable, leading to instability in later parts of the composite.
> > Further,
> > > it doesn't seem to offer much. The composite itself doesn't perform
> > > side-effects, so there is no benefit to having the annotation there --
> > > instead, we allow the annotation to be put where it is relevant and
> > > important -- on the DoFn's that actually have side-effects that require
> > > stability.
> > >
> > > On Thu, Aug 10, 2017 at 9:23 AM Reuven Lax <re...@google.com.invalid>
> > > wrote:
> > >
> > > > I don't think it really makes sense to to do this on Combine. And I
> > agree
> > > > with you, it doesn't make sense on composites either.
> > > >
> > > > On Thu, Aug 10, 2017 at 9:19 AM, Scott Wegner
> > <swegner@google.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Does requires-stable-input only apply to ParDo transforms?
> > > > >
> > > > > I don't think it would make sense to annotate to composite, because
> > > > > checkpointing should happen as close to the side-effecting
> operation
> > as
> > > > > possible, since upstream transforms within a composite could
> > introduce
> > > > > non-determinism. So it's the primitive transform that should own
> the
> > > > > requirement.
> > > > >
> > > > > Are there other primitives that should be annotated? 'Combine' is
> > > > > interesting because it optimized in Dataflow (and perhaps other
> > > runners)
> > > > to
> > > > > partially apply before a GroupByKey.
> > > > >
> > > > > On Thu, Aug 10, 2017 at 9:01 AM Tyler Akidau
> > > <takidau@google.com.invalid
> > > > >
> > > > > wrote:
> > > > >
> > > > > > +1 to the annotation idea, and to having it on processTimer.
> > > > > >
> > > > > > -Tyler
> > > > > >
> > > > > > On Thu, Aug 10, 2017 at 2:15 AM Aljoscha Krettek <
> > > aljoscha@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 to the annotation approach. I outlined how implementing this
> > > would
> > > > > > work
> > > > > > > in the Flink runner in the Thread about the exactly-once Kafka
> > > Sink.
> > > > > > >
> > > > > > > > On 9. Aug 2017, at 23:03, Reuven Lax
> <relax@google.com.INVALID
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > Yes - I don't think we should try and make any deterministic
> > > > > guarantees
> > > > > > > > about what is in a bundle. Stability guarantees are per
> element
> > > > only.
> > > > > > > >
> > > > > > > > On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh
> > > > <tgroh@google.com.invalid
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> +1 to the annotation-on-ProcessElement approach.
> > ProcessElement
> > > is
> > > > > the
> > > > > > > >> minimum implementation requirement of a DoFn, and should be
> > > where
> > > > > the
> > > > > > > >> processing logic which depends on characteristics of the
> > inputs
> > > > lie.
> > > > > > > It's a
> > > > > > > >> good way of signalling the requirements of the Fn, and
> letting
> > > the
> > > > > > > runner
> > > > > > > >> decide.
> > > > > > > >>
> > > > > > > >> I have a minor concern that this may not work as expected
> for
> > > > users
> > > > > > that
> > > > > > > >> try to batch remote calls in `FinishBundle` - we should make
> > > sure
> > > > we
> > > > > > > >> document that it is explicitly the input elements that will
> be
> > > > > > replayed,
> > > > > > > >> and bundles and other operational are still arbitrary.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax
> > > > > <relax@google.com.invalid
> > > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> I think deterministic here means deterministically
> > replayable.
> > > > i.e.
> > > > > > no
> > > > > > > >>> matter how many times the element is retried, it will
> always
> > be
> > > > the
> > > > > > > same.
> > > > > > > >>>
> > > > > > > >>> I think we should also allow specifying this on
> processTimer.
> > > > This
> > > > > > > would
> > > > > > > >>> mean that any keyed state written in a previous
> > processElement
> > > > must
> > > > > > be
> > > > > > > >>> guaranteed durable before processTimer is called.
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <
> > > > > bchambers@apache.org>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>>> I strongly agree with this proposal. I think moving away
> > from
> > > > > "just
> > > > > > > >>> insert
> > > > > > > >>>> a GroupByKey for one of the 3 different reasons you may
> want
> > > it"
> > > > > > > >> towards
> > > > > > > >>>> APIs that allow code to express the requirements they have
> > and
> > > > the
> > > > > > > >> runner
> > > > > > > >>>> to choose the best way to meet this is a major step
> forwards
> > > in
> > > > > > terms
> > > > > > > >> of
> > > > > > > >>>> portability.
> > > > > > > >>>>
> > > > > > > >>>> I think "deterministic" may be misleading. The actual
> > contents
> > > > of
> > > > > > the
> > > > > > > >>>> collection aren't deterministic if upstream computations
> > > aren't.
> > > > > The
> > > > > > > >>>> property we really need is that once an input may have
> been
> > > > > observed
> > > > > > > by
> > > > > > > >>> the
> > > > > > > >>>> side-effecting code it will never be observed with a
> > different
> > > > > > value.
> > > > > > > >>>>
> > > > > > > >>>> I would propose something RequiresStableInput, to indicate
> > > that
> > > > > the
> > > > > > > >> input
> > > > > > > >>>> must be stable as observed by the function. I could also
> see
> > > > > > something
> > > > > > > >>>> hinting at the fact we don't recompute the input, such as
> > > > > > > >>>> RequiresMemoizedInput or RequiresNoRecomputation.
> > > > > > > >>>>
> > > > > > > >>>> -- Ben
> > > > > > > >>>>
> > > > > > > >>>> P.S For anyone interested other uses of GroupByKey that we
> > may
> > > > > want
> > > > > > to
> > > > > > > >>>> discuss APIs for would be preventing retry across steps
> > (eg.,
> > > > > > > >> preventing
> > > > > > > >>>> fusion) and redistributing inputs across workers.
> > > > > > > >>>>
> > > > > > > >>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles
> > > > > > <klk@google.com.invalid
> > > > > > > >>>
> > > > > > > >>>> wrote:
> > > > > > > >>>>
> > > > > > > >>>>> This came up again, so I wanted to push it along by
> > > proposing a
> > > > > > > >>> specific
> > > > > > > >>>>> API for Java that could have a derived API in Python. I
> am
> > > > > writing
> > > > > > > >> this
> > > > > > > >>>>> quickly to get something out there, so I welcome
> > suggestions
> > > > for
> > > > > > > >>>> revision.
> > > > > > > >>>>>
> > > > > > > >>>>> Today a DoFn has a @ProcessElement annotated method with
> > > > various
> > > > > > > >>>> automated
> > > > > > > >>>>> parameters, but most fundamentally this:
> > > > > > > >>>>>
> > > > > > > >>>>> @ProcessElement
> > > > > > > >>>>> public void process(ProcessContext ctx) {
> > > > > > > >>>>>  ctx.element() // to access the current input element
> > > > > > > >>>>>  ctx.output(something) // to write to default output
> > > collection
> > > > > > > >>>>>  ctx.output(tag, something) // to write to other output
> > > > > collections
> > > > > > > >>>>> }
> > > > > > > >>>>>
> > > > > > > >>>>> For some time, we have hoped to unpack the context - it
> is
> > a
> > > > > > > >>>>> backwards-compatibility pattern made obsolete by the new
> > DoFn
> > > > > > design.
> > > > > > > >>> So
> > > > > > > >>>>> instead it would look like this:
> > > > > > > >>>>>
> > > > > > > >>>>> @ProcessElement
> > > > > > > >>>>> public void process(Element element, MainOutput
> mainOutput,
> > > > ...)
> > > > > {
> > > > > > > >>>>>  element.get() // to access the current input element
> > > > > > > >>>>>  mainOutput.output(something) // to write to the default
> > > output
> > > > > > > >>>> collection
> > > > > > > >>>>>  other.output(something) // to write to other output
> > > collection
> > > > > > > >>>>> }
> > > > > > > >>>>>
> > > > > > > >>>>> I've deliberately left out the undecided syntax for side
> > > > outputs.
> > > > > > But
> > > > > > > >>> it
> > > > > > > >>>>> would be nice for the tag to be built in to the parameter
> > so
> > > it
> > > > > > > >> doesn't
> > > > > > > >>>>> have to be used when calling output().
> > > > > > > >>>>>
> > > > > > > >>>>> One way to enhance this to deterministic input would just
> > be
> > > > > this:
> > > > > > > >>>>>
> > > > > > > >>>>> @ProcessElement
> > > > > > > >>>>> @RequiresDeterministicInput
> > > > > > > >>>>> public void process(Element element, MainOutput
> mainOutput,
> > > > ...)
> > > > > {
> > > > > > > >>>>>  element.get() // to access the current input element
> > > > > > > >>>>>  mainOutput.output(something) // to write to the default
> > > output
> > > > > > > >>>> collection
> > > > > > > >>>>>  other.output(something) // to write to other output
> > > collection
> > > > > > > >>>>> }
> > > > > > > >>>>>
> > > > > > > >>>>> There are really a lot of places where we could put an
> > > > annotation
> > > > > > or
> > > > > > > >>>> change
> > > > > > > >>>>> a type to indicate that the input PCollection should be
> > > > > > > >>>>> well-defined/deterministically-replayable. I don't have
> a
> > > > really
> > > > > > > >>> strong
> > > > > > > >>>>> opinion.
> > > > > > > >>>>>
> > > > > > > >>>>> Kenn
> > > > > > > >>>>>
> > > > > > > >>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > > > > > >>>> <bchambers@google.com.invalid
> > > > > > > >>>>>>
> > > > > > > >>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>>> Allowing an annotation on DoFn's that produce
> > deterministic
> > > > > output
> > > > > > > >>>> could
> > > > > > > >>>>> be
> > > > > > > >>>>>> added in the future, but doesn't seem like a great
> option.
> > > > > > > >>>>>>
> > > > > > > >>>>>> 1. It is a correctness issue to assume a DoFn is
> > > deterministic
> > > > > and
> > > > > > > >> be
> > > > > > > >>>>>> wrong, so we would need to assume all transform outputs
> > are
> > > > > > > >>>>>> non-deterministic unless annotated. Getting this correct
> > is
> > > > > > > >> difficult
> > > > > > > >>>>> (for
> > > > > > > >>>>>> example, GBK is surprisingly non-deterministic except in
> > > > > specific
> > > > > > > >>>> cases).
> > > > > > > >>>>>>
> > > > > > > >>>>>> 2. It is unlikely to be a major performance improvement,
> > > given
> > > > > > that
> > > > > > > >>> any
> > > > > > > >>>>>> non-deterministic transform prior to a sink (which are
> > most
> > > > > likely
> > > > > > > >> to
> > > > > > > >>>>>> require deterministic input) will cause additional work
> to
> > > be
> > > > > > > >> needed.
> > > > > > > >>>>>>
> > > > > > > >>>>>> Based on this, it seems like the risk of allowing an
> > > > annotation
> > > > > is
> > > > > > > >>> high
> > > > > > > >>>>>> while the potential for performance improvements is low.
> > The
> > > > > > > >> current
> > > > > > > >>>>>> proposal (not allowing an annotation) makes sense for
> now,
> > > > until
> > > > > > we
> > > > > > > >>> can
> > > > > > > >>>>>> demonstrate that the impact on performance is high in
> > cases
> > > > that
> > > > > > > >>> could
> > > > > > > >>>> be
> > > > > > > >>>>>> avoided with an annotation (in real-world use).
> > > > > > > >>>>>>
> > > > > > > >>>>>> -- Ben
> > > > > > > >>>>>>
> > > > > > > >>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <
> > > vikky.rk@gmail.com>
> > > > > > > >> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>> +1 for the general idea of runners handling it over
> > > hard-coded
> > > > > > > >>>>>> implementation strategy.
> > > > > > > >>>>>>
> > > > > > > >>>>>> For the Write transform I believe you are talking about
> > > > > > > >>>> ApplyShardingKey
> > > > > > > >>>>>> <
> > > > > > > >>>>>>
> > > > > > https://github.com/apache/beam/blob/
> d66029cafde152c0a46ebd276ddfa4
> > > > > > > >>>>>> c3e7fd3433/sdks/java/core/src/
> > > main/java/org/apache/beam/sdk/
> > > > > > > >>>>>> io/Write.java#L304
> > > > > > > >>>>>>>
> > > > > > > >>>>>> which
> > > > > > > >>>>>> introduces non deterministic behavior when retried?
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> *Let a DoFn declare (mechanism not important right now)
> > that
> > > > it
> > > > > > > >>>>>> "requiresdeterministic input"*
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> *Each runner will need a way to induce deterministic
> > input -
> > > > the
> > > > > > > >>>>>> obviouschoice being a materialization.*
> > > > > > > >>>>>>
> > > > > > > >>>>>> Does this mean that a runner will always materialize (or
> > > > > whatever
> > > > > > > >> the
> > > > > > > >>>>>> strategy is) an input PCollection to this DoFn even
> though
> > > the
> > > > > > > >>>>> PCollection
> > > > > > > >>>>>> might have been produced by deterministic transforms?
> > Would
> > > it
> > > > > > make
> > > > > > > >>>> sense
> > > > > > > >>>>>> to also let DoFns declare if they produce
> > non-deterministic
> > > > > > output?
> > > > > > > >>>>>>
> > > > > > > >>>>>> -Vikas
> > > > > > > >>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>> On 21 March 2017 at 13:52, Stephen Sisk
> > > > <sisk@google.com.invalid
> > > > > >
> > > > > > > >>>> wrote:
> > > > > > > >>>>>>
> > > > > > > >>>>>>> Hey Kenn-
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> this seems important, but I don't have all the context
> on
> > > > what
> > > > > > > >> the
> > > > > > > >>>>>> problem
> > > > > > > >>>>>>> is.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Can you explain this sentence "Specifically, there is
> > > > > > > >> pseudorandom
> > > > > > > >>>> data
> > > > > > > >>>>>>> generated and once it has been observed and used to
> > > produce a
> > > > > > > >> side
> > > > > > > >>>>>> effect,
> > > > > > > >>>>>>> it cannot be regenerated without erroneous results." ?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> Where is the pseudorandom data coming from? Perhaps a
> > > > concrete
> > > > > > > >>>> example
> > > > > > > >>>>>>> would help?
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> S
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > > > > > >>>> <klk@google.com.invalid
> > > > > > > >>>>>>
> > > > > > > >>>>>>> wrote:
> > > > > > > >>>>>>>
> > > > > > > >>>>>>>> Problem:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> I will drop all nuance and say that the `Write`
> > transform
> > > as
> > > > > it
> > > > > > > >>>>> exists
> > > > > > > >>>>>> in
> > > > > > > >>>>>>>> the SDK is incorrect until we add some specification
> and
> > > > APIs.
> > > > > > > >> We
> > > > > > > >>>>> can't
> > > > > > > >>>>>>>> keep shipping an SDK with an unsafe transform in it,
> and
> > > IMO
> > > > > > > >> this
> > > > > > > >>>>>>> certainly
> > > > > > > >>>>>>>> blocks a stable release.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Specifically, there is pseudorandom data generated and
> > > once
> > > > it
> > > > > > > >>> has
> > > > > > > >>>>> been
> > > > > > > >>>>>>>> observed and used to produce a side effect, it cannot
> be
> > > > > > > >>>> regenerated
> > > > > > > >>>>>>>> without erroneous results.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> This generalizes: For some side-effecting user-defined
> > > > > > > >> functions,
> > > > > > > >>>> it
> > > > > > > >>>>> is
> > > > > > > >>>>>>>> vital that even across retries/replays they have a
> > > > consistent
> > > > > > > >>> view
> > > > > > > >>>> of
> > > > > > > >>>>>> the
> > > > > > > >>>>>>>> contents of their input PCollection, because their
> > effect
> > > on
> > > > > > > >> the
> > > > > > > >>>>>> outside
> > > > > > > >>>>>>>> world cannot be retracted if/when they fail and are
> > > retried.
> > > > > > > >> Once
> > > > > > > >>>> the
> > > > > > > >>>>>>>> runner ensures a consistent view of the input, it is
> > then
> > > > > their
> > > > > > > >>> own
> > > > > > > >>>>>>>> responsibility to be idempotent.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Ideally we should specify this requirement for the
> > > > > user-defined
> > > > > > > >>>>>> function
> > > > > > > >>>>>>>> without imposing any particular implementation
> strategy
> > on
> > > > > Beam
> > > > > > > >>>>>> runners.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Proposal:
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> 1. Let a DoFn declare (mechanism not important right
> > now)
> > > > that
> > > > > > > >> it
> > > > > > > >>>>>>> "requires
> > > > > > > >>>>>>>> deterministic input".
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> 2. Each runner will need a way to induce deterministic
> > > > input -
> > > > > > > >>> the
> > > > > > > >>>>>>> obvious
> > > > > > > >>>>>>>> choice being a materialization.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> I want to keep the discussion focused, so I'm leaving
> > out
> > > > any
> > > > > > > >>>>>>> possibilities
> > > > > > > >>>>>>>> of taking this further.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Regarding performance: Today places that require this
> > tend
> > > > to
> > > > > > > >> be
> > > > > > > >>>>>> already
> > > > > > > >>>>>>>> paying the cost via GroupByKey / Reshuffle operations,
> > > since
> > > > > > > >> that
> > > > > > > >>>>> was a
> > > > > > > >>>>>>>> simple way to induce determinism in batch Dataflow*
> > > (doesn't
> > > > > > > >> work
> > > > > > > >>>> for
> > > > > > > >>>>>>> most
> > > > > > > >>>>>>>> other runners nor for streaming Dataflow). This change
> > > will
> > > > > > > >>>> replace a
> > > > > > > >>>>>>>> hard-coded implementation strategy with a requirement
> > that
> > > > may
> > > > > > > >> be
> > > > > > > >>>>>>> fulfilled
> > > > > > > >>>>>>>> in the most efficient way available.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Thoughts?
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> Kenn (w/ lots of consult from colleagues, especially
> > Ben)
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>> * There is some overlap with the
> reshuffle/redistribute
> > > > > > > >>> discussion
> > > > > > > >>>>>>> because
> > > > > > > >>>>>>>> of this historical situation, but I would like to
> leave
> > > that
> > > > > > > >>>> broader
> > > > > > > >>>>>>>> discussion out of this correctness issue.
> > > > > > > >>>>>>>>
> > > > > > > >>>>>>>
> > > > > > > >>>>>>
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Reuven Lax <re...@google.com.INVALID>.
It means that single element replay is stable.

On Thu, Aug 10, 2017 at 10:56 AM, Raghu Angadi <ra...@google.com.invalid>
wrote:

> Can we define what exactly is meant by deterministic/stable/replayable
> etc?
>
>    - Does it imply a fixed order? If yes, it implies fixed order of
>    processElement() invocations, right? Are there any qualifiers (within a
>    window+key etc)?
>

No, no ordering guarantee.


>    - Does it also imply fixed length and content for value iterators?
>

Good point. With our current runner api, it does not. the KV<K,
Iterable<V>> has no good way of being deterministic if there is late data.
We could do so by forcing the Iterable to be materialized into a single
element, but that would also mean that the entire Iterable must fit in
memory (which at least the Dataflow runner does not require).


>    - Some examples to clarify nuances would be very useful.
>
> State durability semantics for timers that Reuven proposed seem to be
> unrelated to stable input (at model level). It might be simpler to add
> these semantics first. A lot of deterministic side-effects issues can be
> handled by durable state in timers. One thing I like about timers approach
> is that it makes the cost more transparent to the user since the state is
> explicitly stored.
>
>
> On Thu, Aug 10, 2017 at 10:02 AM, Ben Chambers
> <bchambers@google.com.invalid
> > wrote:
>
> > I think it only makes sense in places where a user might reasonable
> require
> > stable input to ensure idempotency of side-effects. It also only makes
> > sense in places where a runner could reasonably provide such a guarantee.
> >
> > A given Combine is unlikely to have side effects so it is less likely to
> > benefit from stability of the input. Further, the reason writing a
> Combine
> > is desirable is because its execution can be split up and moved to the
> > mapper-side (before the GroupByKey). But this division is inherently
> > non-deterministic, and so it seems unlikely to benefit from stability.
> And
> > many cases where I could see wanting side-effects would end up in
> > extractOutput, for which there is an easy (arguably better) solution --
> > have extractOutput return the accumulators and do the side-effects in a
> > DoFn afterwards.
> >
> > For composites, it is a bit trickier. I could see a case for supporting
> it
> > on composites, but it would need to make it very clear that it only
> > affected the input to the composite. If any of the operations within the
> > composite were non-deterministic, then the outputs of that could be
> > unstable, leading to instability in later parts of the composite.
> Further,
> > it doesn't seem to offer much. The composite itself doesn't perform
> > side-effects, so there is no benefit to having the annotation there --
> > instead, we allow the annotation to be put where it is relevant and
> > important -- on the DoFn's that actually have side-effects that require
> > stability.
> >
> > On Thu, Aug 10, 2017 at 9:23 AM Reuven Lax <re...@google.com.invalid>
> > wrote:
> >
> > > I don't think it really makes sense to to do this on Combine. And I
> agree
> > > with you, it doesn't make sense on composites either.
> > >
> > > On Thu, Aug 10, 2017 at 9:19 AM, Scott Wegner
> <swegner@google.com.invalid
> > >
> > > wrote:
> > >
> > > > Does requires-stable-input only apply to ParDo transforms?
> > > >
> > > > I don't think it would make sense to annotate to composite, because
> > > > checkpointing should happen as close to the side-effecting operation
> as
> > > > possible, since upstream transforms within a composite could
> introduce
> > > > non-determinism. So it's the primitive transform that should own the
> > > > requirement.
> > > >
> > > > Are there other primitives that should be annotated? 'Combine' is
> > > > interesting because it optimized in Dataflow (and perhaps other
> > runners)
> > > to
> > > > partially apply before a GroupByKey.
> > > >
> > > > On Thu, Aug 10, 2017 at 9:01 AM Tyler Akidau
> > <takidau@google.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > +1 to the annotation idea, and to having it on processTimer.
> > > > >
> > > > > -Tyler
> > > > >
> > > > > On Thu, Aug 10, 2017 at 2:15 AM Aljoscha Krettek <
> > aljoscha@apache.org>
> > > > > wrote:
> > > > >
> > > > > > +1 to the annotation approach. I outlined how implementing this
> > would
> > > > > work
> > > > > > in the Flink runner in the Thread about the exactly-once Kafka
> > Sink.
> > > > > >
> > > > > > > On 9. Aug 2017, at 23:03, Reuven Lax <relax@google.com.INVALID
> >
> > > > wrote:
> > > > > > >
> > > > > > > Yes - I don't think we should try and make any deterministic
> > > > guarantees
> > > > > > > about what is in a bundle. Stability guarantees are per element
> > > only.
> > > > > > >
> > > > > > > On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh
> > > <tgroh@google.com.invalid
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> +1 to the annotation-on-ProcessElement approach.
> ProcessElement
> > is
> > > > the
> > > > > > >> minimum implementation requirement of a DoFn, and should be
> > where
> > > > the
> > > > > > >> processing logic which depends on characteristics of the
> inputs
> > > lie.
> > > > > > It's a
> > > > > > >> good way of signalling the requirements of the Fn, and letting
> > the
> > > > > > runner
> > > > > > >> decide.
> > > > > > >>
> > > > > > >> I have a minor concern that this may not work as expected for
> > > users
> > > > > that
> > > > > > >> try to batch remote calls in `FinishBundle` - we should make
> > sure
> > > we
> > > > > > >> document that it is explicitly the input elements that will be
> > > > > replayed,
> > > > > > >> and bundles and other operational are still arbitrary.
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax
> > > > <relax@google.com.invalid
> > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> I think deterministic here means deterministically
> replayable.
> > > i.e.
> > > > > no
> > > > > > >>> matter how many times the element is retried, it will always
> be
> > > the
> > > > > > same.
> > > > > > >>>
> > > > > > >>> I think we should also allow specifying this on processTimer.
> > > This
> > > > > > would
> > > > > > >>> mean that any keyed state written in a previous
> processElement
> > > must
> > > > > be
> > > > > > >>> guaranteed durable before processTimer is called.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <
> > > > bchambers@apache.org>
> > > > > > >>> wrote:
> > > > > > >>>
> > > > > > >>>> I strongly agree with this proposal. I think moving away
> from
> > > > "just
> > > > > > >>> insert
> > > > > > >>>> a GroupByKey for one of the 3 different reasons you may want
> > it"
> > > > > > >> towards
> > > > > > >>>> APIs that allow code to express the requirements they have
> and
> > > the
> > > > > > >> runner
> > > > > > >>>> to choose the best way to meet this is a major step forwards
> > in
> > > > > terms
> > > > > > >> of
> > > > > > >>>> portability.
> > > > > > >>>>
> > > > > > >>>> I think "deterministic" may be misleading. The actual
> contents
> > > of
> > > > > the
> > > > > > >>>> collection aren't deterministic if upstream computations
> > aren't.
> > > > The
> > > > > > >>>> property we really need is that once an input may have been
> > > > observed
> > > > > > by
> > > > > > >>> the
> > > > > > >>>> side-effecting code it will never be observed with a
> different
> > > > > value.
> > > > > > >>>>
> > > > > > >>>> I would propose something RequiresStableInput, to indicate
> > that
> > > > the
> > > > > > >> input
> > > > > > >>>> must be stable as observed by the function. I could also see
> > > > > something
> > > > > > >>>> hinting at the fact we don't recompute the input, such as
> > > > > > >>>> RequiresMemoizedInput or RequiresNoRecomputation.
> > > > > > >>>>
> > > > > > >>>> -- Ben
> > > > > > >>>>
> > > > > > >>>> P.S For anyone interested other uses of GroupByKey that we
> may
> > > > want
> > > > > to
> > > > > > >>>> discuss APIs for would be preventing retry across steps
> (eg.,
> > > > > > >> preventing
> > > > > > >>>> fusion) and redistributing inputs across workers.
> > > > > > >>>>
> > > > > > >>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles
> > > > > <klk@google.com.invalid
> > > > > > >>>
> > > > > > >>>> wrote:
> > > > > > >>>>
> > > > > > >>>>> This came up again, so I wanted to push it along by
> > proposing a
> > > > > > >>> specific
> > > > > > >>>>> API for Java that could have a derived API in Python. I am
> > > > writing
> > > > > > >> this
> > > > > > >>>>> quickly to get something out there, so I welcome
> suggestions
> > > for
> > > > > > >>>> revision.
> > > > > > >>>>>
> > > > > > >>>>> Today a DoFn has a @ProcessElement annotated method with
> > > various
> > > > > > >>>> automated
> > > > > > >>>>> parameters, but most fundamentally this:
> > > > > > >>>>>
> > > > > > >>>>> @ProcessElement
> > > > > > >>>>> public void process(ProcessContext ctx) {
> > > > > > >>>>>  ctx.element() // to access the current input element
> > > > > > >>>>>  ctx.output(something) // to write to default output
> > collection
> > > > > > >>>>>  ctx.output(tag, something) // to write to other output
> > > > collections
> > > > > > >>>>> }
> > > > > > >>>>>
> > > > > > >>>>> For some time, we have hoped to unpack the context - it is
> a
> > > > > > >>>>> backwards-compatibility pattern made obsolete by the new
> DoFn
> > > > > design.
> > > > > > >>> So
> > > > > > >>>>> instead it would look like this:
> > > > > > >>>>>
> > > > > > >>>>> @ProcessElement
> > > > > > >>>>> public void process(Element element, MainOutput mainOutput,
> > > ...)
> > > > {
> > > > > > >>>>>  element.get() // to access the current input element
> > > > > > >>>>>  mainOutput.output(something) // to write to the default
> > output
> > > > > > >>>> collection
> > > > > > >>>>>  other.output(something) // to write to other output
> > collection
> > > > > > >>>>> }
> > > > > > >>>>>
> > > > > > >>>>> I've deliberately left out the undecided syntax for side
> > > outputs.
> > > > > But
> > > > > > >>> it
> > > > > > >>>>> would be nice for the tag to be built in to the parameter
> so
> > it
> > > > > > >> doesn't
> > > > > > >>>>> have to be used when calling output().
> > > > > > >>>>>
> > > > > > >>>>> One way to enhance this to deterministic input would just
> be
> > > > this:
> > > > > > >>>>>
> > > > > > >>>>> @ProcessElement
> > > > > > >>>>> @RequiresDeterministicInput
> > > > > > >>>>> public void process(Element element, MainOutput mainOutput,
> > > ...)
> > > > {
> > > > > > >>>>>  element.get() // to access the current input element
> > > > > > >>>>>  mainOutput.output(something) // to write to the default
> > output
> > > > > > >>>> collection
> > > > > > >>>>>  other.output(something) // to write to other output
> > collection
> > > > > > >>>>> }
> > > > > > >>>>>
> > > > > > >>>>> There are really a lot of places where we could put an
> > > annotation
> > > > > or
> > > > > > >>>> change
> > > > > > >>>>> a type to indicate that the input PCollection should be
> > > > > > >>>>> well-defined/deterministically-replayable. I don't have a
> > > really
> > > > > > >>> strong
> > > > > > >>>>> opinion.
> > > > > > >>>>>
> > > > > > >>>>> Kenn
> > > > > > >>>>>
> > > > > > >>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > > > > >>>> <bchambers@google.com.invalid
> > > > > > >>>>>>
> > > > > > >>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>>> Allowing an annotation on DoFn's that produce
> deterministic
> > > > output
> > > > > > >>>> could
> > > > > > >>>>> be
> > > > > > >>>>>> added in the future, but doesn't seem like a great option.
> > > > > > >>>>>>
> > > > > > >>>>>> 1. It is a correctness issue to assume a DoFn is
> > deterministic
> > > > and
> > > > > > >> be
> > > > > > >>>>>> wrong, so we would need to assume all transform outputs
> are
> > > > > > >>>>>> non-deterministic unless annotated. Getting this correct
> is
> > > > > > >> difficult
> > > > > > >>>>> (for
> > > > > > >>>>>> example, GBK is surprisingly non-deterministic except in
> > > > specific
> > > > > > >>>> cases).
> > > > > > >>>>>>
> > > > > > >>>>>> 2. It is unlikely to be a major performance improvement,
> > given
> > > > > that
> > > > > > >>> any
> > > > > > >>>>>> non-deterministic transform prior to a sink (which are
> most
> > > > likely
> > > > > > >> to
> > > > > > >>>>>> require deterministic input) will cause additional work to
> > be
> > > > > > >> needed.
> > > > > > >>>>>>
> > > > > > >>>>>> Based on this, it seems like the risk of allowing an
> > > annotation
> > > > is
> > > > > > >>> high
> > > > > > >>>>>> while the potential for performance improvements is low.
> The
> > > > > > >> current
> > > > > > >>>>>> proposal (not allowing an annotation) makes sense for now,
> > > until
> > > > > we
> > > > > > >>> can
> > > > > > >>>>>> demonstrate that the impact on performance is high in
> cases
> > > that
> > > > > > >>> could
> > > > > > >>>> be
> > > > > > >>>>>> avoided with an annotation (in real-world use).
> > > > > > >>>>>>
> > > > > > >>>>>> -- Ben
> > > > > > >>>>>>
> > > > > > >>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <
> > vikky.rk@gmail.com>
> > > > > > >> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>> +1 for the general idea of runners handling it over
> > hard-coded
> > > > > > >>>>>> implementation strategy.
> > > > > > >>>>>>
> > > > > > >>>>>> For the Write transform I believe you are talking about
> > > > > > >>>> ApplyShardingKey
> > > > > > >>>>>> <
> > > > > > >>>>>>
> > > > > https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > > > > > >>>>>> c3e7fd3433/sdks/java/core/src/
> > main/java/org/apache/beam/sdk/
> > > > > > >>>>>> io/Write.java#L304
> > > > > > >>>>>>>
> > > > > > >>>>>> which
> > > > > > >>>>>> introduces non deterministic behavior when retried?
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> *Let a DoFn declare (mechanism not important right now)
> that
> > > it
> > > > > > >>>>>> "requiresdeterministic input"*
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> *Each runner will need a way to induce deterministic
> input -
> > > the
> > > > > > >>>>>> obviouschoice being a materialization.*
> > > > > > >>>>>>
> > > > > > >>>>>> Does this mean that a runner will always materialize (or
> > > > whatever
> > > > > > >> the
> > > > > > >>>>>> strategy is) an input PCollection to this DoFn even though
> > the
> > > > > > >>>>> PCollection
> > > > > > >>>>>> might have been produced by deterministic transforms?
> Would
> > it
> > > > > make
> > > > > > >>>> sense
> > > > > > >>>>>> to also let DoFns declare if they produce
> non-deterministic
> > > > > output?
> > > > > > >>>>>>
> > > > > > >>>>>> -Vikas
> > > > > > >>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>> On 21 March 2017 at 13:52, Stephen Sisk
> > > <sisk@google.com.invalid
> > > > >
> > > > > > >>>> wrote:
> > > > > > >>>>>>
> > > > > > >>>>>>> Hey Kenn-
> > > > > > >>>>>>>
> > > > > > >>>>>>> this seems important, but I don't have all the context on
> > > what
> > > > > > >> the
> > > > > > >>>>>> problem
> > > > > > >>>>>>> is.
> > > > > > >>>>>>>
> > > > > > >>>>>>> Can you explain this sentence "Specifically, there is
> > > > > > >> pseudorandom
> > > > > > >>>> data
> > > > > > >>>>>>> generated and once it has been observed and used to
> > produce a
> > > > > > >> side
> > > > > > >>>>>> effect,
> > > > > > >>>>>>> it cannot be regenerated without erroneous results." ?
> > > > > > >>>>>>>
> > > > > > >>>>>>> Where is the pseudorandom data coming from? Perhaps a
> > > concrete
> > > > > > >>>> example
> > > > > > >>>>>>> would help?
> > > > > > >>>>>>>
> > > > > > >>>>>>> S
> > > > > > >>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > > > > >>>> <klk@google.com.invalid
> > > > > > >>>>>>
> > > > > > >>>>>>> wrote:
> > > > > > >>>>>>>
> > > > > > >>>>>>>> Problem:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I will drop all nuance and say that the `Write`
> transform
> > as
> > > > it
> > > > > > >>>>> exists
> > > > > > >>>>>> in
> > > > > > >>>>>>>> the SDK is incorrect until we add some specification and
> > > APIs.
> > > > > > >> We
> > > > > > >>>>> can't
> > > > > > >>>>>>>> keep shipping an SDK with an unsafe transform in it, and
> > IMO
> > > > > > >> this
> > > > > > >>>>>>> certainly
> > > > > > >>>>>>>> blocks a stable release.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Specifically, there is pseudorandom data generated and
> > once
> > > it
> > > > > > >>> has
> > > > > > >>>>> been
> > > > > > >>>>>>>> observed and used to produce a side effect, it cannot be
> > > > > > >>>> regenerated
> > > > > > >>>>>>>> without erroneous results.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> This generalizes: For some side-effecting user-defined
> > > > > > >> functions,
> > > > > > >>>> it
> > > > > > >>>>> is
> > > > > > >>>>>>>> vital that even across retries/replays they have a
> > > consistent
> > > > > > >>> view
> > > > > > >>>> of
> > > > > > >>>>>> the
> > > > > > >>>>>>>> contents of their input PCollection, because their
> effect
> > on
> > > > > > >> the
> > > > > > >>>>>> outside
> > > > > > >>>>>>>> world cannot be retracted if/when they fail and are
> > retried.
> > > > > > >> Once
> > > > > > >>>> the
> > > > > > >>>>>>>> runner ensures a consistent view of the input, it is
> then
> > > > their
> > > > > > >>> own
> > > > > > >>>>>>>> responsibility to be idempotent.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Ideally we should specify this requirement for the
> > > > user-defined
> > > > > > >>>>>> function
> > > > > > >>>>>>>> without imposing any particular implementation strategy
> on
> > > > Beam
> > > > > > >>>>>> runners.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Proposal:
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> 1. Let a DoFn declare (mechanism not important right
> now)
> > > that
> > > > > > >> it
> > > > > > >>>>>>> "requires
> > > > > > >>>>>>>> deterministic input".
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> 2. Each runner will need a way to induce deterministic
> > > input -
> > > > > > >>> the
> > > > > > >>>>>>> obvious
> > > > > > >>>>>>>> choice being a materialization.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> I want to keep the discussion focused, so I'm leaving
> out
> > > any
> > > > > > >>>>>>> possibilities
> > > > > > >>>>>>>> of taking this further.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Regarding performance: Today places that require this
> tend
> > > to
> > > > > > >> be
> > > > > > >>>>>> already
> > > > > > >>>>>>>> paying the cost via GroupByKey / Reshuffle operations,
> > since
> > > > > > >> that
> > > > > > >>>>> was a
> > > > > > >>>>>>>> simple way to induce determinism in batch Dataflow*
> > (doesn't
> > > > > > >> work
> > > > > > >>>> for
> > > > > > >>>>>>> most
> > > > > > >>>>>>>> other runners nor for streaming Dataflow). This change
> > will
> > > > > > >>>> replace a
> > > > > > >>>>>>>> hard-coded implementation strategy with a requirement
> that
> > > may
> > > > > > >> be
> > > > > > >>>>>>> fulfilled
> > > > > > >>>>>>>> in the most efficient way available.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Thoughts?
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> Kenn (w/ lots of consult from colleagues, especially
> Ben)
> > > > > > >>>>>>>>
> > > > > > >>>>>>>> * There is some overlap with the reshuffle/redistribute
> > > > > > >>> discussion
> > > > > > >>>>>>> because
> > > > > > >>>>>>>> of this historical situation, but I would like to leave
> > that
> > > > > > >>>> broader
> > > > > > >>>>>>>> discussion out of this correctness issue.
> > > > > > >>>>>>>>
> > > > > > >>>>>>>
> > > > > > >>>>>>
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Raghu Angadi <ra...@google.com.INVALID>.
Can we define what exactly is meant by deterministic/stable/replayable etc?

   - Does it imply a fixed order? If yes, it implies fixed order of
   processElement() invocations, right? Are there any qualifiers (within a
   window+key etc)?
   - Does it also imply fixed length and content for value iterators?
   - Some examples to clarify nuances would be very useful.

State durability semantics for timers that Reuven proposed seem to be
unrelated to stable input (at model level). It might be simpler to add
these semantics first. A lot of deterministic side-effects issues can be
handled by durable state in timers. One thing I like about timers approach
is that it makes the cost more transparent to the user since the state is
explicitly stored.


On Thu, Aug 10, 2017 at 10:02 AM, Ben Chambers <bchambers@google.com.invalid
> wrote:

> I think it only makes sense in places where a user might reasonable require
> stable input to ensure idempotency of side-effects. It also only makes
> sense in places where a runner could reasonably provide such a guarantee.
>
> A given Combine is unlikely to have side effects so it is less likely to
> benefit from stability of the input. Further, the reason writing a Combine
> is desirable is because its execution can be split up and moved to the
> mapper-side (before the GroupByKey). But this division is inherently
> non-deterministic, and so it seems unlikely to benefit from stability. And
> many cases where I could see wanting side-effects would end up in
> extractOutput, for which there is an easy (arguably better) solution --
> have extractOutput return the accumulators and do the side-effects in a
> DoFn afterwards.
>
> For composites, it is a bit trickier. I could see a case for supporting it
> on composites, but it would need to make it very clear that it only
> affected the input to the composite. If any of the operations within the
> composite were non-deterministic, then the outputs of that could be
> unstable, leading to instability in later parts of the composite. Further,
> it doesn't seem to offer much. The composite itself doesn't perform
> side-effects, so there is no benefit to having the annotation there --
> instead, we allow the annotation to be put where it is relevant and
> important -- on the DoFn's that actually have side-effects that require
> stability.
>
> On Thu, Aug 10, 2017 at 9:23 AM Reuven Lax <re...@google.com.invalid>
> wrote:
>
> > I don't think it really makes sense to to do this on Combine. And I agree
> > with you, it doesn't make sense on composites either.
> >
> > On Thu, Aug 10, 2017 at 9:19 AM, Scott Wegner <swegner@google.com.invalid
> >
> > wrote:
> >
> > > Does requires-stable-input only apply to ParDo transforms?
> > >
> > > I don't think it would make sense to annotate to composite, because
> > > checkpointing should happen as close to the side-effecting operation as
> > > possible, since upstream transforms within a composite could introduce
> > > non-determinism. So it's the primitive transform that should own the
> > > requirement.
> > >
> > > Are there other primitives that should be annotated? 'Combine' is
> > > interesting because it optimized in Dataflow (and perhaps other
> runners)
> > to
> > > partially apply before a GroupByKey.
> > >
> > > On Thu, Aug 10, 2017 at 9:01 AM Tyler Akidau
> <takidau@google.com.invalid
> > >
> > > wrote:
> > >
> > > > +1 to the annotation idea, and to having it on processTimer.
> > > >
> > > > -Tyler
> > > >
> > > > On Thu, Aug 10, 2017 at 2:15 AM Aljoscha Krettek <
> aljoscha@apache.org>
> > > > wrote:
> > > >
> > > > > +1 to the annotation approach. I outlined how implementing this
> would
> > > > work
> > > > > in the Flink runner in the Thread about the exactly-once Kafka
> Sink.
> > > > >
> > > > > > On 9. Aug 2017, at 23:03, Reuven Lax <re...@google.com.INVALID>
> > > wrote:
> > > > > >
> > > > > > Yes - I don't think we should try and make any deterministic
> > > guarantees
> > > > > > about what is in a bundle. Stability guarantees are per element
> > only.
> > > > > >
> > > > > > On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh
> > <tgroh@google.com.invalid
> > > >
> > > > > > wrote:
> > > > > >
> > > > > >> +1 to the annotation-on-ProcessElement approach. ProcessElement
> is
> > > the
> > > > > >> minimum implementation requirement of a DoFn, and should be
> where
> > > the
> > > > > >> processing logic which depends on characteristics of the inputs
> > lie.
> > > > > It's a
> > > > > >> good way of signalling the requirements of the Fn, and letting
> the
> > > > > runner
> > > > > >> decide.
> > > > > >>
> > > > > >> I have a minor concern that this may not work as expected for
> > users
> > > > that
> > > > > >> try to batch remote calls in `FinishBundle` - we should make
> sure
> > we
> > > > > >> document that it is explicitly the input elements that will be
> > > > replayed,
> > > > > >> and bundles and other operational are still arbitrary.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax
> > > <relax@google.com.invalid
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> I think deterministic here means deterministically replayable.
> > i.e.
> > > > no
> > > > > >>> matter how many times the element is retried, it will always be
> > the
> > > > > same.
> > > > > >>>
> > > > > >>> I think we should also allow specifying this on processTimer.
> > This
> > > > > would
> > > > > >>> mean that any keyed state written in a previous processElement
> > must
> > > > be
> > > > > >>> guaranteed durable before processTimer is called.
> > > > > >>>
> > > > > >>>
> > > > > >>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <
> > > bchambers@apache.org>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>>> I strongly agree with this proposal. I think moving away from
> > > "just
> > > > > >>> insert
> > > > > >>>> a GroupByKey for one of the 3 different reasons you may want
> it"
> > > > > >> towards
> > > > > >>>> APIs that allow code to express the requirements they have and
> > the
> > > > > >> runner
> > > > > >>>> to choose the best way to meet this is a major step forwards
> in
> > > > terms
> > > > > >> of
> > > > > >>>> portability.
> > > > > >>>>
> > > > > >>>> I think "deterministic" may be misleading. The actual contents
> > of
> > > > the
> > > > > >>>> collection aren't deterministic if upstream computations
> aren't.
> > > The
> > > > > >>>> property we really need is that once an input may have been
> > > observed
> > > > > by
> > > > > >>> the
> > > > > >>>> side-effecting code it will never be observed with a different
> > > > value.
> > > > > >>>>
> > > > > >>>> I would propose something RequiresStableInput, to indicate
> that
> > > the
> > > > > >> input
> > > > > >>>> must be stable as observed by the function. I could also see
> > > > something
> > > > > >>>> hinting at the fact we don't recompute the input, such as
> > > > > >>>> RequiresMemoizedInput or RequiresNoRecomputation.
> > > > > >>>>
> > > > > >>>> -- Ben
> > > > > >>>>
> > > > > >>>> P.S For anyone interested other uses of GroupByKey that we may
> > > want
> > > > to
> > > > > >>>> discuss APIs for would be preventing retry across steps (eg.,
> > > > > >> preventing
> > > > > >>>> fusion) and redistributing inputs across workers.
> > > > > >>>>
> > > > > >>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles
> > > > <klk@google.com.invalid
> > > > > >>>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> This came up again, so I wanted to push it along by
> proposing a
> > > > > >>> specific
> > > > > >>>>> API for Java that could have a derived API in Python. I am
> > > writing
> > > > > >> this
> > > > > >>>>> quickly to get something out there, so I welcome suggestions
> > for
> > > > > >>>> revision.
> > > > > >>>>>
> > > > > >>>>> Today a DoFn has a @ProcessElement annotated method with
> > various
> > > > > >>>> automated
> > > > > >>>>> parameters, but most fundamentally this:
> > > > > >>>>>
> > > > > >>>>> @ProcessElement
> > > > > >>>>> public void process(ProcessContext ctx) {
> > > > > >>>>>  ctx.element() // to access the current input element
> > > > > >>>>>  ctx.output(something) // to write to default output
> collection
> > > > > >>>>>  ctx.output(tag, something) // to write to other output
> > > collections
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> For some time, we have hoped to unpack the context - it is a
> > > > > >>>>> backwards-compatibility pattern made obsolete by the new DoFn
> > > > design.
> > > > > >>> So
> > > > > >>>>> instead it would look like this:
> > > > > >>>>>
> > > > > >>>>> @ProcessElement
> > > > > >>>>> public void process(Element element, MainOutput mainOutput,
> > ...)
> > > {
> > > > > >>>>>  element.get() // to access the current input element
> > > > > >>>>>  mainOutput.output(something) // to write to the default
> output
> > > > > >>>> collection
> > > > > >>>>>  other.output(something) // to write to other output
> collection
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> I've deliberately left out the undecided syntax for side
> > outputs.
> > > > But
> > > > > >>> it
> > > > > >>>>> would be nice for the tag to be built in to the parameter so
> it
> > > > > >> doesn't
> > > > > >>>>> have to be used when calling output().
> > > > > >>>>>
> > > > > >>>>> One way to enhance this to deterministic input would just be
> > > this:
> > > > > >>>>>
> > > > > >>>>> @ProcessElement
> > > > > >>>>> @RequiresDeterministicInput
> > > > > >>>>> public void process(Element element, MainOutput mainOutput,
> > ...)
> > > {
> > > > > >>>>>  element.get() // to access the current input element
> > > > > >>>>>  mainOutput.output(something) // to write to the default
> output
> > > > > >>>> collection
> > > > > >>>>>  other.output(something) // to write to other output
> collection
> > > > > >>>>> }
> > > > > >>>>>
> > > > > >>>>> There are really a lot of places where we could put an
> > annotation
> > > > or
> > > > > >>>> change
> > > > > >>>>> a type to indicate that the input PCollection should be
> > > > > >>>>> well-defined/deterministically-replayable. I don't have a
> > really
> > > > > >>> strong
> > > > > >>>>> opinion.
> > > > > >>>>>
> > > > > >>>>> Kenn
> > > > > >>>>>
> > > > > >>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > > > >>>> <bchambers@google.com.invalid
> > > > > >>>>>>
> > > > > >>>>> wrote:
> > > > > >>>>>
> > > > > >>>>>> Allowing an annotation on DoFn's that produce deterministic
> > > output
> > > > > >>>> could
> > > > > >>>>> be
> > > > > >>>>>> added in the future, but doesn't seem like a great option.
> > > > > >>>>>>
> > > > > >>>>>> 1. It is a correctness issue to assume a DoFn is
> deterministic
> > > and
> > > > > >> be
> > > > > >>>>>> wrong, so we would need to assume all transform outputs are
> > > > > >>>>>> non-deterministic unless annotated. Getting this correct is
> > > > > >> difficult
> > > > > >>>>> (for
> > > > > >>>>>> example, GBK is surprisingly non-deterministic except in
> > > specific
> > > > > >>>> cases).
> > > > > >>>>>>
> > > > > >>>>>> 2. It is unlikely to be a major performance improvement,
> given
> > > > that
> > > > > >>> any
> > > > > >>>>>> non-deterministic transform prior to a sink (which are most
> > > likely
> > > > > >> to
> > > > > >>>>>> require deterministic input) will cause additional work to
> be
> > > > > >> needed.
> > > > > >>>>>>
> > > > > >>>>>> Based on this, it seems like the risk of allowing an
> > annotation
> > > is
> > > > > >>> high
> > > > > >>>>>> while the potential for performance improvements is low. The
> > > > > >> current
> > > > > >>>>>> proposal (not allowing an annotation) makes sense for now,
> > until
> > > > we
> > > > > >>> can
> > > > > >>>>>> demonstrate that the impact on performance is high in cases
> > that
> > > > > >>> could
> > > > > >>>> be
> > > > > >>>>>> avoided with an annotation (in real-world use).
> > > > > >>>>>>
> > > > > >>>>>> -- Ben
> > > > > >>>>>>
> > > > > >>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <
> vikky.rk@gmail.com>
> > > > > >> wrote:
> > > > > >>>>>>
> > > > > >>>>>> +1 for the general idea of runners handling it over
> hard-coded
> > > > > >>>>>> implementation strategy.
> > > > > >>>>>>
> > > > > >>>>>> For the Write transform I believe you are talking about
> > > > > >>>> ApplyShardingKey
> > > > > >>>>>> <
> > > > > >>>>>>
> > > > https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > > > > >>>>>> c3e7fd3433/sdks/java/core/src/
> main/java/org/apache/beam/sdk/
> > > > > >>>>>> io/Write.java#L304
> > > > > >>>>>>>
> > > > > >>>>>> which
> > > > > >>>>>> introduces non deterministic behavior when retried?
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> *Let a DoFn declare (mechanism not important right now) that
> > it
> > > > > >>>>>> "requiresdeterministic input"*
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> *Each runner will need a way to induce deterministic input -
> > the
> > > > > >>>>>> obviouschoice being a materialization.*
> > > > > >>>>>>
> > > > > >>>>>> Does this mean that a runner will always materialize (or
> > > whatever
> > > > > >> the
> > > > > >>>>>> strategy is) an input PCollection to this DoFn even though
> the
> > > > > >>>>> PCollection
> > > > > >>>>>> might have been produced by deterministic transforms? Would
> it
> > > > make
> > > > > >>>> sense
> > > > > >>>>>> to also let DoFns declare if they produce non-deterministic
> > > > output?
> > > > > >>>>>>
> > > > > >>>>>> -Vikas
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On 21 March 2017 at 13:52, Stephen Sisk
> > <sisk@google.com.invalid
> > > >
> > > > > >>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hey Kenn-
> > > > > >>>>>>>
> > > > > >>>>>>> this seems important, but I don't have all the context on
> > what
> > > > > >> the
> > > > > >>>>>> problem
> > > > > >>>>>>> is.
> > > > > >>>>>>>
> > > > > >>>>>>> Can you explain this sentence "Specifically, there is
> > > > > >> pseudorandom
> > > > > >>>> data
> > > > > >>>>>>> generated and once it has been observed and used to
> produce a
> > > > > >> side
> > > > > >>>>>> effect,
> > > > > >>>>>>> it cannot be regenerated without erroneous results." ?
> > > > > >>>>>>>
> > > > > >>>>>>> Where is the pseudorandom data coming from? Perhaps a
> > concrete
> > > > > >>>> example
> > > > > >>>>>>> would help?
> > > > > >>>>>>>
> > > > > >>>>>>> S
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > > > >>>> <klk@google.com.invalid
> > > > > >>>>>>
> > > > > >>>>>>> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>>> Problem:
> > > > > >>>>>>>>
> > > > > >>>>>>>> I will drop all nuance and say that the `Write` transform
> as
> > > it
> > > > > >>>>> exists
> > > > > >>>>>> in
> > > > > >>>>>>>> the SDK is incorrect until we add some specification and
> > APIs.
> > > > > >> We
> > > > > >>>>> can't
> > > > > >>>>>>>> keep shipping an SDK with an unsafe transform in it, and
> IMO
> > > > > >> this
> > > > > >>>>>>> certainly
> > > > > >>>>>>>> blocks a stable release.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Specifically, there is pseudorandom data generated and
> once
> > it
> > > > > >>> has
> > > > > >>>>> been
> > > > > >>>>>>>> observed and used to produce a side effect, it cannot be
> > > > > >>>> regenerated
> > > > > >>>>>>>> without erroneous results.
> > > > > >>>>>>>>
> > > > > >>>>>>>> This generalizes: For some side-effecting user-defined
> > > > > >> functions,
> > > > > >>>> it
> > > > > >>>>> is
> > > > > >>>>>>>> vital that even across retries/replays they have a
> > consistent
> > > > > >>> view
> > > > > >>>> of
> > > > > >>>>>> the
> > > > > >>>>>>>> contents of their input PCollection, because their effect
> on
> > > > > >> the
> > > > > >>>>>> outside
> > > > > >>>>>>>> world cannot be retracted if/when they fail and are
> retried.
> > > > > >> Once
> > > > > >>>> the
> > > > > >>>>>>>> runner ensures a consistent view of the input, it is then
> > > their
> > > > > >>> own
> > > > > >>>>>>>> responsibility to be idempotent.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Ideally we should specify this requirement for the
> > > user-defined
> > > > > >>>>>> function
> > > > > >>>>>>>> without imposing any particular implementation strategy on
> > > Beam
> > > > > >>>>>> runners.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Proposal:
> > > > > >>>>>>>>
> > > > > >>>>>>>> 1. Let a DoFn declare (mechanism not important right now)
> > that
> > > > > >> it
> > > > > >>>>>>> "requires
> > > > > >>>>>>>> deterministic input".
> > > > > >>>>>>>>
> > > > > >>>>>>>> 2. Each runner will need a way to induce deterministic
> > input -
> > > > > >>> the
> > > > > >>>>>>> obvious
> > > > > >>>>>>>> choice being a materialization.
> > > > > >>>>>>>>
> > > > > >>>>>>>> I want to keep the discussion focused, so I'm leaving out
> > any
> > > > > >>>>>>> possibilities
> > > > > >>>>>>>> of taking this further.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Regarding performance: Today places that require this tend
> > to
> > > > > >> be
> > > > > >>>>>> already
> > > > > >>>>>>>> paying the cost via GroupByKey / Reshuffle operations,
> since
> > > > > >> that
> > > > > >>>>> was a
> > > > > >>>>>>>> simple way to induce determinism in batch Dataflow*
> (doesn't
> > > > > >> work
> > > > > >>>> for
> > > > > >>>>>>> most
> > > > > >>>>>>>> other runners nor for streaming Dataflow). This change
> will
> > > > > >>>> replace a
> > > > > >>>>>>>> hard-coded implementation strategy with a requirement that
> > may
> > > > > >> be
> > > > > >>>>>>> fulfilled
> > > > > >>>>>>>> in the most efficient way available.
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thoughts?
> > > > > >>>>>>>>
> > > > > >>>>>>>> Kenn (w/ lots of consult from colleagues, especially Ben)
> > > > > >>>>>>>>
> > > > > >>>>>>>> * There is some overlap with the reshuffle/redistribute
> > > > > >>> discussion
> > > > > >>>>>>> because
> > > > > >>>>>>>> of this historical situation, but I would like to leave
> that
> > > > > >>>> broader
> > > > > >>>>>>>> discussion out of this correctness issue.
> > > > > >>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Ben Chambers <bc...@google.com.INVALID>.
I think it only makes sense in places where a user might reasonable require
stable input to ensure idempotency of side-effects. It also only makes
sense in places where a runner could reasonably provide such a guarantee.

A given Combine is unlikely to have side effects so it is less likely to
benefit from stability of the input. Further, the reason writing a Combine
is desirable is because its execution can be split up and moved to the
mapper-side (before the GroupByKey). But this division is inherently
non-deterministic, and so it seems unlikely to benefit from stability. And
many cases where I could see wanting side-effects would end up in
extractOutput, for which there is an easy (arguably better) solution --
have extractOutput return the accumulators and do the side-effects in a
DoFn afterwards.

For composites, it is a bit trickier. I could see a case for supporting it
on composites, but it would need to make it very clear that it only
affected the input to the composite. If any of the operations within the
composite were non-deterministic, then the outputs of that could be
unstable, leading to instability in later parts of the composite. Further,
it doesn't seem to offer much. The composite itself doesn't perform
side-effects, so there is no benefit to having the annotation there --
instead, we allow the annotation to be put where it is relevant and
important -- on the DoFn's that actually have side-effects that require
stability.

On Thu, Aug 10, 2017 at 9:23 AM Reuven Lax <re...@google.com.invalid> wrote:

> I don't think it really makes sense to to do this on Combine. And I agree
> with you, it doesn't make sense on composites either.
>
> On Thu, Aug 10, 2017 at 9:19 AM, Scott Wegner <sw...@google.com.invalid>
> wrote:
>
> > Does requires-stable-input only apply to ParDo transforms?
> >
> > I don't think it would make sense to annotate to composite, because
> > checkpointing should happen as close to the side-effecting operation as
> > possible, since upstream transforms within a composite could introduce
> > non-determinism. So it's the primitive transform that should own the
> > requirement.
> >
> > Are there other primitives that should be annotated? 'Combine' is
> > interesting because it optimized in Dataflow (and perhaps other runners)
> to
> > partially apply before a GroupByKey.
> >
> > On Thu, Aug 10, 2017 at 9:01 AM Tyler Akidau <takidau@google.com.invalid
> >
> > wrote:
> >
> > > +1 to the annotation idea, and to having it on processTimer.
> > >
> > > -Tyler
> > >
> > > On Thu, Aug 10, 2017 at 2:15 AM Aljoscha Krettek <al...@apache.org>
> > > wrote:
> > >
> > > > +1 to the annotation approach. I outlined how implementing this would
> > > work
> > > > in the Flink runner in the Thread about the exactly-once Kafka Sink.
> > > >
> > > > > On 9. Aug 2017, at 23:03, Reuven Lax <re...@google.com.INVALID>
> > wrote:
> > > > >
> > > > > Yes - I don't think we should try and make any deterministic
> > guarantees
> > > > > about what is in a bundle. Stability guarantees are per element
> only.
> > > > >
> > > > > On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh
> <tgroh@google.com.invalid
> > >
> > > > > wrote:
> > > > >
> > > > >> +1 to the annotation-on-ProcessElement approach. ProcessElement is
> > the
> > > > >> minimum implementation requirement of a DoFn, and should be where
> > the
> > > > >> processing logic which depends on characteristics of the inputs
> lie.
> > > > It's a
> > > > >> good way of signalling the requirements of the Fn, and letting the
> > > > runner
> > > > >> decide.
> > > > >>
> > > > >> I have a minor concern that this may not work as expected for
> users
> > > that
> > > > >> try to batch remote calls in `FinishBundle` - we should make sure
> we
> > > > >> document that it is explicitly the input elements that will be
> > > replayed,
> > > > >> and bundles and other operational are still arbitrary.
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax
> > <relax@google.com.invalid
> > > >
> > > > >> wrote:
> > > > >>
> > > > >>> I think deterministic here means deterministically replayable.
> i.e.
> > > no
> > > > >>> matter how many times the element is retried, it will always be
> the
> > > > same.
> > > > >>>
> > > > >>> I think we should also allow specifying this on processTimer.
> This
> > > > would
> > > > >>> mean that any keyed state written in a previous processElement
> must
> > > be
> > > > >>> guaranteed durable before processTimer is called.
> > > > >>>
> > > > >>>
> > > > >>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <
> > bchambers@apache.org>
> > > > >>> wrote:
> > > > >>>
> > > > >>>> I strongly agree with this proposal. I think moving away from
> > "just
> > > > >>> insert
> > > > >>>> a GroupByKey for one of the 3 different reasons you may want it"
> > > > >> towards
> > > > >>>> APIs that allow code to express the requirements they have and
> the
> > > > >> runner
> > > > >>>> to choose the best way to meet this is a major step forwards in
> > > terms
> > > > >> of
> > > > >>>> portability.
> > > > >>>>
> > > > >>>> I think "deterministic" may be misleading. The actual contents
> of
> > > the
> > > > >>>> collection aren't deterministic if upstream computations aren't.
> > The
> > > > >>>> property we really need is that once an input may have been
> > observed
> > > > by
> > > > >>> the
> > > > >>>> side-effecting code it will never be observed with a different
> > > value.
> > > > >>>>
> > > > >>>> I would propose something RequiresStableInput, to indicate that
> > the
> > > > >> input
> > > > >>>> must be stable as observed by the function. I could also see
> > > something
> > > > >>>> hinting at the fact we don't recompute the input, such as
> > > > >>>> RequiresMemoizedInput or RequiresNoRecomputation.
> > > > >>>>
> > > > >>>> -- Ben
> > > > >>>>
> > > > >>>> P.S For anyone interested other uses of GroupByKey that we may
> > want
> > > to
> > > > >>>> discuss APIs for would be preventing retry across steps (eg.,
> > > > >> preventing
> > > > >>>> fusion) and redistributing inputs across workers.
> > > > >>>>
> > > > >>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles
> > > <klk@google.com.invalid
> > > > >>>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> This came up again, so I wanted to push it along by proposing a
> > > > >>> specific
> > > > >>>>> API for Java that could have a derived API in Python. I am
> > writing
> > > > >> this
> > > > >>>>> quickly to get something out there, so I welcome suggestions
> for
> > > > >>>> revision.
> > > > >>>>>
> > > > >>>>> Today a DoFn has a @ProcessElement annotated method with
> various
> > > > >>>> automated
> > > > >>>>> parameters, but most fundamentally this:
> > > > >>>>>
> > > > >>>>> @ProcessElement
> > > > >>>>> public void process(ProcessContext ctx) {
> > > > >>>>>  ctx.element() // to access the current input element
> > > > >>>>>  ctx.output(something) // to write to default output collection
> > > > >>>>>  ctx.output(tag, something) // to write to other output
> > collections
> > > > >>>>> }
> > > > >>>>>
> > > > >>>>> For some time, we have hoped to unpack the context - it is a
> > > > >>>>> backwards-compatibility pattern made obsolete by the new DoFn
> > > design.
> > > > >>> So
> > > > >>>>> instead it would look like this:
> > > > >>>>>
> > > > >>>>> @ProcessElement
> > > > >>>>> public void process(Element element, MainOutput mainOutput,
> ...)
> > {
> > > > >>>>>  element.get() // to access the current input element
> > > > >>>>>  mainOutput.output(something) // to write to the default output
> > > > >>>> collection
> > > > >>>>>  other.output(something) // to write to other output collection
> > > > >>>>> }
> > > > >>>>>
> > > > >>>>> I've deliberately left out the undecided syntax for side
> outputs.
> > > But
> > > > >>> it
> > > > >>>>> would be nice for the tag to be built in to the parameter so it
> > > > >> doesn't
> > > > >>>>> have to be used when calling output().
> > > > >>>>>
> > > > >>>>> One way to enhance this to deterministic input would just be
> > this:
> > > > >>>>>
> > > > >>>>> @ProcessElement
> > > > >>>>> @RequiresDeterministicInput
> > > > >>>>> public void process(Element element, MainOutput mainOutput,
> ...)
> > {
> > > > >>>>>  element.get() // to access the current input element
> > > > >>>>>  mainOutput.output(something) // to write to the default output
> > > > >>>> collection
> > > > >>>>>  other.output(something) // to write to other output collection
> > > > >>>>> }
> > > > >>>>>
> > > > >>>>> There are really a lot of places where we could put an
> annotation
> > > or
> > > > >>>> change
> > > > >>>>> a type to indicate that the input PCollection should be
> > > > >>>>> well-defined/deterministically-replayable. I don't have a
> really
> > > > >>> strong
> > > > >>>>> opinion.
> > > > >>>>>
> > > > >>>>> Kenn
> > > > >>>>>
> > > > >>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > > >>>> <bchambers@google.com.invalid
> > > > >>>>>>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Allowing an annotation on DoFn's that produce deterministic
> > output
> > > > >>>> could
> > > > >>>>> be
> > > > >>>>>> added in the future, but doesn't seem like a great option.
> > > > >>>>>>
> > > > >>>>>> 1. It is a correctness issue to assume a DoFn is deterministic
> > and
> > > > >> be
> > > > >>>>>> wrong, so we would need to assume all transform outputs are
> > > > >>>>>> non-deterministic unless annotated. Getting this correct is
> > > > >> difficult
> > > > >>>>> (for
> > > > >>>>>> example, GBK is surprisingly non-deterministic except in
> > specific
> > > > >>>> cases).
> > > > >>>>>>
> > > > >>>>>> 2. It is unlikely to be a major performance improvement, given
> > > that
> > > > >>> any
> > > > >>>>>> non-deterministic transform prior to a sink (which are most
> > likely
> > > > >> to
> > > > >>>>>> require deterministic input) will cause additional work to be
> > > > >> needed.
> > > > >>>>>>
> > > > >>>>>> Based on this, it seems like the risk of allowing an
> annotation
> > is
> > > > >>> high
> > > > >>>>>> while the potential for performance improvements is low. The
> > > > >> current
> > > > >>>>>> proposal (not allowing an annotation) makes sense for now,
> until
> > > we
> > > > >>> can
> > > > >>>>>> demonstrate that the impact on performance is high in cases
> that
> > > > >>> could
> > > > >>>> be
> > > > >>>>>> avoided with an annotation (in real-world use).
> > > > >>>>>>
> > > > >>>>>> -- Ben
> > > > >>>>>>
> > > > >>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com>
> > > > >> wrote:
> > > > >>>>>>
> > > > >>>>>> +1 for the general idea of runners handling it over hard-coded
> > > > >>>>>> implementation strategy.
> > > > >>>>>>
> > > > >>>>>> For the Write transform I believe you are talking about
> > > > >>>> ApplyShardingKey
> > > > >>>>>> <
> > > > >>>>>>
> > > https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > > > >>>>>> c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> > > > >>>>>> io/Write.java#L304
> > > > >>>>>>>
> > > > >>>>>> which
> > > > >>>>>> introduces non deterministic behavior when retried?
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> *Let a DoFn declare (mechanism not important right now) that
> it
> > > > >>>>>> "requiresdeterministic input"*
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> *Each runner will need a way to induce deterministic input -
> the
> > > > >>>>>> obviouschoice being a materialization.*
> > > > >>>>>>
> > > > >>>>>> Does this mean that a runner will always materialize (or
> > whatever
> > > > >> the
> > > > >>>>>> strategy is) an input PCollection to this DoFn even though the
> > > > >>>>> PCollection
> > > > >>>>>> might have been produced by deterministic transforms? Would it
> > > make
> > > > >>>> sense
> > > > >>>>>> to also let DoFns declare if they produce non-deterministic
> > > output?
> > > > >>>>>>
> > > > >>>>>> -Vikas
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On 21 March 2017 at 13:52, Stephen Sisk
> <sisk@google.com.invalid
> > >
> > > > >>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> Hey Kenn-
> > > > >>>>>>>
> > > > >>>>>>> this seems important, but I don't have all the context on
> what
> > > > >> the
> > > > >>>>>> problem
> > > > >>>>>>> is.
> > > > >>>>>>>
> > > > >>>>>>> Can you explain this sentence "Specifically, there is
> > > > >> pseudorandom
> > > > >>>> data
> > > > >>>>>>> generated and once it has been observed and used to produce a
> > > > >> side
> > > > >>>>>> effect,
> > > > >>>>>>> it cannot be regenerated without erroneous results." ?
> > > > >>>>>>>
> > > > >>>>>>> Where is the pseudorandom data coming from? Perhaps a
> concrete
> > > > >>>> example
> > > > >>>>>>> would help?
> > > > >>>>>>>
> > > > >>>>>>> S
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > > >>>> <klk@google.com.invalid
> > > > >>>>>>
> > > > >>>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Problem:
> > > > >>>>>>>>
> > > > >>>>>>>> I will drop all nuance and say that the `Write` transform as
> > it
> > > > >>>>> exists
> > > > >>>>>> in
> > > > >>>>>>>> the SDK is incorrect until we add some specification and
> APIs.
> > > > >> We
> > > > >>>>> can't
> > > > >>>>>>>> keep shipping an SDK with an unsafe transform in it, and IMO
> > > > >> this
> > > > >>>>>>> certainly
> > > > >>>>>>>> blocks a stable release.
> > > > >>>>>>>>
> > > > >>>>>>>> Specifically, there is pseudorandom data generated and once
> it
> > > > >>> has
> > > > >>>>> been
> > > > >>>>>>>> observed and used to produce a side effect, it cannot be
> > > > >>>> regenerated
> > > > >>>>>>>> without erroneous results.
> > > > >>>>>>>>
> > > > >>>>>>>> This generalizes: For some side-effecting user-defined
> > > > >> functions,
> > > > >>>> it
> > > > >>>>> is
> > > > >>>>>>>> vital that even across retries/replays they have a
> consistent
> > > > >>> view
> > > > >>>> of
> > > > >>>>>> the
> > > > >>>>>>>> contents of their input PCollection, because their effect on
> > > > >> the
> > > > >>>>>> outside
> > > > >>>>>>>> world cannot be retracted if/when they fail and are retried.
> > > > >> Once
> > > > >>>> the
> > > > >>>>>>>> runner ensures a consistent view of the input, it is then
> > their
> > > > >>> own
> > > > >>>>>>>> responsibility to be idempotent.
> > > > >>>>>>>>
> > > > >>>>>>>> Ideally we should specify this requirement for the
> > user-defined
> > > > >>>>>> function
> > > > >>>>>>>> without imposing any particular implementation strategy on
> > Beam
> > > > >>>>>> runners.
> > > > >>>>>>>>
> > > > >>>>>>>> Proposal:
> > > > >>>>>>>>
> > > > >>>>>>>> 1. Let a DoFn declare (mechanism not important right now)
> that
> > > > >> it
> > > > >>>>>>> "requires
> > > > >>>>>>>> deterministic input".
> > > > >>>>>>>>
> > > > >>>>>>>> 2. Each runner will need a way to induce deterministic
> input -
> > > > >>> the
> > > > >>>>>>> obvious
> > > > >>>>>>>> choice being a materialization.
> > > > >>>>>>>>
> > > > >>>>>>>> I want to keep the discussion focused, so I'm leaving out
> any
> > > > >>>>>>> possibilities
> > > > >>>>>>>> of taking this further.
> > > > >>>>>>>>
> > > > >>>>>>>> Regarding performance: Today places that require this tend
> to
> > > > >> be
> > > > >>>>>> already
> > > > >>>>>>>> paying the cost via GroupByKey / Reshuffle operations, since
> > > > >> that
> > > > >>>>> was a
> > > > >>>>>>>> simple way to induce determinism in batch Dataflow* (doesn't
> > > > >> work
> > > > >>>> for
> > > > >>>>>>> most
> > > > >>>>>>>> other runners nor for streaming Dataflow). This change will
> > > > >>>> replace a
> > > > >>>>>>>> hard-coded implementation strategy with a requirement that
> may
> > > > >> be
> > > > >>>>>>> fulfilled
> > > > >>>>>>>> in the most efficient way available.
> > > > >>>>>>>>
> > > > >>>>>>>> Thoughts?
> > > > >>>>>>>>
> > > > >>>>>>>> Kenn (w/ lots of consult from colleagues, especially Ben)
> > > > >>>>>>>>
> > > > >>>>>>>> * There is some overlap with the reshuffle/redistribute
> > > > >>> discussion
> > > > >>>>>>> because
> > > > >>>>>>>> of this historical situation, but I would like to leave that
> > > > >>>> broader
> > > > >>>>>>>> discussion out of this correctness issue.
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Reuven Lax <re...@google.com.INVALID>.
I don't think it really makes sense to to do this on Combine. And I agree
with you, it doesn't make sense on composites either.

On Thu, Aug 10, 2017 at 9:19 AM, Scott Wegner <sw...@google.com.invalid>
wrote:

> Does requires-stable-input only apply to ParDo transforms?
>
> I don't think it would make sense to annotate to composite, because
> checkpointing should happen as close to the side-effecting operation as
> possible, since upstream transforms within a composite could introduce
> non-determinism. So it's the primitive transform that should own the
> requirement.
>
> Are there other primitives that should be annotated? 'Combine' is
> interesting because it optimized in Dataflow (and perhaps other runners) to
> partially apply before a GroupByKey.
>
> On Thu, Aug 10, 2017 at 9:01 AM Tyler Akidau <ta...@google.com.invalid>
> wrote:
>
> > +1 to the annotation idea, and to having it on processTimer.
> >
> > -Tyler
> >
> > On Thu, Aug 10, 2017 at 2:15 AM Aljoscha Krettek <al...@apache.org>
> > wrote:
> >
> > > +1 to the annotation approach. I outlined how implementing this would
> > work
> > > in the Flink runner in the Thread about the exactly-once Kafka Sink.
> > >
> > > > On 9. Aug 2017, at 23:03, Reuven Lax <re...@google.com.INVALID>
> wrote:
> > > >
> > > > Yes - I don't think we should try and make any deterministic
> guarantees
> > > > about what is in a bundle. Stability guarantees are per element only.
> > > >
> > > > On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh <tgroh@google.com.invalid
> >
> > > > wrote:
> > > >
> > > >> +1 to the annotation-on-ProcessElement approach. ProcessElement is
> the
> > > >> minimum implementation requirement of a DoFn, and should be where
> the
> > > >> processing logic which depends on characteristics of the inputs lie.
> > > It's a
> > > >> good way of signalling the requirements of the Fn, and letting the
> > > runner
> > > >> decide.
> > > >>
> > > >> I have a minor concern that this may not work as expected for users
> > that
> > > >> try to batch remote calls in `FinishBundle` - we should make sure we
> > > >> document that it is explicitly the input elements that will be
> > replayed,
> > > >> and bundles and other operational are still arbitrary.
> > > >>
> > > >>
> > > >>
> > > >> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax
> <relax@google.com.invalid
> > >
> > > >> wrote:
> > > >>
> > > >>> I think deterministic here means deterministically replayable. i.e.
> > no
> > > >>> matter how many times the element is retried, it will always be the
> > > same.
> > > >>>
> > > >>> I think we should also allow specifying this on processTimer. This
> > > would
> > > >>> mean that any keyed state written in a previous processElement must
> > be
> > > >>> guaranteed durable before processTimer is called.
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <
> bchambers@apache.org>
> > > >>> wrote:
> > > >>>
> > > >>>> I strongly agree with this proposal. I think moving away from
> "just
> > > >>> insert
> > > >>>> a GroupByKey for one of the 3 different reasons you may want it"
> > > >> towards
> > > >>>> APIs that allow code to express the requirements they have and the
> > > >> runner
> > > >>>> to choose the best way to meet this is a major step forwards in
> > terms
> > > >> of
> > > >>>> portability.
> > > >>>>
> > > >>>> I think "deterministic" may be misleading. The actual contents of
> > the
> > > >>>> collection aren't deterministic if upstream computations aren't.
> The
> > > >>>> property we really need is that once an input may have been
> observed
> > > by
> > > >>> the
> > > >>>> side-effecting code it will never be observed with a different
> > value.
> > > >>>>
> > > >>>> I would propose something RequiresStableInput, to indicate that
> the
> > > >> input
> > > >>>> must be stable as observed by the function. I could also see
> > something
> > > >>>> hinting at the fact we don't recompute the input, such as
> > > >>>> RequiresMemoizedInput or RequiresNoRecomputation.
> > > >>>>
> > > >>>> -- Ben
> > > >>>>
> > > >>>> P.S For anyone interested other uses of GroupByKey that we may
> want
> > to
> > > >>>> discuss APIs for would be preventing retry across steps (eg.,
> > > >> preventing
> > > >>>> fusion) and redistributing inputs across workers.
> > > >>>>
> > > >>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles
> > <klk@google.com.invalid
> > > >>>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> This came up again, so I wanted to push it along by proposing a
> > > >>> specific
> > > >>>>> API for Java that could have a derived API in Python. I am
> writing
> > > >> this
> > > >>>>> quickly to get something out there, so I welcome suggestions for
> > > >>>> revision.
> > > >>>>>
> > > >>>>> Today a DoFn has a @ProcessElement annotated method with various
> > > >>>> automated
> > > >>>>> parameters, but most fundamentally this:
> > > >>>>>
> > > >>>>> @ProcessElement
> > > >>>>> public void process(ProcessContext ctx) {
> > > >>>>>  ctx.element() // to access the current input element
> > > >>>>>  ctx.output(something) // to write to default output collection
> > > >>>>>  ctx.output(tag, something) // to write to other output
> collections
> > > >>>>> }
> > > >>>>>
> > > >>>>> For some time, we have hoped to unpack the context - it is a
> > > >>>>> backwards-compatibility pattern made obsolete by the new DoFn
> > design.
> > > >>> So
> > > >>>>> instead it would look like this:
> > > >>>>>
> > > >>>>> @ProcessElement
> > > >>>>> public void process(Element element, MainOutput mainOutput, ...)
> {
> > > >>>>>  element.get() // to access the current input element
> > > >>>>>  mainOutput.output(something) // to write to the default output
> > > >>>> collection
> > > >>>>>  other.output(something) // to write to other output collection
> > > >>>>> }
> > > >>>>>
> > > >>>>> I've deliberately left out the undecided syntax for side outputs.
> > But
> > > >>> it
> > > >>>>> would be nice for the tag to be built in to the parameter so it
> > > >> doesn't
> > > >>>>> have to be used when calling output().
> > > >>>>>
> > > >>>>> One way to enhance this to deterministic input would just be
> this:
> > > >>>>>
> > > >>>>> @ProcessElement
> > > >>>>> @RequiresDeterministicInput
> > > >>>>> public void process(Element element, MainOutput mainOutput, ...)
> {
> > > >>>>>  element.get() // to access the current input element
> > > >>>>>  mainOutput.output(something) // to write to the default output
> > > >>>> collection
> > > >>>>>  other.output(something) // to write to other output collection
> > > >>>>> }
> > > >>>>>
> > > >>>>> There are really a lot of places where we could put an annotation
> > or
> > > >>>> change
> > > >>>>> a type to indicate that the input PCollection should be
> > > >>>>> well-defined/deterministically-replayable. I don't have a really
> > > >>> strong
> > > >>>>> opinion.
> > > >>>>>
> > > >>>>> Kenn
> > > >>>>>
> > > >>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > >>>> <bchambers@google.com.invalid
> > > >>>>>>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> Allowing an annotation on DoFn's that produce deterministic
> output
> > > >>>> could
> > > >>>>> be
> > > >>>>>> added in the future, but doesn't seem like a great option.
> > > >>>>>>
> > > >>>>>> 1. It is a correctness issue to assume a DoFn is deterministic
> and
> > > >> be
> > > >>>>>> wrong, so we would need to assume all transform outputs are
> > > >>>>>> non-deterministic unless annotated. Getting this correct is
> > > >> difficult
> > > >>>>> (for
> > > >>>>>> example, GBK is surprisingly non-deterministic except in
> specific
> > > >>>> cases).
> > > >>>>>>
> > > >>>>>> 2. It is unlikely to be a major performance improvement, given
> > that
> > > >>> any
> > > >>>>>> non-deterministic transform prior to a sink (which are most
> likely
> > > >> to
> > > >>>>>> require deterministic input) will cause additional work to be
> > > >> needed.
> > > >>>>>>
> > > >>>>>> Based on this, it seems like the risk of allowing an annotation
> is
> > > >>> high
> > > >>>>>> while the potential for performance improvements is low. The
> > > >> current
> > > >>>>>> proposal (not allowing an annotation) makes sense for now, until
> > we
> > > >>> can
> > > >>>>>> demonstrate that the impact on performance is high in cases that
> > > >>> could
> > > >>>> be
> > > >>>>>> avoided with an annotation (in real-world use).
> > > >>>>>>
> > > >>>>>> -- Ben
> > > >>>>>>
> > > >>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com>
> > > >> wrote:
> > > >>>>>>
> > > >>>>>> +1 for the general idea of runners handling it over hard-coded
> > > >>>>>> implementation strategy.
> > > >>>>>>
> > > >>>>>> For the Write transform I believe you are talking about
> > > >>>> ApplyShardingKey
> > > >>>>>> <
> > > >>>>>>
> > https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > > >>>>>> c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> > > >>>>>> io/Write.java#L304
> > > >>>>>>>
> > > >>>>>> which
> > > >>>>>> introduces non deterministic behavior when retried?
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> *Let a DoFn declare (mechanism not important right now) that it
> > > >>>>>> "requiresdeterministic input"*
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> *Each runner will need a way to induce deterministic input - the
> > > >>>>>> obviouschoice being a materialization.*
> > > >>>>>>
> > > >>>>>> Does this mean that a runner will always materialize (or
> whatever
> > > >> the
> > > >>>>>> strategy is) an input PCollection to this DoFn even though the
> > > >>>>> PCollection
> > > >>>>>> might have been produced by deterministic transforms? Would it
> > make
> > > >>>> sense
> > > >>>>>> to also let DoFns declare if they produce non-deterministic
> > output?
> > > >>>>>>
> > > >>>>>> -Vikas
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> On 21 March 2017 at 13:52, Stephen Sisk <sisk@google.com.invalid
> >
> > > >>>> wrote:
> > > >>>>>>
> > > >>>>>>> Hey Kenn-
> > > >>>>>>>
> > > >>>>>>> this seems important, but I don't have all the context on what
> > > >> the
> > > >>>>>> problem
> > > >>>>>>> is.
> > > >>>>>>>
> > > >>>>>>> Can you explain this sentence "Specifically, there is
> > > >> pseudorandom
> > > >>>> data
> > > >>>>>>> generated and once it has been observed and used to produce a
> > > >> side
> > > >>>>>> effect,
> > > >>>>>>> it cannot be regenerated without erroneous results." ?
> > > >>>>>>>
> > > >>>>>>> Where is the pseudorandom data coming from? Perhaps a concrete
> > > >>>> example
> > > >>>>>>> would help?
> > > >>>>>>>
> > > >>>>>>> S
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > >>>> <klk@google.com.invalid
> > > >>>>>>
> > > >>>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> Problem:
> > > >>>>>>>>
> > > >>>>>>>> I will drop all nuance and say that the `Write` transform as
> it
> > > >>>>> exists
> > > >>>>>> in
> > > >>>>>>>> the SDK is incorrect until we add some specification and APIs.
> > > >> We
> > > >>>>> can't
> > > >>>>>>>> keep shipping an SDK with an unsafe transform in it, and IMO
> > > >> this
> > > >>>>>>> certainly
> > > >>>>>>>> blocks a stable release.
> > > >>>>>>>>
> > > >>>>>>>> Specifically, there is pseudorandom data generated and once it
> > > >>> has
> > > >>>>> been
> > > >>>>>>>> observed and used to produce a side effect, it cannot be
> > > >>>> regenerated
> > > >>>>>>>> without erroneous results.
> > > >>>>>>>>
> > > >>>>>>>> This generalizes: For some side-effecting user-defined
> > > >> functions,
> > > >>>> it
> > > >>>>> is
> > > >>>>>>>> vital that even across retries/replays they have a consistent
> > > >>> view
> > > >>>> of
> > > >>>>>> the
> > > >>>>>>>> contents of their input PCollection, because their effect on
> > > >> the
> > > >>>>>> outside
> > > >>>>>>>> world cannot be retracted if/when they fail and are retried.
> > > >> Once
> > > >>>> the
> > > >>>>>>>> runner ensures a consistent view of the input, it is then
> their
> > > >>> own
> > > >>>>>>>> responsibility to be idempotent.
> > > >>>>>>>>
> > > >>>>>>>> Ideally we should specify this requirement for the
> user-defined
> > > >>>>>> function
> > > >>>>>>>> without imposing any particular implementation strategy on
> Beam
> > > >>>>>> runners.
> > > >>>>>>>>
> > > >>>>>>>> Proposal:
> > > >>>>>>>>
> > > >>>>>>>> 1. Let a DoFn declare (mechanism not important right now) that
> > > >> it
> > > >>>>>>> "requires
> > > >>>>>>>> deterministic input".
> > > >>>>>>>>
> > > >>>>>>>> 2. Each runner will need a way to induce deterministic input -
> > > >>> the
> > > >>>>>>> obvious
> > > >>>>>>>> choice being a materialization.
> > > >>>>>>>>
> > > >>>>>>>> I want to keep the discussion focused, so I'm leaving out any
> > > >>>>>>> possibilities
> > > >>>>>>>> of taking this further.
> > > >>>>>>>>
> > > >>>>>>>> Regarding performance: Today places that require this tend to
> > > >> be
> > > >>>>>> already
> > > >>>>>>>> paying the cost via GroupByKey / Reshuffle operations, since
> > > >> that
> > > >>>>> was a
> > > >>>>>>>> simple way to induce determinism in batch Dataflow* (doesn't
> > > >> work
> > > >>>> for
> > > >>>>>>> most
> > > >>>>>>>> other runners nor for streaming Dataflow). This change will
> > > >>>> replace a
> > > >>>>>>>> hard-coded implementation strategy with a requirement that may
> > > >> be
> > > >>>>>>> fulfilled
> > > >>>>>>>> in the most efficient way available.
> > > >>>>>>>>
> > > >>>>>>>> Thoughts?
> > > >>>>>>>>
> > > >>>>>>>> Kenn (w/ lots of consult from colleagues, especially Ben)
> > > >>>>>>>>
> > > >>>>>>>> * There is some overlap with the reshuffle/redistribute
> > > >>> discussion
> > > >>>>>>> because
> > > >>>>>>>> of this historical situation, but I would like to leave that
> > > >>>> broader
> > > >>>>>>>> discussion out of this correctness issue.
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >>
> > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Scott Wegner <sw...@google.com.INVALID>.
Does requires-stable-input only apply to ParDo transforms?

I don't think it would make sense to annotate to composite, because
checkpointing should happen as close to the side-effecting operation as
possible, since upstream transforms within a composite could introduce
non-determinism. So it's the primitive transform that should own the
requirement.

Are there other primitives that should be annotated? 'Combine' is
interesting because it optimized in Dataflow (and perhaps other runners) to
partially apply before a GroupByKey.

On Thu, Aug 10, 2017 at 9:01 AM Tyler Akidau <ta...@google.com.invalid>
wrote:

> +1 to the annotation idea, and to having it on processTimer.
>
> -Tyler
>
> On Thu, Aug 10, 2017 at 2:15 AM Aljoscha Krettek <al...@apache.org>
> wrote:
>
> > +1 to the annotation approach. I outlined how implementing this would
> work
> > in the Flink runner in the Thread about the exactly-once Kafka Sink.
> >
> > > On 9. Aug 2017, at 23:03, Reuven Lax <re...@google.com.INVALID> wrote:
> > >
> > > Yes - I don't think we should try and make any deterministic guarantees
> > > about what is in a bundle. Stability guarantees are per element only.
> > >
> > > On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh <tg...@google.com.invalid>
> > > wrote:
> > >
> > >> +1 to the annotation-on-ProcessElement approach. ProcessElement is the
> > >> minimum implementation requirement of a DoFn, and should be where the
> > >> processing logic which depends on characteristics of the inputs lie.
> > It's a
> > >> good way of signalling the requirements of the Fn, and letting the
> > runner
> > >> decide.
> > >>
> > >> I have a minor concern that this may not work as expected for users
> that
> > >> try to batch remote calls in `FinishBundle` - we should make sure we
> > >> document that it is explicitly the input elements that will be
> replayed,
> > >> and bundles and other operational are still arbitrary.
> > >>
> > >>
> > >>
> > >> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax <relax@google.com.invalid
> >
> > >> wrote:
> > >>
> > >>> I think deterministic here means deterministically replayable. i.e.
> no
> > >>> matter how many times the element is retried, it will always be the
> > same.
> > >>>
> > >>> I think we should also allow specifying this on processTimer. This
> > would
> > >>> mean that any keyed state written in a previous processElement must
> be
> > >>> guaranteed durable before processTimer is called.
> > >>>
> > >>>
> > >>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <bc...@apache.org>
> > >>> wrote:
> > >>>
> > >>>> I strongly agree with this proposal. I think moving away from "just
> > >>> insert
> > >>>> a GroupByKey for one of the 3 different reasons you may want it"
> > >> towards
> > >>>> APIs that allow code to express the requirements they have and the
> > >> runner
> > >>>> to choose the best way to meet this is a major step forwards in
> terms
> > >> of
> > >>>> portability.
> > >>>>
> > >>>> I think "deterministic" may be misleading. The actual contents of
> the
> > >>>> collection aren't deterministic if upstream computations aren't. The
> > >>>> property we really need is that once an input may have been observed
> > by
> > >>> the
> > >>>> side-effecting code it will never be observed with a different
> value.
> > >>>>
> > >>>> I would propose something RequiresStableInput, to indicate that the
> > >> input
> > >>>> must be stable as observed by the function. I could also see
> something
> > >>>> hinting at the fact we don't recompute the input, such as
> > >>>> RequiresMemoizedInput or RequiresNoRecomputation.
> > >>>>
> > >>>> -- Ben
> > >>>>
> > >>>> P.S For anyone interested other uses of GroupByKey that we may want
> to
> > >>>> discuss APIs for would be preventing retry across steps (eg.,
> > >> preventing
> > >>>> fusion) and redistributing inputs across workers.
> > >>>>
> > >>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles
> <klk@google.com.invalid
> > >>>
> > >>>> wrote:
> > >>>>
> > >>>>> This came up again, so I wanted to push it along by proposing a
> > >>> specific
> > >>>>> API for Java that could have a derived API in Python. I am writing
> > >> this
> > >>>>> quickly to get something out there, so I welcome suggestions for
> > >>>> revision.
> > >>>>>
> > >>>>> Today a DoFn has a @ProcessElement annotated method with various
> > >>>> automated
> > >>>>> parameters, but most fundamentally this:
> > >>>>>
> > >>>>> @ProcessElement
> > >>>>> public void process(ProcessContext ctx) {
> > >>>>>  ctx.element() // to access the current input element
> > >>>>>  ctx.output(something) // to write to default output collection
> > >>>>>  ctx.output(tag, something) // to write to other output collections
> > >>>>> }
> > >>>>>
> > >>>>> For some time, we have hoped to unpack the context - it is a
> > >>>>> backwards-compatibility pattern made obsolete by the new DoFn
> design.
> > >>> So
> > >>>>> instead it would look like this:
> > >>>>>
> > >>>>> @ProcessElement
> > >>>>> public void process(Element element, MainOutput mainOutput, ...) {
> > >>>>>  element.get() // to access the current input element
> > >>>>>  mainOutput.output(something) // to write to the default output
> > >>>> collection
> > >>>>>  other.output(something) // to write to other output collection
> > >>>>> }
> > >>>>>
> > >>>>> I've deliberately left out the undecided syntax for side outputs.
> But
> > >>> it
> > >>>>> would be nice for the tag to be built in to the parameter so it
> > >> doesn't
> > >>>>> have to be used when calling output().
> > >>>>>
> > >>>>> One way to enhance this to deterministic input would just be this:
> > >>>>>
> > >>>>> @ProcessElement
> > >>>>> @RequiresDeterministicInput
> > >>>>> public void process(Element element, MainOutput mainOutput, ...) {
> > >>>>>  element.get() // to access the current input element
> > >>>>>  mainOutput.output(something) // to write to the default output
> > >>>> collection
> > >>>>>  other.output(something) // to write to other output collection
> > >>>>> }
> > >>>>>
> > >>>>> There are really a lot of places where we could put an annotation
> or
> > >>>> change
> > >>>>> a type to indicate that the input PCollection should be
> > >>>>> well-defined/deterministically-replayable. I don't have a really
> > >>> strong
> > >>>>> opinion.
> > >>>>>
> > >>>>> Kenn
> > >>>>>
> > >>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > >>>> <bchambers@google.com.invalid
> > >>>>>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Allowing an annotation on DoFn's that produce deterministic output
> > >>>> could
> > >>>>> be
> > >>>>>> added in the future, but doesn't seem like a great option.
> > >>>>>>
> > >>>>>> 1. It is a correctness issue to assume a DoFn is deterministic and
> > >> be
> > >>>>>> wrong, so we would need to assume all transform outputs are
> > >>>>>> non-deterministic unless annotated. Getting this correct is
> > >> difficult
> > >>>>> (for
> > >>>>>> example, GBK is surprisingly non-deterministic except in specific
> > >>>> cases).
> > >>>>>>
> > >>>>>> 2. It is unlikely to be a major performance improvement, given
> that
> > >>> any
> > >>>>>> non-deterministic transform prior to a sink (which are most likely
> > >> to
> > >>>>>> require deterministic input) will cause additional work to be
> > >> needed.
> > >>>>>>
> > >>>>>> Based on this, it seems like the risk of allowing an annotation is
> > >>> high
> > >>>>>> while the potential for performance improvements is low. The
> > >> current
> > >>>>>> proposal (not allowing an annotation) makes sense for now, until
> we
> > >>> can
> > >>>>>> demonstrate that the impact on performance is high in cases that
> > >>> could
> > >>>> be
> > >>>>>> avoided with an annotation (in real-world use).
> > >>>>>>
> > >>>>>> -- Ben
> > >>>>>>
> > >>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com>
> > >> wrote:
> > >>>>>>
> > >>>>>> +1 for the general idea of runners handling it over hard-coded
> > >>>>>> implementation strategy.
> > >>>>>>
> > >>>>>> For the Write transform I believe you are talking about
> > >>>> ApplyShardingKey
> > >>>>>> <
> > >>>>>>
> https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > >>>>>> c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> > >>>>>> io/Write.java#L304
> > >>>>>>>
> > >>>>>> which
> > >>>>>> introduces non deterministic behavior when retried?
> > >>>>>>
> > >>>>>>
> > >>>>>> *Let a DoFn declare (mechanism not important right now) that it
> > >>>>>> "requiresdeterministic input"*
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> *Each runner will need a way to induce deterministic input - the
> > >>>>>> obviouschoice being a materialization.*
> > >>>>>>
> > >>>>>> Does this mean that a runner will always materialize (or whatever
> > >> the
> > >>>>>> strategy is) an input PCollection to this DoFn even though the
> > >>>>> PCollection
> > >>>>>> might have been produced by deterministic transforms? Would it
> make
> > >>>> sense
> > >>>>>> to also let DoFns declare if they produce non-deterministic
> output?
> > >>>>>>
> > >>>>>> -Vikas
> > >>>>>>
> > >>>>>>
> > >>>>>> On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid>
> > >>>> wrote:
> > >>>>>>
> > >>>>>>> Hey Kenn-
> > >>>>>>>
> > >>>>>>> this seems important, but I don't have all the context on what
> > >> the
> > >>>>>> problem
> > >>>>>>> is.
> > >>>>>>>
> > >>>>>>> Can you explain this sentence "Specifically, there is
> > >> pseudorandom
> > >>>> data
> > >>>>>>> generated and once it has been observed and used to produce a
> > >> side
> > >>>>>> effect,
> > >>>>>>> it cannot be regenerated without erroneous results." ?
> > >>>>>>>
> > >>>>>>> Where is the pseudorandom data coming from? Perhaps a concrete
> > >>>> example
> > >>>>>>> would help?
> > >>>>>>>
> > >>>>>>> S
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > >>>> <klk@google.com.invalid
> > >>>>>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Problem:
> > >>>>>>>>
> > >>>>>>>> I will drop all nuance and say that the `Write` transform as it
> > >>>>> exists
> > >>>>>> in
> > >>>>>>>> the SDK is incorrect until we add some specification and APIs.
> > >> We
> > >>>>> can't
> > >>>>>>>> keep shipping an SDK with an unsafe transform in it, and IMO
> > >> this
> > >>>>>>> certainly
> > >>>>>>>> blocks a stable release.
> > >>>>>>>>
> > >>>>>>>> Specifically, there is pseudorandom data generated and once it
> > >>> has
> > >>>>> been
> > >>>>>>>> observed and used to produce a side effect, it cannot be
> > >>>> regenerated
> > >>>>>>>> without erroneous results.
> > >>>>>>>>
> > >>>>>>>> This generalizes: For some side-effecting user-defined
> > >> functions,
> > >>>> it
> > >>>>> is
> > >>>>>>>> vital that even across retries/replays they have a consistent
> > >>> view
> > >>>> of
> > >>>>>> the
> > >>>>>>>> contents of their input PCollection, because their effect on
> > >> the
> > >>>>>> outside
> > >>>>>>>> world cannot be retracted if/when they fail and are retried.
> > >> Once
> > >>>> the
> > >>>>>>>> runner ensures a consistent view of the input, it is then their
> > >>> own
> > >>>>>>>> responsibility to be idempotent.
> > >>>>>>>>
> > >>>>>>>> Ideally we should specify this requirement for the user-defined
> > >>>>>> function
> > >>>>>>>> without imposing any particular implementation strategy on Beam
> > >>>>>> runners.
> > >>>>>>>>
> > >>>>>>>> Proposal:
> > >>>>>>>>
> > >>>>>>>> 1. Let a DoFn declare (mechanism not important right now) that
> > >> it
> > >>>>>>> "requires
> > >>>>>>>> deterministic input".
> > >>>>>>>>
> > >>>>>>>> 2. Each runner will need a way to induce deterministic input -
> > >>> the
> > >>>>>>> obvious
> > >>>>>>>> choice being a materialization.
> > >>>>>>>>
> > >>>>>>>> I want to keep the discussion focused, so I'm leaving out any
> > >>>>>>> possibilities
> > >>>>>>>> of taking this further.
> > >>>>>>>>
> > >>>>>>>> Regarding performance: Today places that require this tend to
> > >> be
> > >>>>>> already
> > >>>>>>>> paying the cost via GroupByKey / Reshuffle operations, since
> > >> that
> > >>>>> was a
> > >>>>>>>> simple way to induce determinism in batch Dataflow* (doesn't
> > >> work
> > >>>> for
> > >>>>>>> most
> > >>>>>>>> other runners nor for streaming Dataflow). This change will
> > >>>> replace a
> > >>>>>>>> hard-coded implementation strategy with a requirement that may
> > >> be
> > >>>>>>> fulfilled
> > >>>>>>>> in the most efficient way available.
> > >>>>>>>>
> > >>>>>>>> Thoughts?
> > >>>>>>>>
> > >>>>>>>> Kenn (w/ lots of consult from colleagues, especially Ben)
> > >>>>>>>>
> > >>>>>>>> * There is some overlap with the reshuffle/redistribute
> > >>> discussion
> > >>>>>>> because
> > >>>>>>>> of this historical situation, but I would like to leave that
> > >>>> broader
> > >>>>>>>> discussion out of this correctness issue.
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Tyler Akidau <ta...@google.com.INVALID>.
+1 to the annotation idea, and to having it on processTimer.

-Tyler

On Thu, Aug 10, 2017 at 2:15 AM Aljoscha Krettek <al...@apache.org>
wrote:

> +1 to the annotation approach. I outlined how implementing this would work
> in the Flink runner in the Thread about the exactly-once Kafka Sink.
>
> > On 9. Aug 2017, at 23:03, Reuven Lax <re...@google.com.INVALID> wrote:
> >
> > Yes - I don't think we should try and make any deterministic guarantees
> > about what is in a bundle. Stability guarantees are per element only.
> >
> > On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh <tg...@google.com.invalid>
> > wrote:
> >
> >> +1 to the annotation-on-ProcessElement approach. ProcessElement is the
> >> minimum implementation requirement of a DoFn, and should be where the
> >> processing logic which depends on characteristics of the inputs lie.
> It's a
> >> good way of signalling the requirements of the Fn, and letting the
> runner
> >> decide.
> >>
> >> I have a minor concern that this may not work as expected for users that
> >> try to batch remote calls in `FinishBundle` - we should make sure we
> >> document that it is explicitly the input elements that will be replayed,
> >> and bundles and other operational are still arbitrary.
> >>
> >>
> >>
> >> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax <re...@google.com.invalid>
> >> wrote:
> >>
> >>> I think deterministic here means deterministically replayable. i.e. no
> >>> matter how many times the element is retried, it will always be the
> same.
> >>>
> >>> I think we should also allow specifying this on processTimer. This
> would
> >>> mean that any keyed state written in a previous processElement must be
> >>> guaranteed durable before processTimer is called.
> >>>
> >>>
> >>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <bc...@apache.org>
> >>> wrote:
> >>>
> >>>> I strongly agree with this proposal. I think moving away from "just
> >>> insert
> >>>> a GroupByKey for one of the 3 different reasons you may want it"
> >> towards
> >>>> APIs that allow code to express the requirements they have and the
> >> runner
> >>>> to choose the best way to meet this is a major step forwards in terms
> >> of
> >>>> portability.
> >>>>
> >>>> I think "deterministic" may be misleading. The actual contents of the
> >>>> collection aren't deterministic if upstream computations aren't. The
> >>>> property we really need is that once an input may have been observed
> by
> >>> the
> >>>> side-effecting code it will never be observed with a different value.
> >>>>
> >>>> I would propose something RequiresStableInput, to indicate that the
> >> input
> >>>> must be stable as observed by the function. I could also see something
> >>>> hinting at the fact we don't recompute the input, such as
> >>>> RequiresMemoizedInput or RequiresNoRecomputation.
> >>>>
> >>>> -- Ben
> >>>>
> >>>> P.S For anyone interested other uses of GroupByKey that we may want to
> >>>> discuss APIs for would be preventing retry across steps (eg.,
> >> preventing
> >>>> fusion) and redistributing inputs across workers.
> >>>>
> >>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles <klk@google.com.invalid
> >>>
> >>>> wrote:
> >>>>
> >>>>> This came up again, so I wanted to push it along by proposing a
> >>> specific
> >>>>> API for Java that could have a derived API in Python. I am writing
> >> this
> >>>>> quickly to get something out there, so I welcome suggestions for
> >>>> revision.
> >>>>>
> >>>>> Today a DoFn has a @ProcessElement annotated method with various
> >>>> automated
> >>>>> parameters, but most fundamentally this:
> >>>>>
> >>>>> @ProcessElement
> >>>>> public void process(ProcessContext ctx) {
> >>>>>  ctx.element() // to access the current input element
> >>>>>  ctx.output(something) // to write to default output collection
> >>>>>  ctx.output(tag, something) // to write to other output collections
> >>>>> }
> >>>>>
> >>>>> For some time, we have hoped to unpack the context - it is a
> >>>>> backwards-compatibility pattern made obsolete by the new DoFn design.
> >>> So
> >>>>> instead it would look like this:
> >>>>>
> >>>>> @ProcessElement
> >>>>> public void process(Element element, MainOutput mainOutput, ...) {
> >>>>>  element.get() // to access the current input element
> >>>>>  mainOutput.output(something) // to write to the default output
> >>>> collection
> >>>>>  other.output(something) // to write to other output collection
> >>>>> }
> >>>>>
> >>>>> I've deliberately left out the undecided syntax for side outputs. But
> >>> it
> >>>>> would be nice for the tag to be built in to the parameter so it
> >> doesn't
> >>>>> have to be used when calling output().
> >>>>>
> >>>>> One way to enhance this to deterministic input would just be this:
> >>>>>
> >>>>> @ProcessElement
> >>>>> @RequiresDeterministicInput
> >>>>> public void process(Element element, MainOutput mainOutput, ...) {
> >>>>>  element.get() // to access the current input element
> >>>>>  mainOutput.output(something) // to write to the default output
> >>>> collection
> >>>>>  other.output(something) // to write to other output collection
> >>>>> }
> >>>>>
> >>>>> There are really a lot of places where we could put an annotation or
> >>>> change
> >>>>> a type to indicate that the input PCollection should be
> >>>>> well-defined/deterministically-replayable. I don't have a really
> >>> strong
> >>>>> opinion.
> >>>>>
> >>>>> Kenn
> >>>>>
> >>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> >>>> <bchambers@google.com.invalid
> >>>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Allowing an annotation on DoFn's that produce deterministic output
> >>>> could
> >>>>> be
> >>>>>> added in the future, but doesn't seem like a great option.
> >>>>>>
> >>>>>> 1. It is a correctness issue to assume a DoFn is deterministic and
> >> be
> >>>>>> wrong, so we would need to assume all transform outputs are
> >>>>>> non-deterministic unless annotated. Getting this correct is
> >> difficult
> >>>>> (for
> >>>>>> example, GBK is surprisingly non-deterministic except in specific
> >>>> cases).
> >>>>>>
> >>>>>> 2. It is unlikely to be a major performance improvement, given that
> >>> any
> >>>>>> non-deterministic transform prior to a sink (which are most likely
> >> to
> >>>>>> require deterministic input) will cause additional work to be
> >> needed.
> >>>>>>
> >>>>>> Based on this, it seems like the risk of allowing an annotation is
> >>> high
> >>>>>> while the potential for performance improvements is low. The
> >> current
> >>>>>> proposal (not allowing an annotation) makes sense for now, until we
> >>> can
> >>>>>> demonstrate that the impact on performance is high in cases that
> >>> could
> >>>> be
> >>>>>> avoided with an annotation (in real-world use).
> >>>>>>
> >>>>>> -- Ben
> >>>>>>
> >>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com>
> >> wrote:
> >>>>>>
> >>>>>> +1 for the general idea of runners handling it over hard-coded
> >>>>>> implementation strategy.
> >>>>>>
> >>>>>> For the Write transform I believe you are talking about
> >>>> ApplyShardingKey
> >>>>>> <
> >>>>>> https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> >>>>>> c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> >>>>>> io/Write.java#L304
> >>>>>>>
> >>>>>> which
> >>>>>> introduces non deterministic behavior when retried?
> >>>>>>
> >>>>>>
> >>>>>> *Let a DoFn declare (mechanism not important right now) that it
> >>>>>> "requiresdeterministic input"*
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> *Each runner will need a way to induce deterministic input - the
> >>>>>> obviouschoice being a materialization.*
> >>>>>>
> >>>>>> Does this mean that a runner will always materialize (or whatever
> >> the
> >>>>>> strategy is) an input PCollection to this DoFn even though the
> >>>>> PCollection
> >>>>>> might have been produced by deterministic transforms? Would it make
> >>>> sense
> >>>>>> to also let DoFns declare if they produce non-deterministic output?
> >>>>>>
> >>>>>> -Vikas
> >>>>>>
> >>>>>>
> >>>>>> On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid>
> >>>> wrote:
> >>>>>>
> >>>>>>> Hey Kenn-
> >>>>>>>
> >>>>>>> this seems important, but I don't have all the context on what
> >> the
> >>>>>> problem
> >>>>>>> is.
> >>>>>>>
> >>>>>>> Can you explain this sentence "Specifically, there is
> >> pseudorandom
> >>>> data
> >>>>>>> generated and once it has been observed and used to produce a
> >> side
> >>>>>> effect,
> >>>>>>> it cannot be regenerated without erroneous results." ?
> >>>>>>>
> >>>>>>> Where is the pseudorandom data coming from? Perhaps a concrete
> >>>> example
> >>>>>>> would help?
> >>>>>>>
> >>>>>>> S
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> >>>> <klk@google.com.invalid
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Problem:
> >>>>>>>>
> >>>>>>>> I will drop all nuance and say that the `Write` transform as it
> >>>>> exists
> >>>>>> in
> >>>>>>>> the SDK is incorrect until we add some specification and APIs.
> >> We
> >>>>> can't
> >>>>>>>> keep shipping an SDK with an unsafe transform in it, and IMO
> >> this
> >>>>>>> certainly
> >>>>>>>> blocks a stable release.
> >>>>>>>>
> >>>>>>>> Specifically, there is pseudorandom data generated and once it
> >>> has
> >>>>> been
> >>>>>>>> observed and used to produce a side effect, it cannot be
> >>>> regenerated
> >>>>>>>> without erroneous results.
> >>>>>>>>
> >>>>>>>> This generalizes: For some side-effecting user-defined
> >> functions,
> >>>> it
> >>>>> is
> >>>>>>>> vital that even across retries/replays they have a consistent
> >>> view
> >>>> of
> >>>>>> the
> >>>>>>>> contents of their input PCollection, because their effect on
> >> the
> >>>>>> outside
> >>>>>>>> world cannot be retracted if/when they fail and are retried.
> >> Once
> >>>> the
> >>>>>>>> runner ensures a consistent view of the input, it is then their
> >>> own
> >>>>>>>> responsibility to be idempotent.
> >>>>>>>>
> >>>>>>>> Ideally we should specify this requirement for the user-defined
> >>>>>> function
> >>>>>>>> without imposing any particular implementation strategy on Beam
> >>>>>> runners.
> >>>>>>>>
> >>>>>>>> Proposal:
> >>>>>>>>
> >>>>>>>> 1. Let a DoFn declare (mechanism not important right now) that
> >> it
> >>>>>>> "requires
> >>>>>>>> deterministic input".
> >>>>>>>>
> >>>>>>>> 2. Each runner will need a way to induce deterministic input -
> >>> the
> >>>>>>> obvious
> >>>>>>>> choice being a materialization.
> >>>>>>>>
> >>>>>>>> I want to keep the discussion focused, so I'm leaving out any
> >>>>>>> possibilities
> >>>>>>>> of taking this further.
> >>>>>>>>
> >>>>>>>> Regarding performance: Today places that require this tend to
> >> be
> >>>>>> already
> >>>>>>>> paying the cost via GroupByKey / Reshuffle operations, since
> >> that
> >>>>> was a
> >>>>>>>> simple way to induce determinism in batch Dataflow* (doesn't
> >> work
> >>>> for
> >>>>>>> most
> >>>>>>>> other runners nor for streaming Dataflow). This change will
> >>>> replace a
> >>>>>>>> hard-coded implementation strategy with a requirement that may
> >> be
> >>>>>>> fulfilled
> >>>>>>>> in the most efficient way available.
> >>>>>>>>
> >>>>>>>> Thoughts?
> >>>>>>>>
> >>>>>>>> Kenn (w/ lots of consult from colleagues, especially Ben)
> >>>>>>>>
> >>>>>>>> * There is some overlap with the reshuffle/redistribute
> >>> discussion
> >>>>>>> because
> >>>>>>>> of this historical situation, but I would like to leave that
> >>>> broader
> >>>>>>>> discussion out of this correctness issue.
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Aljoscha Krettek <al...@apache.org>.
+1 to the annotation approach. I outlined how implementing this would work in the Flink runner in the Thread about the exactly-once Kafka Sink.

> On 9. Aug 2017, at 23:03, Reuven Lax <re...@google.com.INVALID> wrote:
> 
> Yes - I don't think we should try and make any deterministic guarantees
> about what is in a bundle. Stability guarantees are per element only.
> 
> On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh <tg...@google.com.invalid>
> wrote:
> 
>> +1 to the annotation-on-ProcessElement approach. ProcessElement is the
>> minimum implementation requirement of a DoFn, and should be where the
>> processing logic which depends on characteristics of the inputs lie. It's a
>> good way of signalling the requirements of the Fn, and letting the runner
>> decide.
>> 
>> I have a minor concern that this may not work as expected for users that
>> try to batch remote calls in `FinishBundle` - we should make sure we
>> document that it is explicitly the input elements that will be replayed,
>> and bundles and other operational are still arbitrary.
>> 
>> 
>> 
>> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax <re...@google.com.invalid>
>> wrote:
>> 
>>> I think deterministic here means deterministically replayable. i.e. no
>>> matter how many times the element is retried, it will always be the same.
>>> 
>>> I think we should also allow specifying this on processTimer. This would
>>> mean that any keyed state written in a previous processElement must be
>>> guaranteed durable before processTimer is called.
>>> 
>>> 
>>> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <bc...@apache.org>
>>> wrote:
>>> 
>>>> I strongly agree with this proposal. I think moving away from "just
>>> insert
>>>> a GroupByKey for one of the 3 different reasons you may want it"
>> towards
>>>> APIs that allow code to express the requirements they have and the
>> runner
>>>> to choose the best way to meet this is a major step forwards in terms
>> of
>>>> portability.
>>>> 
>>>> I think "deterministic" may be misleading. The actual contents of the
>>>> collection aren't deterministic if upstream computations aren't. The
>>>> property we really need is that once an input may have been observed by
>>> the
>>>> side-effecting code it will never be observed with a different value.
>>>> 
>>>> I would propose something RequiresStableInput, to indicate that the
>> input
>>>> must be stable as observed by the function. I could also see something
>>>> hinting at the fact we don't recompute the input, such as
>>>> RequiresMemoizedInput or RequiresNoRecomputation.
>>>> 
>>>> -- Ben
>>>> 
>>>> P.S For anyone interested other uses of GroupByKey that we may want to
>>>> discuss APIs for would be preventing retry across steps (eg.,
>> preventing
>>>> fusion) and redistributing inputs across workers.
>>>> 
>>>> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles <klk@google.com.invalid
>>> 
>>>> wrote:
>>>> 
>>>>> This came up again, so I wanted to push it along by proposing a
>>> specific
>>>>> API for Java that could have a derived API in Python. I am writing
>> this
>>>>> quickly to get something out there, so I welcome suggestions for
>>>> revision.
>>>>> 
>>>>> Today a DoFn has a @ProcessElement annotated method with various
>>>> automated
>>>>> parameters, but most fundamentally this:
>>>>> 
>>>>> @ProcessElement
>>>>> public void process(ProcessContext ctx) {
>>>>>  ctx.element() // to access the current input element
>>>>>  ctx.output(something) // to write to default output collection
>>>>>  ctx.output(tag, something) // to write to other output collections
>>>>> }
>>>>> 
>>>>> For some time, we have hoped to unpack the context - it is a
>>>>> backwards-compatibility pattern made obsolete by the new DoFn design.
>>> So
>>>>> instead it would look like this:
>>>>> 
>>>>> @ProcessElement
>>>>> public void process(Element element, MainOutput mainOutput, ...) {
>>>>>  element.get() // to access the current input element
>>>>>  mainOutput.output(something) // to write to the default output
>>>> collection
>>>>>  other.output(something) // to write to other output collection
>>>>> }
>>>>> 
>>>>> I've deliberately left out the undecided syntax for side outputs. But
>>> it
>>>>> would be nice for the tag to be built in to the parameter so it
>> doesn't
>>>>> have to be used when calling output().
>>>>> 
>>>>> One way to enhance this to deterministic input would just be this:
>>>>> 
>>>>> @ProcessElement
>>>>> @RequiresDeterministicInput
>>>>> public void process(Element element, MainOutput mainOutput, ...) {
>>>>>  element.get() // to access the current input element
>>>>>  mainOutput.output(something) // to write to the default output
>>>> collection
>>>>>  other.output(something) // to write to other output collection
>>>>> }
>>>>> 
>>>>> There are really a lot of places where we could put an annotation or
>>>> change
>>>>> a type to indicate that the input PCollection should be
>>>>> well-defined/deterministically-replayable. I don't have a really
>>> strong
>>>>> opinion.
>>>>> 
>>>>> Kenn
>>>>> 
>>>>> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
>>>> <bchambers@google.com.invalid
>>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> Allowing an annotation on DoFn's that produce deterministic output
>>>> could
>>>>> be
>>>>>> added in the future, but doesn't seem like a great option.
>>>>>> 
>>>>>> 1. It is a correctness issue to assume a DoFn is deterministic and
>> be
>>>>>> wrong, so we would need to assume all transform outputs are
>>>>>> non-deterministic unless annotated. Getting this correct is
>> difficult
>>>>> (for
>>>>>> example, GBK is surprisingly non-deterministic except in specific
>>>> cases).
>>>>>> 
>>>>>> 2. It is unlikely to be a major performance improvement, given that
>>> any
>>>>>> non-deterministic transform prior to a sink (which are most likely
>> to
>>>>>> require deterministic input) will cause additional work to be
>> needed.
>>>>>> 
>>>>>> Based on this, it seems like the risk of allowing an annotation is
>>> high
>>>>>> while the potential for performance improvements is low. The
>> current
>>>>>> proposal (not allowing an annotation) makes sense for now, until we
>>> can
>>>>>> demonstrate that the impact on performance is high in cases that
>>> could
>>>> be
>>>>>> avoided with an annotation (in real-world use).
>>>>>> 
>>>>>> -- Ben
>>>>>> 
>>>>>> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com>
>> wrote:
>>>>>> 
>>>>>> +1 for the general idea of runners handling it over hard-coded
>>>>>> implementation strategy.
>>>>>> 
>>>>>> For the Write transform I believe you are talking about
>>>> ApplyShardingKey
>>>>>> <
>>>>>> https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
>>>>>> c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
>>>>>> io/Write.java#L304
>>>>>>> 
>>>>>> which
>>>>>> introduces non deterministic behavior when retried?
>>>>>> 
>>>>>> 
>>>>>> *Let a DoFn declare (mechanism not important right now) that it
>>>>>> "requiresdeterministic input"*
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> *Each runner will need a way to induce deterministic input - the
>>>>>> obviouschoice being a materialization.*
>>>>>> 
>>>>>> Does this mean that a runner will always materialize (or whatever
>> the
>>>>>> strategy is) an input PCollection to this DoFn even though the
>>>>> PCollection
>>>>>> might have been produced by deterministic transforms? Would it make
>>>> sense
>>>>>> to also let DoFns declare if they produce non-deterministic output?
>>>>>> 
>>>>>> -Vikas
>>>>>> 
>>>>>> 
>>>>>> On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid>
>>>> wrote:
>>>>>> 
>>>>>>> Hey Kenn-
>>>>>>> 
>>>>>>> this seems important, but I don't have all the context on what
>> the
>>>>>> problem
>>>>>>> is.
>>>>>>> 
>>>>>>> Can you explain this sentence "Specifically, there is
>> pseudorandom
>>>> data
>>>>>>> generated and once it has been observed and used to produce a
>> side
>>>>>> effect,
>>>>>>> it cannot be regenerated without erroneous results." ?
>>>>>>> 
>>>>>>> Where is the pseudorandom data coming from? Perhaps a concrete
>>>> example
>>>>>>> would help?
>>>>>>> 
>>>>>>> S
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
>>>> <klk@google.com.invalid
>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Problem:
>>>>>>>> 
>>>>>>>> I will drop all nuance and say that the `Write` transform as it
>>>>> exists
>>>>>> in
>>>>>>>> the SDK is incorrect until we add some specification and APIs.
>> We
>>>>> can't
>>>>>>>> keep shipping an SDK with an unsafe transform in it, and IMO
>> this
>>>>>>> certainly
>>>>>>>> blocks a stable release.
>>>>>>>> 
>>>>>>>> Specifically, there is pseudorandom data generated and once it
>>> has
>>>>> been
>>>>>>>> observed and used to produce a side effect, it cannot be
>>>> regenerated
>>>>>>>> without erroneous results.
>>>>>>>> 
>>>>>>>> This generalizes: For some side-effecting user-defined
>> functions,
>>>> it
>>>>> is
>>>>>>>> vital that even across retries/replays they have a consistent
>>> view
>>>> of
>>>>>> the
>>>>>>>> contents of their input PCollection, because their effect on
>> the
>>>>>> outside
>>>>>>>> world cannot be retracted if/when they fail and are retried.
>> Once
>>>> the
>>>>>>>> runner ensures a consistent view of the input, it is then their
>>> own
>>>>>>>> responsibility to be idempotent.
>>>>>>>> 
>>>>>>>> Ideally we should specify this requirement for the user-defined
>>>>>> function
>>>>>>>> without imposing any particular implementation strategy on Beam
>>>>>> runners.
>>>>>>>> 
>>>>>>>> Proposal:
>>>>>>>> 
>>>>>>>> 1. Let a DoFn declare (mechanism not important right now) that
>> it
>>>>>>> "requires
>>>>>>>> deterministic input".
>>>>>>>> 
>>>>>>>> 2. Each runner will need a way to induce deterministic input -
>>> the
>>>>>>> obvious
>>>>>>>> choice being a materialization.
>>>>>>>> 
>>>>>>>> I want to keep the discussion focused, so I'm leaving out any
>>>>>>> possibilities
>>>>>>>> of taking this further.
>>>>>>>> 
>>>>>>>> Regarding performance: Today places that require this tend to
>> be
>>>>>> already
>>>>>>>> paying the cost via GroupByKey / Reshuffle operations, since
>> that
>>>>> was a
>>>>>>>> simple way to induce determinism in batch Dataflow* (doesn't
>> work
>>>> for
>>>>>>> most
>>>>>>>> other runners nor for streaming Dataflow). This change will
>>>> replace a
>>>>>>>> hard-coded implementation strategy with a requirement that may
>> be
>>>>>>> fulfilled
>>>>>>>> in the most efficient way available.
>>>>>>>> 
>>>>>>>> Thoughts?
>>>>>>>> 
>>>>>>>> Kenn (w/ lots of consult from colleagues, especially Ben)
>>>>>>>> 
>>>>>>>> * There is some overlap with the reshuffle/redistribute
>>> discussion
>>>>>>> because
>>>>>>>> of this historical situation, but I would like to leave that
>>>> broader
>>>>>>>> discussion out of this correctness issue.
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 


Re: [PROPOSAL] "Requires deterministic input"

Posted by Reuven Lax <re...@google.com.INVALID>.
Yes - I don't think we should try and make any deterministic guarantees
about what is in a bundle. Stability guarantees are per element only.

On Wed, Aug 9, 2017 at 1:30 PM, Thomas Groh <tg...@google.com.invalid>
wrote:

> +1 to the annotation-on-ProcessElement approach. ProcessElement is the
> minimum implementation requirement of a DoFn, and should be where the
> processing logic which depends on characteristics of the inputs lie. It's a
> good way of signalling the requirements of the Fn, and letting the runner
> decide.
>
> I have a minor concern that this may not work as expected for users that
> try to batch remote calls in `FinishBundle` - we should make sure we
> document that it is explicitly the input elements that will be replayed,
> and bundles and other operational are still arbitrary.
>
>
>
> On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax <re...@google.com.invalid>
> wrote:
>
> > I think deterministic here means deterministically replayable. i.e. no
> > matter how many times the element is retried, it will always be the same.
> >
> > I think we should also allow specifying this on processTimer. This would
> > mean that any keyed state written in a previous processElement must be
> > guaranteed durable before processTimer is called.
> >
> >
> > On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <bc...@apache.org>
> > wrote:
> >
> > > I strongly agree with this proposal. I think moving away from "just
> > insert
> > > a GroupByKey for one of the 3 different reasons you may want it"
> towards
> > > APIs that allow code to express the requirements they have and the
> runner
> > > to choose the best way to meet this is a major step forwards in terms
> of
> > > portability.
> > >
> > > I think "deterministic" may be misleading. The actual contents of the
> > > collection aren't deterministic if upstream computations aren't. The
> > > property we really need is that once an input may have been observed by
> > the
> > > side-effecting code it will never be observed with a different value.
> > >
> > > I would propose something RequiresStableInput, to indicate that the
> input
> > > must be stable as observed by the function. I could also see something
> > > hinting at the fact we don't recompute the input, such as
> > > RequiresMemoizedInput or RequiresNoRecomputation.
> > >
> > > -- Ben
> > >
> > > P.S For anyone interested other uses of GroupByKey that we may want to
> > > discuss APIs for would be preventing retry across steps (eg.,
> preventing
> > > fusion) and redistributing inputs across workers.
> > >
> > > On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles <klk@google.com.invalid
> >
> > > wrote:
> > >
> > > > This came up again, so I wanted to push it along by proposing a
> > specific
> > > > API for Java that could have a derived API in Python. I am writing
> this
> > > > quickly to get something out there, so I welcome suggestions for
> > > revision.
> > > >
> > > > Today a DoFn has a @ProcessElement annotated method with various
> > > automated
> > > > parameters, but most fundamentally this:
> > > >
> > > > @ProcessElement
> > > > public void process(ProcessContext ctx) {
> > > >   ctx.element() // to access the current input element
> > > >   ctx.output(something) // to write to default output collection
> > > >   ctx.output(tag, something) // to write to other output collections
> > > > }
> > > >
> > > > For some time, we have hoped to unpack the context - it is a
> > > > backwards-compatibility pattern made obsolete by the new DoFn design.
> > So
> > > > instead it would look like this:
> > > >
> > > > @ProcessElement
> > > > public void process(Element element, MainOutput mainOutput, ...) {
> > > >   element.get() // to access the current input element
> > > >   mainOutput.output(something) // to write to the default output
> > > collection
> > > >   other.output(something) // to write to other output collection
> > > > }
> > > >
> > > > I've deliberately left out the undecided syntax for side outputs. But
> > it
> > > > would be nice for the tag to be built in to the parameter so it
> doesn't
> > > > have to be used when calling output().
> > > >
> > > > One way to enhance this to deterministic input would just be this:
> > > >
> > > > @ProcessElement
> > > > @RequiresDeterministicInput
> > > > public void process(Element element, MainOutput mainOutput, ...) {
> > > >   element.get() // to access the current input element
> > > >   mainOutput.output(something) // to write to the default output
> > > collection
> > > >   other.output(something) // to write to other output collection
> > > > }
> > > >
> > > > There are really a lot of places where we could put an annotation or
> > > change
> > > > a type to indicate that the input PCollection should be
> > > > well-defined/deterministically-replayable. I don't have a really
> > strong
> > > > opinion.
> > > >
> > > > Kenn
> > > >
> > > > On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > > <bchambers@google.com.invalid
> > > > >
> > > > wrote:
> > > >
> > > > > Allowing an annotation on DoFn's that produce deterministic output
> > > could
> > > > be
> > > > > added in the future, but doesn't seem like a great option.
> > > > >
> > > > > 1. It is a correctness issue to assume a DoFn is deterministic and
> be
> > > > > wrong, so we would need to assume all transform outputs are
> > > > > non-deterministic unless annotated. Getting this correct is
> difficult
> > > > (for
> > > > > example, GBK is surprisingly non-deterministic except in specific
> > > cases).
> > > > >
> > > > > 2. It is unlikely to be a major performance improvement, given that
> > any
> > > > > non-deterministic transform prior to a sink (which are most likely
> to
> > > > > require deterministic input) will cause additional work to be
> needed.
> > > > >
> > > > > Based on this, it seems like the risk of allowing an annotation is
> > high
> > > > > while the potential for performance improvements is low. The
> current
> > > > > proposal (not allowing an annotation) makes sense for now, until we
> > can
> > > > > demonstrate that the impact on performance is high in cases that
> > could
> > > be
> > > > > avoided with an annotation (in real-world use).
> > > > >
> > > > > -- Ben
> > > > >
> > > > > On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com>
> wrote:
> > > > >
> > > > > +1 for the general idea of runners handling it over hard-coded
> > > > > implementation strategy.
> > > > >
> > > > > For the Write transform I believe you are talking about
> > > ApplyShardingKey
> > > > > <
> > > > > https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > > > > c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> > > > > io/Write.java#L304
> > > > > >
> > > > > which
> > > > > introduces non deterministic behavior when retried?
> > > > >
> > > > >
> > > > > *Let a DoFn declare (mechanism not important right now) that it
> > > > > "requiresdeterministic input"*
> > > > >
> > > > >
> > > > >
> > > > > *Each runner will need a way to induce deterministic input - the
> > > > > obviouschoice being a materialization.*
> > > > >
> > > > > Does this mean that a runner will always materialize (or whatever
> the
> > > > > strategy is) an input PCollection to this DoFn even though the
> > > > PCollection
> > > > > might have been produced by deterministic transforms? Would it make
> > > sense
> > > > > to also let DoFns declare if they produce non-deterministic output?
> > > > >
> > > > > -Vikas
> > > > >
> > > > >
> > > > > On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid>
> > > wrote:
> > > > >
> > > > > > Hey Kenn-
> > > > > >
> > > > > > this seems important, but I don't have all the context on what
> the
> > > > > problem
> > > > > > is.
> > > > > >
> > > > > > Can you explain this sentence "Specifically, there is
> pseudorandom
> > > data
> > > > > > generated and once it has been observed and used to produce a
> side
> > > > > effect,
> > > > > > it cannot be regenerated without erroneous results." ?
> > > > > >
> > > > > > Where is the pseudorandom data coming from? Perhaps a concrete
> > > example
> > > > > > would help?
> > > > > >
> > > > > > S
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > > <klk@google.com.invalid
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Problem:
> > > > > > >
> > > > > > > I will drop all nuance and say that the `Write` transform as it
> > > > exists
> > > > > in
> > > > > > > the SDK is incorrect until we add some specification and APIs.
> We
> > > > can't
> > > > > > > keep shipping an SDK with an unsafe transform in it, and IMO
> this
> > > > > > certainly
> > > > > > > blocks a stable release.
> > > > > > >
> > > > > > > Specifically, there is pseudorandom data generated and once it
> > has
> > > > been
> > > > > > > observed and used to produce a side effect, it cannot be
> > > regenerated
> > > > > > > without erroneous results.
> > > > > > >
> > > > > > > This generalizes: For some side-effecting user-defined
> functions,
> > > it
> > > > is
> > > > > > > vital that even across retries/replays they have a consistent
> > view
> > > of
> > > > > the
> > > > > > > contents of their input PCollection, because their effect on
> the
> > > > > outside
> > > > > > > world cannot be retracted if/when they fail and are retried.
> Once
> > > the
> > > > > > > runner ensures a consistent view of the input, it is then their
> > own
> > > > > > > responsibility to be idempotent.
> > > > > > >
> > > > > > > Ideally we should specify this requirement for the user-defined
> > > > > function
> > > > > > > without imposing any particular implementation strategy on Beam
> > > > > runners.
> > > > > > >
> > > > > > > Proposal:
> > > > > > >
> > > > > > > 1. Let a DoFn declare (mechanism not important right now) that
> it
> > > > > > "requires
> > > > > > > deterministic input".
> > > > > > >
> > > > > > > 2. Each runner will need a way to induce deterministic input -
> > the
> > > > > > obvious
> > > > > > > choice being a materialization.
> > > > > > >
> > > > > > > I want to keep the discussion focused, so I'm leaving out any
> > > > > > possibilities
> > > > > > > of taking this further.
> > > > > > >
> > > > > > > Regarding performance: Today places that require this tend to
> be
> > > > > already
> > > > > > > paying the cost via GroupByKey / Reshuffle operations, since
> that
> > > > was a
> > > > > > > simple way to induce determinism in batch Dataflow* (doesn't
> work
> > > for
> > > > > > most
> > > > > > > other runners nor for streaming Dataflow). This change will
> > > replace a
> > > > > > > hard-coded implementation strategy with a requirement that may
> be
> > > > > > fulfilled
> > > > > > > in the most efficient way available.
> > > > > > >
> > > > > > > Thoughts?
> > > > > > >
> > > > > > > Kenn (w/ lots of consult from colleagues, especially Ben)
> > > > > > >
> > > > > > > * There is some overlap with the reshuffle/redistribute
> > discussion
> > > > > > because
> > > > > > > of this historical situation, but I would like to leave that
> > > broader
> > > > > > > discussion out of this correctness issue.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Thomas Groh <tg...@google.com.INVALID>.
+1 to the annotation-on-ProcessElement approach. ProcessElement is the
minimum implementation requirement of a DoFn, and should be where the
processing logic which depends on characteristics of the inputs lie. It's a
good way of signalling the requirements of the Fn, and letting the runner
decide.

I have a minor concern that this may not work as expected for users that
try to batch remote calls in `FinishBundle` - we should make sure we
document that it is explicitly the input elements that will be replayed,
and bundles and other operational are still arbitrary.



On Wed, Aug 9, 2017 at 10:37 AM, Reuven Lax <re...@google.com.invalid>
wrote:

> I think deterministic here means deterministically replayable. i.e. no
> matter how many times the element is retried, it will always be the same.
>
> I think we should also allow specifying this on processTimer. This would
> mean that any keyed state written in a previous processElement must be
> guaranteed durable before processTimer is called.
>
>
> On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <bc...@apache.org>
> wrote:
>
> > I strongly agree with this proposal. I think moving away from "just
> insert
> > a GroupByKey for one of the 3 different reasons you may want it" towards
> > APIs that allow code to express the requirements they have and the runner
> > to choose the best way to meet this is a major step forwards in terms of
> > portability.
> >
> > I think "deterministic" may be misleading. The actual contents of the
> > collection aren't deterministic if upstream computations aren't. The
> > property we really need is that once an input may have been observed by
> the
> > side-effecting code it will never be observed with a different value.
> >
> > I would propose something RequiresStableInput, to indicate that the input
> > must be stable as observed by the function. I could also see something
> > hinting at the fact we don't recompute the input, such as
> > RequiresMemoizedInput or RequiresNoRecomputation.
> >
> > -- Ben
> >
> > P.S For anyone interested other uses of GroupByKey that we may want to
> > discuss APIs for would be preventing retry across steps (eg., preventing
> > fusion) and redistributing inputs across workers.
> >
> > On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles <kl...@google.com.invalid>
> > wrote:
> >
> > > This came up again, so I wanted to push it along by proposing a
> specific
> > > API for Java that could have a derived API in Python. I am writing this
> > > quickly to get something out there, so I welcome suggestions for
> > revision.
> > >
> > > Today a DoFn has a @ProcessElement annotated method with various
> > automated
> > > parameters, but most fundamentally this:
> > >
> > > @ProcessElement
> > > public void process(ProcessContext ctx) {
> > >   ctx.element() // to access the current input element
> > >   ctx.output(something) // to write to default output collection
> > >   ctx.output(tag, something) // to write to other output collections
> > > }
> > >
> > > For some time, we have hoped to unpack the context - it is a
> > > backwards-compatibility pattern made obsolete by the new DoFn design.
> So
> > > instead it would look like this:
> > >
> > > @ProcessElement
> > > public void process(Element element, MainOutput mainOutput, ...) {
> > >   element.get() // to access the current input element
> > >   mainOutput.output(something) // to write to the default output
> > collection
> > >   other.output(something) // to write to other output collection
> > > }
> > >
> > > I've deliberately left out the undecided syntax for side outputs. But
> it
> > > would be nice for the tag to be built in to the parameter so it doesn't
> > > have to be used when calling output().
> > >
> > > One way to enhance this to deterministic input would just be this:
> > >
> > > @ProcessElement
> > > @RequiresDeterministicInput
> > > public void process(Element element, MainOutput mainOutput, ...) {
> > >   element.get() // to access the current input element
> > >   mainOutput.output(something) // to write to the default output
> > collection
> > >   other.output(something) // to write to other output collection
> > > }
> > >
> > > There are really a lot of places where we could put an annotation or
> > change
> > > a type to indicate that the input PCollection should be
> > > well-defined/deterministically-replayable. I don't have a really
> strong
> > > opinion.
> > >
> > > Kenn
> > >
> > > On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> > <bchambers@google.com.invalid
> > > >
> > > wrote:
> > >
> > > > Allowing an annotation on DoFn's that produce deterministic output
> > could
> > > be
> > > > added in the future, but doesn't seem like a great option.
> > > >
> > > > 1. It is a correctness issue to assume a DoFn is deterministic and be
> > > > wrong, so we would need to assume all transform outputs are
> > > > non-deterministic unless annotated. Getting this correct is difficult
> > > (for
> > > > example, GBK is surprisingly non-deterministic except in specific
> > cases).
> > > >
> > > > 2. It is unlikely to be a major performance improvement, given that
> any
> > > > non-deterministic transform prior to a sink (which are most likely to
> > > > require deterministic input) will cause additional work to be needed.
> > > >
> > > > Based on this, it seems like the risk of allowing an annotation is
> high
> > > > while the potential for performance improvements is low. The current
> > > > proposal (not allowing an annotation) makes sense for now, until we
> can
> > > > demonstrate that the impact on performance is high in cases that
> could
> > be
> > > > avoided with an annotation (in real-world use).
> > > >
> > > > -- Ben
> > > >
> > > > On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com> wrote:
> > > >
> > > > +1 for the general idea of runners handling it over hard-coded
> > > > implementation strategy.
> > > >
> > > > For the Write transform I believe you are talking about
> > ApplyShardingKey
> > > > <
> > > > https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > > > c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> > > > io/Write.java#L304
> > > > >
> > > > which
> > > > introduces non deterministic behavior when retried?
> > > >
> > > >
> > > > *Let a DoFn declare (mechanism not important right now) that it
> > > > "requiresdeterministic input"*
> > > >
> > > >
> > > >
> > > > *Each runner will need a way to induce deterministic input - the
> > > > obviouschoice being a materialization.*
> > > >
> > > > Does this mean that a runner will always materialize (or whatever the
> > > > strategy is) an input PCollection to this DoFn even though the
> > > PCollection
> > > > might have been produced by deterministic transforms? Would it make
> > sense
> > > > to also let DoFns declare if they produce non-deterministic output?
> > > >
> > > > -Vikas
> > > >
> > > >
> > > > On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid>
> > wrote:
> > > >
> > > > > Hey Kenn-
> > > > >
> > > > > this seems important, but I don't have all the context on what the
> > > > problem
> > > > > is.
> > > > >
> > > > > Can you explain this sentence "Specifically, there is pseudorandom
> > data
> > > > > generated and once it has been observed and used to produce a side
> > > > effect,
> > > > > it cannot be regenerated without erroneous results." ?
> > > > >
> > > > > Where is the pseudorandom data coming from? Perhaps a concrete
> > example
> > > > > would help?
> > > > >
> > > > > S
> > > > >
> > > > >
> > > > > On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> > <klk@google.com.invalid
> > > >
> > > > > wrote:
> > > > >
> > > > > > Problem:
> > > > > >
> > > > > > I will drop all nuance and say that the `Write` transform as it
> > > exists
> > > > in
> > > > > > the SDK is incorrect until we add some specification and APIs. We
> > > can't
> > > > > > keep shipping an SDK with an unsafe transform in it, and IMO this
> > > > > certainly
> > > > > > blocks a stable release.
> > > > > >
> > > > > > Specifically, there is pseudorandom data generated and once it
> has
> > > been
> > > > > > observed and used to produce a side effect, it cannot be
> > regenerated
> > > > > > without erroneous results.
> > > > > >
> > > > > > This generalizes: For some side-effecting user-defined functions,
> > it
> > > is
> > > > > > vital that even across retries/replays they have a consistent
> view
> > of
> > > > the
> > > > > > contents of their input PCollection, because their effect on the
> > > > outside
> > > > > > world cannot be retracted if/when they fail and are retried. Once
> > the
> > > > > > runner ensures a consistent view of the input, it is then their
> own
> > > > > > responsibility to be idempotent.
> > > > > >
> > > > > > Ideally we should specify this requirement for the user-defined
> > > > function
> > > > > > without imposing any particular implementation strategy on Beam
> > > > runners.
> > > > > >
> > > > > > Proposal:
> > > > > >
> > > > > > 1. Let a DoFn declare (mechanism not important right now) that it
> > > > > "requires
> > > > > > deterministic input".
> > > > > >
> > > > > > 2. Each runner will need a way to induce deterministic input -
> the
> > > > > obvious
> > > > > > choice being a materialization.
> > > > > >
> > > > > > I want to keep the discussion focused, so I'm leaving out any
> > > > > possibilities
> > > > > > of taking this further.
> > > > > >
> > > > > > Regarding performance: Today places that require this tend to be
> > > > already
> > > > > > paying the cost via GroupByKey / Reshuffle operations, since that
> > > was a
> > > > > > simple way to induce determinism in batch Dataflow* (doesn't work
> > for
> > > > > most
> > > > > > other runners nor for streaming Dataflow). This change will
> > replace a
> > > > > > hard-coded implementation strategy with a requirement that may be
> > > > > fulfilled
> > > > > > in the most efficient way available.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > Kenn (w/ lots of consult from colleagues, especially Ben)
> > > > > >
> > > > > > * There is some overlap with the reshuffle/redistribute
> discussion
> > > > > because
> > > > > > of this historical situation, but I would like to leave that
> > broader
> > > > > > discussion out of this correctness issue.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Reuven Lax <re...@google.com.INVALID>.
I think deterministic here means deterministically replayable. i.e. no
matter how many times the element is retried, it will always be the same.

I think we should also allow specifying this on processTimer. This would
mean that any keyed state written in a previous processElement must be
guaranteed durable before processTimer is called.


On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <bc...@apache.org> wrote:

> I strongly agree with this proposal. I think moving away from "just insert
> a GroupByKey for one of the 3 different reasons you may want it" towards
> APIs that allow code to express the requirements they have and the runner
> to choose the best way to meet this is a major step forwards in terms of
> portability.
>
> I think "deterministic" may be misleading. The actual contents of the
> collection aren't deterministic if upstream computations aren't. The
> property we really need is that once an input may have been observed by the
> side-effecting code it will never be observed with a different value.
>
> I would propose something RequiresStableInput, to indicate that the input
> must be stable as observed by the function. I could also see something
> hinting at the fact we don't recompute the input, such as
> RequiresMemoizedInput or RequiresNoRecomputation.
>
> -- Ben
>
> P.S For anyone interested other uses of GroupByKey that we may want to
> discuss APIs for would be preventing retry across steps (eg., preventing
> fusion) and redistributing inputs across workers.
>
> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles <kl...@google.com.invalid>
> wrote:
>
> > This came up again, so I wanted to push it along by proposing a specific
> > API for Java that could have a derived API in Python. I am writing this
> > quickly to get something out there, so I welcome suggestions for
> revision.
> >
> > Today a DoFn has a @ProcessElement annotated method with various
> automated
> > parameters, but most fundamentally this:
> >
> > @ProcessElement
> > public void process(ProcessContext ctx) {
> >   ctx.element() // to access the current input element
> >   ctx.output(something) // to write to default output collection
> >   ctx.output(tag, something) // to write to other output collections
> > }
> >
> > For some time, we have hoped to unpack the context - it is a
> > backwards-compatibility pattern made obsolete by the new DoFn design. So
> > instead it would look like this:
> >
> > @ProcessElement
> > public void process(Element element, MainOutput mainOutput, ...) {
> >   element.get() // to access the current input element
> >   mainOutput.output(something) // to write to the default output
> collection
> >   other.output(something) // to write to other output collection
> > }
> >
> > I've deliberately left out the undecided syntax for side outputs. But it
> > would be nice for the tag to be built in to the parameter so it doesn't
> > have to be used when calling output().
> >
> > One way to enhance this to deterministic input would just be this:
> >
> > @ProcessElement
> > @RequiresDeterministicInput
> > public void process(Element element, MainOutput mainOutput, ...) {
> >   element.get() // to access the current input element
> >   mainOutput.output(something) // to write to the default output
> collection
> >   other.output(something) // to write to other output collection
> > }
> >
> > There are really a lot of places where we could put an annotation or
> change
> > a type to indicate that the input PCollection should be
> > well-defined/deterministically-replayable. I don't have a really strong
> > opinion.
> >
> > Kenn
> >
> > On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> <bchambers@google.com.invalid
> > >
> > wrote:
> >
> > > Allowing an annotation on DoFn's that produce deterministic output
> could
> > be
> > > added in the future, but doesn't seem like a great option.
> > >
> > > 1. It is a correctness issue to assume a DoFn is deterministic and be
> > > wrong, so we would need to assume all transform outputs are
> > > non-deterministic unless annotated. Getting this correct is difficult
> > (for
> > > example, GBK is surprisingly non-deterministic except in specific
> cases).
> > >
> > > 2. It is unlikely to be a major performance improvement, given that any
> > > non-deterministic transform prior to a sink (which are most likely to
> > > require deterministic input) will cause additional work to be needed.
> > >
> > > Based on this, it seems like the risk of allowing an annotation is high
> > > while the potential for performance improvements is low. The current
> > > proposal (not allowing an annotation) makes sense for now, until we can
> > > demonstrate that the impact on performance is high in cases that could
> be
> > > avoided with an annotation (in real-world use).
> > >
> > > -- Ben
> > >
> > > On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com> wrote:
> > >
> > > +1 for the general idea of runners handling it over hard-coded
> > > implementation strategy.
> > >
> > > For the Write transform I believe you are talking about
> ApplyShardingKey
> > > <
> > > https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > > c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> > > io/Write.java#L304
> > > >
> > > which
> > > introduces non deterministic behavior when retried?
> > >
> > >
> > > *Let a DoFn declare (mechanism not important right now) that it
> > > "requiresdeterministic input"*
> > >
> > >
> > >
> > > *Each runner will need a way to induce deterministic input - the
> > > obviouschoice being a materialization.*
> > >
> > > Does this mean that a runner will always materialize (or whatever the
> > > strategy is) an input PCollection to this DoFn even though the
> > PCollection
> > > might have been produced by deterministic transforms? Would it make
> sense
> > > to also let DoFns declare if they produce non-deterministic output?
> > >
> > > -Vikas
> > >
> > >
> > > On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid>
> wrote:
> > >
> > > > Hey Kenn-
> > > >
> > > > this seems important, but I don't have all the context on what the
> > > problem
> > > > is.
> > > >
> > > > Can you explain this sentence "Specifically, there is pseudorandom
> data
> > > > generated and once it has been observed and used to produce a side
> > > effect,
> > > > it cannot be regenerated without erroneous results." ?
> > > >
> > > > Where is the pseudorandom data coming from? Perhaps a concrete
> example
> > > > would help?
> > > >
> > > > S
> > > >
> > > >
> > > > On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> <klk@google.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > Problem:
> > > > >
> > > > > I will drop all nuance and say that the `Write` transform as it
> > exists
> > > in
> > > > > the SDK is incorrect until we add some specification and APIs. We
> > can't
> > > > > keep shipping an SDK with an unsafe transform in it, and IMO this
> > > > certainly
> > > > > blocks a stable release.
> > > > >
> > > > > Specifically, there is pseudorandom data generated and once it has
> > been
> > > > > observed and used to produce a side effect, it cannot be
> regenerated
> > > > > without erroneous results.
> > > > >
> > > > > This generalizes: For some side-effecting user-defined functions,
> it
> > is
> > > > > vital that even across retries/replays they have a consistent view
> of
> > > the
> > > > > contents of their input PCollection, because their effect on the
> > > outside
> > > > > world cannot be retracted if/when they fail and are retried. Once
> the
> > > > > runner ensures a consistent view of the input, it is then their own
> > > > > responsibility to be idempotent.
> > > > >
> > > > > Ideally we should specify this requirement for the user-defined
> > > function
> > > > > without imposing any particular implementation strategy on Beam
> > > runners.
> > > > >
> > > > > Proposal:
> > > > >
> > > > > 1. Let a DoFn declare (mechanism not important right now) that it
> > > > "requires
> > > > > deterministic input".
> > > > >
> > > > > 2. Each runner will need a way to induce deterministic input - the
> > > > obvious
> > > > > choice being a materialization.
> > > > >
> > > > > I want to keep the discussion focused, so I'm leaving out any
> > > > possibilities
> > > > > of taking this further.
> > > > >
> > > > > Regarding performance: Today places that require this tend to be
> > > already
> > > > > paying the cost via GroupByKey / Reshuffle operations, since that
> > was a
> > > > > simple way to induce determinism in batch Dataflow* (doesn't work
> for
> > > > most
> > > > > other runners nor for streaming Dataflow). This change will
> replace a
> > > > > hard-coded implementation strategy with a requirement that may be
> > > > fulfilled
> > > > > in the most efficient way available.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Kenn (w/ lots of consult from colleagues, especially Ben)
> > > > >
> > > > > * There is some overlap with the reshuffle/redistribute discussion
> > > > because
> > > > > of this historical situation, but I would like to leave that
> broader
> > > > > discussion out of this correctness issue.
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Kenneth Knowles <kl...@google.com.INVALID>.
I like "Stable" too. I can try to make up other scenarios to try out
different vocabulary.

Here are a couple:

 - redundant processing to mitigate stragglers
 - duplication in the course of optimizations*

This expands the scope of the feature to be not just agreement on the
PCollection contents between retries, but between various other execution
strategies, etc.

Kenn

*actually the case where two consumers need to agree on contents is not
addressed directly, but I think can be implemented in userland

On Wed, Aug 9, 2017 at 10:10 AM, Ben Chambers <bc...@apache.org> wrote:

> I strongly agree with this proposal. I think moving away from "just insert
> a GroupByKey for one of the 3 different reasons you may want it" towards
> APIs that allow code to express the requirements they have and the runner
> to choose the best way to meet this is a major step forwards in terms of
> portability.
>
> I think "deterministic" may be misleading. The actual contents of the
> collection aren't deterministic if upstream computations aren't. The
> property we really need is that once an input may have been observed by the
> side-effecting code it will never be observed with a different value.
>
> I would propose something RequiresStableInput, to indicate that the input
> must be stable as observed by the function. I could also see something
> hinting at the fact we don't recompute the input, such as
> RequiresMemoizedInput or RequiresNoRecomputation.
>
> -- Ben
>
> P.S For anyone interested other uses of GroupByKey that we may want to
> discuss APIs for would be preventing retry across steps (eg., preventing
> fusion) and redistributing inputs across workers.
>
> On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles <kl...@google.com.invalid>
> wrote:
>
> > This came up again, so I wanted to push it along by proposing a specific
> > API for Java that could have a derived API in Python. I am writing this
> > quickly to get something out there, so I welcome suggestions for
> revision.
> >
> > Today a DoFn has a @ProcessElement annotated method with various
> automated
> > parameters, but most fundamentally this:
> >
> > @ProcessElement
> > public void process(ProcessContext ctx) {
> >   ctx.element() // to access the current input element
> >   ctx.output(something) // to write to default output collection
> >   ctx.output(tag, something) // to write to other output collections
> > }
> >
> > For some time, we have hoped to unpack the context - it is a
> > backwards-compatibility pattern made obsolete by the new DoFn design. So
> > instead it would look like this:
> >
> > @ProcessElement
> > public void process(Element element, MainOutput mainOutput, ...) {
> >   element.get() // to access the current input element
> >   mainOutput.output(something) // to write to the default output
> collection
> >   other.output(something) // to write to other output collection
> > }
> >
> > I've deliberately left out the undecided syntax for side outputs. But it
> > would be nice for the tag to be built in to the parameter so it doesn't
> > have to be used when calling output().
> >
> > One way to enhance this to deterministic input would just be this:
> >
> > @ProcessElement
> > @RequiresDeterministicInput
> > public void process(Element element, MainOutput mainOutput, ...) {
> >   element.get() // to access the current input element
> >   mainOutput.output(something) // to write to the default output
> collection
> >   other.output(something) // to write to other output collection
> > }
> >
> > There are really a lot of places where we could put an annotation or
> change
> > a type to indicate that the input PCollection should be
> > well-defined/deterministically-replayable. I don't have a really strong
> > opinion.
> >
> > Kenn
> >
> > On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers
> <bchambers@google.com.invalid
> > >
> > wrote:
> >
> > > Allowing an annotation on DoFn's that produce deterministic output
> could
> > be
> > > added in the future, but doesn't seem like a great option.
> > >
> > > 1. It is a correctness issue to assume a DoFn is deterministic and be
> > > wrong, so we would need to assume all transform outputs are
> > > non-deterministic unless annotated. Getting this correct is difficult
> > (for
> > > example, GBK is surprisingly non-deterministic except in specific
> cases).
> > >
> > > 2. It is unlikely to be a major performance improvement, given that any
> > > non-deterministic transform prior to a sink (which are most likely to
> > > require deterministic input) will cause additional work to be needed.
> > >
> > > Based on this, it seems like the risk of allowing an annotation is high
> > > while the potential for performance improvements is low. The current
> > > proposal (not allowing an annotation) makes sense for now, until we can
> > > demonstrate that the impact on performance is high in cases that could
> be
> > > avoided with an annotation (in real-world use).
> > >
> > > -- Ben
> > >
> > > On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com> wrote:
> > >
> > > +1 for the general idea of runners handling it over hard-coded
> > > implementation strategy.
> > >
> > > For the Write transform I believe you are talking about
> ApplyShardingKey
> > > <
> > > https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > > c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> > > io/Write.java#L304
> > > >
> > > which
> > > introduces non deterministic behavior when retried?
> > >
> > >
> > > *Let a DoFn declare (mechanism not important right now) that it
> > > "requiresdeterministic input"*
> > >
> > >
> > >
> > > *Each runner will need a way to induce deterministic input - the
> > > obviouschoice being a materialization.*
> > >
> > > Does this mean that a runner will always materialize (or whatever the
> > > strategy is) an input PCollection to this DoFn even though the
> > PCollection
> > > might have been produced by deterministic transforms? Would it make
> sense
> > > to also let DoFns declare if they produce non-deterministic output?
> > >
> > > -Vikas
> > >
> > >
> > > On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid>
> wrote:
> > >
> > > > Hey Kenn-
> > > >
> > > > this seems important, but I don't have all the context on what the
> > > problem
> > > > is.
> > > >
> > > > Can you explain this sentence "Specifically, there is pseudorandom
> data
> > > > generated and once it has been observed and used to produce a side
> > > effect,
> > > > it cannot be regenerated without erroneous results." ?
> > > >
> > > > Where is the pseudorandom data coming from? Perhaps a concrete
> example
> > > > would help?
> > > >
> > > > S
> > > >
> > > >
> > > > On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles
> <klk@google.com.invalid
> > >
> > > > wrote:
> > > >
> > > > > Problem:
> > > > >
> > > > > I will drop all nuance and say that the `Write` transform as it
> > exists
> > > in
> > > > > the SDK is incorrect until we add some specification and APIs. We
> > can't
> > > > > keep shipping an SDK with an unsafe transform in it, and IMO this
> > > > certainly
> > > > > blocks a stable release.
> > > > >
> > > > > Specifically, there is pseudorandom data generated and once it has
> > been
> > > > > observed and used to produce a side effect, it cannot be
> regenerated
> > > > > without erroneous results.
> > > > >
> > > > > This generalizes: For some side-effecting user-defined functions,
> it
> > is
> > > > > vital that even across retries/replays they have a consistent view
> of
> > > the
> > > > > contents of their input PCollection, because their effect on the
> > > outside
> > > > > world cannot be retracted if/when they fail and are retried. Once
> the
> > > > > runner ensures a consistent view of the input, it is then their own
> > > > > responsibility to be idempotent.
> > > > >
> > > > > Ideally we should specify this requirement for the user-defined
> > > function
> > > > > without imposing any particular implementation strategy on Beam
> > > runners.
> > > > >
> > > > > Proposal:
> > > > >
> > > > > 1. Let a DoFn declare (mechanism not important right now) that it
> > > > "requires
> > > > > deterministic input".
> > > > >
> > > > > 2. Each runner will need a way to induce deterministic input - the
> > > > obvious
> > > > > choice being a materialization.
> > > > >
> > > > > I want to keep the discussion focused, so I'm leaving out any
> > > > possibilities
> > > > > of taking this further.
> > > > >
> > > > > Regarding performance: Today places that require this tend to be
> > > already
> > > > > paying the cost via GroupByKey / Reshuffle operations, since that
> > was a
> > > > > simple way to induce determinism in batch Dataflow* (doesn't work
> for
> > > > most
> > > > > other runners nor for streaming Dataflow). This change will
> replace a
> > > > > hard-coded implementation strategy with a requirement that may be
> > > > fulfilled
> > > > > in the most efficient way available.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > Kenn (w/ lots of consult from colleagues, especially Ben)
> > > > >
> > > > > * There is some overlap with the reshuffle/redistribute discussion
> > > > because
> > > > > of this historical situation, but I would like to leave that
> broader
> > > > > discussion out of this correctness issue.
> > > > >
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Ben Chambers <bc...@apache.org>.
I strongly agree with this proposal. I think moving away from "just insert
a GroupByKey for one of the 3 different reasons you may want it" towards
APIs that allow code to express the requirements they have and the runner
to choose the best way to meet this is a major step forwards in terms of
portability.

I think "deterministic" may be misleading. The actual contents of the
collection aren't deterministic if upstream computations aren't. The
property we really need is that once an input may have been observed by the
side-effecting code it will never be observed with a different value.

I would propose something RequiresStableInput, to indicate that the input
must be stable as observed by the function. I could also see something
hinting at the fact we don't recompute the input, such as
RequiresMemoizedInput or RequiresNoRecomputation.

-- Ben

P.S For anyone interested other uses of GroupByKey that we may want to
discuss APIs for would be preventing retry across steps (eg., preventing
fusion) and redistributing inputs across workers.

On Wed, Aug 9, 2017 at 9:53 AM Kenneth Knowles <kl...@google.com.invalid>
wrote:

> This came up again, so I wanted to push it along by proposing a specific
> API for Java that could have a derived API in Python. I am writing this
> quickly to get something out there, so I welcome suggestions for revision.
>
> Today a DoFn has a @ProcessElement annotated method with various automated
> parameters, but most fundamentally this:
>
> @ProcessElement
> public void process(ProcessContext ctx) {
>   ctx.element() // to access the current input element
>   ctx.output(something) // to write to default output collection
>   ctx.output(tag, something) // to write to other output collections
> }
>
> For some time, we have hoped to unpack the context - it is a
> backwards-compatibility pattern made obsolete by the new DoFn design. So
> instead it would look like this:
>
> @ProcessElement
> public void process(Element element, MainOutput mainOutput, ...) {
>   element.get() // to access the current input element
>   mainOutput.output(something) // to write to the default output collection
>   other.output(something) // to write to other output collection
> }
>
> I've deliberately left out the undecided syntax for side outputs. But it
> would be nice for the tag to be built in to the parameter so it doesn't
> have to be used when calling output().
>
> One way to enhance this to deterministic input would just be this:
>
> @ProcessElement
> @RequiresDeterministicInput
> public void process(Element element, MainOutput mainOutput, ...) {
>   element.get() // to access the current input element
>   mainOutput.output(something) // to write to the default output collection
>   other.output(something) // to write to other output collection
> }
>
> There are really a lot of places where we could put an annotation or change
> a type to indicate that the input PCollection should be
> well-defined/deterministically-replayable. I don't have a really strong
> opinion.
>
> Kenn
>
> On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers <bchambers@google.com.invalid
> >
> wrote:
>
> > Allowing an annotation on DoFn's that produce deterministic output could
> be
> > added in the future, but doesn't seem like a great option.
> >
> > 1. It is a correctness issue to assume a DoFn is deterministic and be
> > wrong, so we would need to assume all transform outputs are
> > non-deterministic unless annotated. Getting this correct is difficult
> (for
> > example, GBK is surprisingly non-deterministic except in specific cases).
> >
> > 2. It is unlikely to be a major performance improvement, given that any
> > non-deterministic transform prior to a sink (which are most likely to
> > require deterministic input) will cause additional work to be needed.
> >
> > Based on this, it seems like the risk of allowing an annotation is high
> > while the potential for performance improvements is low. The current
> > proposal (not allowing an annotation) makes sense for now, until we can
> > demonstrate that the impact on performance is high in cases that could be
> > avoided with an annotation (in real-world use).
> >
> > -- Ben
> >
> > On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com> wrote:
> >
> > +1 for the general idea of runners handling it over hard-coded
> > implementation strategy.
> >
> > For the Write transform I believe you are talking about ApplyShardingKey
> > <
> > https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> > c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> > io/Write.java#L304
> > >
> > which
> > introduces non deterministic behavior when retried?
> >
> >
> > *Let a DoFn declare (mechanism not important right now) that it
> > "requiresdeterministic input"*
> >
> >
> >
> > *Each runner will need a way to induce deterministic input - the
> > obviouschoice being a materialization.*
> >
> > Does this mean that a runner will always materialize (or whatever the
> > strategy is) an input PCollection to this DoFn even though the
> PCollection
> > might have been produced by deterministic transforms? Would it make sense
> > to also let DoFns declare if they produce non-deterministic output?
> >
> > -Vikas
> >
> >
> > On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid> wrote:
> >
> > > Hey Kenn-
> > >
> > > this seems important, but I don't have all the context on what the
> > problem
> > > is.
> > >
> > > Can you explain this sentence "Specifically, there is pseudorandom data
> > > generated and once it has been observed and used to produce a side
> > effect,
> > > it cannot be regenerated without erroneous results." ?
> > >
> > > Where is the pseudorandom data coming from? Perhaps a concrete example
> > > would help?
> > >
> > > S
> > >
> > >
> > > On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles <klk@google.com.invalid
> >
> > > wrote:
> > >
> > > > Problem:
> > > >
> > > > I will drop all nuance and say that the `Write` transform as it
> exists
> > in
> > > > the SDK is incorrect until we add some specification and APIs. We
> can't
> > > > keep shipping an SDK with an unsafe transform in it, and IMO this
> > > certainly
> > > > blocks a stable release.
> > > >
> > > > Specifically, there is pseudorandom data generated and once it has
> been
> > > > observed and used to produce a side effect, it cannot be regenerated
> > > > without erroneous results.
> > > >
> > > > This generalizes: For some side-effecting user-defined functions, it
> is
> > > > vital that even across retries/replays they have a consistent view of
> > the
> > > > contents of their input PCollection, because their effect on the
> > outside
> > > > world cannot be retracted if/when they fail and are retried. Once the
> > > > runner ensures a consistent view of the input, it is then their own
> > > > responsibility to be idempotent.
> > > >
> > > > Ideally we should specify this requirement for the user-defined
> > function
> > > > without imposing any particular implementation strategy on Beam
> > runners.
> > > >
> > > > Proposal:
> > > >
> > > > 1. Let a DoFn declare (mechanism not important right now) that it
> > > "requires
> > > > deterministic input".
> > > >
> > > > 2. Each runner will need a way to induce deterministic input - the
> > > obvious
> > > > choice being a materialization.
> > > >
> > > > I want to keep the discussion focused, so I'm leaving out any
> > > possibilities
> > > > of taking this further.
> > > >
> > > > Regarding performance: Today places that require this tend to be
> > already
> > > > paying the cost via GroupByKey / Reshuffle operations, since that
> was a
> > > > simple way to induce determinism in batch Dataflow* (doesn't work for
> > > most
> > > > other runners nor for streaming Dataflow). This change will replace a
> > > > hard-coded implementation strategy with a requirement that may be
> > > fulfilled
> > > > in the most efficient way available.
> > > >
> > > > Thoughts?
> > > >
> > > > Kenn (w/ lots of consult from colleagues, especially Ben)
> > > >
> > > > * There is some overlap with the reshuffle/redistribute discussion
> > > because
> > > > of this historical situation, but I would like to leave that broader
> > > > discussion out of this correctness issue.
> > > >
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Kenneth Knowles <kl...@google.com.INVALID>.
This came up again, so I wanted to push it along by proposing a specific
API for Java that could have a derived API in Python. I am writing this
quickly to get something out there, so I welcome suggestions for revision.

Today a DoFn has a @ProcessElement annotated method with various automated
parameters, but most fundamentally this:

@ProcessElement
public void process(ProcessContext ctx) {
  ctx.element() // to access the current input element
  ctx.output(something) // to write to default output collection
  ctx.output(tag, something) // to write to other output collections
}

For some time, we have hoped to unpack the context - it is a
backwards-compatibility pattern made obsolete by the new DoFn design. So
instead it would look like this:

@ProcessElement
public void process(Element element, MainOutput mainOutput, ...) {
  element.get() // to access the current input element
  mainOutput.output(something) // to write to the default output collection
  other.output(something) // to write to other output collection
}

I've deliberately left out the undecided syntax for side outputs. But it
would be nice for the tag to be built in to the parameter so it doesn't
have to be used when calling output().

One way to enhance this to deterministic input would just be this:

@ProcessElement
@RequiresDeterministicInput
public void process(Element element, MainOutput mainOutput, ...) {
  element.get() // to access the current input element
  mainOutput.output(something) // to write to the default output collection
  other.output(something) // to write to other output collection
}

There are really a lot of places where we could put an annotation or change
a type to indicate that the input PCollection should be
well-defined/deterministically-replayable. I don't have a really strong
opinion.

Kenn

On Tue, Mar 21, 2017 at 4:53 PM, Ben Chambers <bc...@google.com.invalid>
wrote:

> Allowing an annotation on DoFn's that produce deterministic output could be
> added in the future, but doesn't seem like a great option.
>
> 1. It is a correctness issue to assume a DoFn is deterministic and be
> wrong, so we would need to assume all transform outputs are
> non-deterministic unless annotated. Getting this correct is difficult (for
> example, GBK is surprisingly non-deterministic except in specific cases).
>
> 2. It is unlikely to be a major performance improvement, given that any
> non-deterministic transform prior to a sink (which are most likely to
> require deterministic input) will cause additional work to be needed.
>
> Based on this, it seems like the risk of allowing an annotation is high
> while the potential for performance improvements is low. The current
> proposal (not allowing an annotation) makes sense for now, until we can
> demonstrate that the impact on performance is high in cases that could be
> avoided with an annotation (in real-world use).
>
> -- Ben
>
> On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com> wrote:
>
> +1 for the general idea of runners handling it over hard-coded
> implementation strategy.
>
> For the Write transform I believe you are talking about ApplyShardingKey
> <
> https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> io/Write.java#L304
> >
> which
> introduces non deterministic behavior when retried?
>
>
> *Let a DoFn declare (mechanism not important right now) that it
> "requiresdeterministic input"*
>
>
>
> *Each runner will need a way to induce deterministic input - the
> obviouschoice being a materialization.*
>
> Does this mean that a runner will always materialize (or whatever the
> strategy is) an input PCollection to this DoFn even though the PCollection
> might have been produced by deterministic transforms? Would it make sense
> to also let DoFns declare if they produce non-deterministic output?
>
> -Vikas
>
>
> On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid> wrote:
>
> > Hey Kenn-
> >
> > this seems important, but I don't have all the context on what the
> problem
> > is.
> >
> > Can you explain this sentence "Specifically, there is pseudorandom data
> > generated and once it has been observed and used to produce a side
> effect,
> > it cannot be regenerated without erroneous results." ?
> >
> > Where is the pseudorandom data coming from? Perhaps a concrete example
> > would help?
> >
> > S
> >
> >
> > On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles <kl...@google.com.invalid>
> > wrote:
> >
> > > Problem:
> > >
> > > I will drop all nuance and say that the `Write` transform as it exists
> in
> > > the SDK is incorrect until we add some specification and APIs. We can't
> > > keep shipping an SDK with an unsafe transform in it, and IMO this
> > certainly
> > > blocks a stable release.
> > >
> > > Specifically, there is pseudorandom data generated and once it has been
> > > observed and used to produce a side effect, it cannot be regenerated
> > > without erroneous results.
> > >
> > > This generalizes: For some side-effecting user-defined functions, it is
> > > vital that even across retries/replays they have a consistent view of
> the
> > > contents of their input PCollection, because their effect on the
> outside
> > > world cannot be retracted if/when they fail and are retried. Once the
> > > runner ensures a consistent view of the input, it is then their own
> > > responsibility to be idempotent.
> > >
> > > Ideally we should specify this requirement for the user-defined
> function
> > > without imposing any particular implementation strategy on Beam
> runners.
> > >
> > > Proposal:
> > >
> > > 1. Let a DoFn declare (mechanism not important right now) that it
> > "requires
> > > deterministic input".
> > >
> > > 2. Each runner will need a way to induce deterministic input - the
> > obvious
> > > choice being a materialization.
> > >
> > > I want to keep the discussion focused, so I'm leaving out any
> > possibilities
> > > of taking this further.
> > >
> > > Regarding performance: Today places that require this tend to be
> already
> > > paying the cost via GroupByKey / Reshuffle operations, since that was a
> > > simple way to induce determinism in batch Dataflow* (doesn't work for
> > most
> > > other runners nor for streaming Dataflow). This change will replace a
> > > hard-coded implementation strategy with a requirement that may be
> > fulfilled
> > > in the most efficient way available.
> > >
> > > Thoughts?
> > >
> > > Kenn (w/ lots of consult from colleagues, especially Ben)
> > >
> > > * There is some overlap with the reshuffle/redistribute discussion
> > because
> > > of this historical situation, but I would like to leave that broader
> > > discussion out of this correctness issue.
> > >
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Ben Chambers <bc...@google.com.INVALID>.
Allowing an annotation on DoFn's that produce deterministic output could be
added in the future, but doesn't seem like a great option.

1. It is a correctness issue to assume a DoFn is deterministic and be
wrong, so we would need to assume all transform outputs are
non-deterministic unless annotated. Getting this correct is difficult (for
example, GBK is surprisingly non-deterministic except in specific cases).

2. It is unlikely to be a major performance improvement, given that any
non-deterministic transform prior to a sink (which are most likely to
require deterministic input) will cause additional work to be needed.

Based on this, it seems like the risk of allowing an annotation is high
while the potential for performance improvements is low. The current
proposal (not allowing an annotation) makes sense for now, until we can
demonstrate that the impact on performance is high in cases that could be
avoided with an annotation (in real-world use).

-- Ben

On Tue, Mar 21, 2017 at 2:05 PM vikas rk <vi...@gmail.com> wrote:

+1 for the general idea of runners handling it over hard-coded
implementation strategy.

For the Write transform I believe you are talking about ApplyShardingKey
<
https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Write.java#L304
>
which
introduces non deterministic behavior when retried?


*Let a DoFn declare (mechanism not important right now) that it
"requiresdeterministic input"*



*Each runner will need a way to induce deterministic input - the
obviouschoice being a materialization.*

Does this mean that a runner will always materialize (or whatever the
strategy is) an input PCollection to this DoFn even though the PCollection
might have been produced by deterministic transforms? Would it make sense
to also let DoFns declare if they produce non-deterministic output?

-Vikas


On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid> wrote:

> Hey Kenn-
>
> this seems important, but I don't have all the context on what the problem
> is.
>
> Can you explain this sentence "Specifically, there is pseudorandom data
> generated and once it has been observed and used to produce a side effect,
> it cannot be regenerated without erroneous results." ?
>
> Where is the pseudorandom data coming from? Perhaps a concrete example
> would help?
>
> S
>
>
> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles <kl...@google.com.invalid>
> wrote:
>
> > Problem:
> >
> > I will drop all nuance and say that the `Write` transform as it exists
in
> > the SDK is incorrect until we add some specification and APIs. We can't
> > keep shipping an SDK with an unsafe transform in it, and IMO this
> certainly
> > blocks a stable release.
> >
> > Specifically, there is pseudorandom data generated and once it has been
> > observed and used to produce a side effect, it cannot be regenerated
> > without erroneous results.
> >
> > This generalizes: For some side-effecting user-defined functions, it is
> > vital that even across retries/replays they have a consistent view of
the
> > contents of their input PCollection, because their effect on the outside
> > world cannot be retracted if/when they fail and are retried. Once the
> > runner ensures a consistent view of the input, it is then their own
> > responsibility to be idempotent.
> >
> > Ideally we should specify this requirement for the user-defined function
> > without imposing any particular implementation strategy on Beam runners.
> >
> > Proposal:
> >
> > 1. Let a DoFn declare (mechanism not important right now) that it
> "requires
> > deterministic input".
> >
> > 2. Each runner will need a way to induce deterministic input - the
> obvious
> > choice being a materialization.
> >
> > I want to keep the discussion focused, so I'm leaving out any
> possibilities
> > of taking this further.
> >
> > Regarding performance: Today places that require this tend to be already
> > paying the cost via GroupByKey / Reshuffle operations, since that was a
> > simple way to induce determinism in batch Dataflow* (doesn't work for
> most
> > other runners nor for streaming Dataflow). This change will replace a
> > hard-coded implementation strategy with a requirement that may be
> fulfilled
> > in the most efficient way available.
> >
> > Thoughts?
> >
> > Kenn (w/ lots of consult from colleagues, especially Ben)
> >
> > * There is some overlap with the reshuffle/redistribute discussion
> because
> > of this historical situation, but I would like to leave that broader
> > discussion out of this correctness issue.
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Kenneth Knowles <kl...@google.com.INVALID>.
Good points & questions. I'll try to be more clear.



> On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid> wrote:
>
> > Hey Kenn-
> >
> > this seems important, but I don't have all the context on what the
> problem
> > is.
> >
> > Can you explain this sentence "Specifically, there is pseudorandom data
> > generated and once it has been observed and used to produce a side
> effect,
> > it cannot be regenerated without erroneous results." ?
>

On Tue, Mar 21, 2017 at 2:04 PM, vikas rk <vi...@gmail.com> wrote:

>
> For the Write transform I believe you are talking about ApplyShardingKey
> <https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4
> c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/
> io/Write.java#L304>
> which
> introduces non deterministic behavior when retried?


Yes, exactly this. If the sharding key changes, then the rest of the
transform doesn't function correctly.


> Where is the pseudorandom data coming from? Perhaps a concrete example
> > would help?
>

I think the Write transform is a particularly complex example because of
the layers of abstraction. A simplified strawman might be:

Transform 1: Build RPC write descriptors identified by pseudo-random UUIDs.
Transform 2: Issue RPCs with those identifiers, so the endpoint will ignore
repeats of the same UUID (I tend to call this an "idempotency key" so I
might slip into that terminology sometimes)

In this case, transform 2 requires deterministic input because if the write
fails and is retried a new UUID means the endpoint won't know it is a
retry, resulting in duplicate data.

Is this clearer?

Kenn

Re: [PROPOSAL] "Requires deterministic input"

Posted by vikas rk <vi...@gmail.com>.
+1 for the general idea of runners handling it over hard-coded
implementation strategy.

For the Write transform I believe you are talking about ApplyShardingKey
<https://github.com/apache/beam/blob/d66029cafde152c0a46ebd276ddfa4c3e7fd3433/sdks/java/core/src/main/java/org/apache/beam/sdk/io/Write.java#L304>
which
introduces non deterministic behavior when retried?


*Let a DoFn declare (mechanism not important right now) that it
"requiresdeterministic input"*



*Each runner will need a way to induce deterministic input - the
obviouschoice being a materialization.*

Does this mean that a runner will always materialize (or whatever the
strategy is) an input PCollection to this DoFn even though the PCollection
might have been produced by deterministic transforms? Would it make sense
to also let DoFns declare if they produce non-deterministic output?

-Vikas


On 21 March 2017 at 13:52, Stephen Sisk <si...@google.com.invalid> wrote:

> Hey Kenn-
>
> this seems important, but I don't have all the context on what the problem
> is.
>
> Can you explain this sentence "Specifically, there is pseudorandom data
> generated and once it has been observed and used to produce a side effect,
> it cannot be regenerated without erroneous results." ?
>
> Where is the pseudorandom data coming from? Perhaps a concrete example
> would help?
>
> S
>
>
> On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles <kl...@google.com.invalid>
> wrote:
>
> > Problem:
> >
> > I will drop all nuance and say that the `Write` transform as it exists in
> > the SDK is incorrect until we add some specification and APIs. We can't
> > keep shipping an SDK with an unsafe transform in it, and IMO this
> certainly
> > blocks a stable release.
> >
> > Specifically, there is pseudorandom data generated and once it has been
> > observed and used to produce a side effect, it cannot be regenerated
> > without erroneous results.
> >
> > This generalizes: For some side-effecting user-defined functions, it is
> > vital that even across retries/replays they have a consistent view of the
> > contents of their input PCollection, because their effect on the outside
> > world cannot be retracted if/when they fail and are retried. Once the
> > runner ensures a consistent view of the input, it is then their own
> > responsibility to be idempotent.
> >
> > Ideally we should specify this requirement for the user-defined function
> > without imposing any particular implementation strategy on Beam runners.
> >
> > Proposal:
> >
> > 1. Let a DoFn declare (mechanism not important right now) that it
> "requires
> > deterministic input".
> >
> > 2. Each runner will need a way to induce deterministic input - the
> obvious
> > choice being a materialization.
> >
> > I want to keep the discussion focused, so I'm leaving out any
> possibilities
> > of taking this further.
> >
> > Regarding performance: Today places that require this tend to be already
> > paying the cost via GroupByKey / Reshuffle operations, since that was a
> > simple way to induce determinism in batch Dataflow* (doesn't work for
> most
> > other runners nor for streaming Dataflow). This change will replace a
> > hard-coded implementation strategy with a requirement that may be
> fulfilled
> > in the most efficient way available.
> >
> > Thoughts?
> >
> > Kenn (w/ lots of consult from colleagues, especially Ben)
> >
> > * There is some overlap with the reshuffle/redistribute discussion
> because
> > of this historical situation, but I would like to leave that broader
> > discussion out of this correctness issue.
> >
>

Re: [PROPOSAL] "Requires deterministic input"

Posted by Stephen Sisk <si...@google.com.INVALID>.
Hey Kenn-

this seems important, but I don't have all the context on what the problem
is.

Can you explain this sentence "Specifically, there is pseudorandom data
generated and once it has been observed and used to produce a side effect,
it cannot be regenerated without erroneous results." ?

Where is the pseudorandom data coming from? Perhaps a concrete example
would help?

S


On Tue, Mar 21, 2017 at 1:22 PM Kenneth Knowles <kl...@google.com.invalid>
wrote:

> Problem:
>
> I will drop all nuance and say that the `Write` transform as it exists in
> the SDK is incorrect until we add some specification and APIs. We can't
> keep shipping an SDK with an unsafe transform in it, and IMO this certainly
> blocks a stable release.
>
> Specifically, there is pseudorandom data generated and once it has been
> observed and used to produce a side effect, it cannot be regenerated
> without erroneous results.
>
> This generalizes: For some side-effecting user-defined functions, it is
> vital that even across retries/replays they have a consistent view of the
> contents of their input PCollection, because their effect on the outside
> world cannot be retracted if/when they fail and are retried. Once the
> runner ensures a consistent view of the input, it is then their own
> responsibility to be idempotent.
>
> Ideally we should specify this requirement for the user-defined function
> without imposing any particular implementation strategy on Beam runners.
>
> Proposal:
>
> 1. Let a DoFn declare (mechanism not important right now) that it "requires
> deterministic input".
>
> 2. Each runner will need a way to induce deterministic input - the obvious
> choice being a materialization.
>
> I want to keep the discussion focused, so I'm leaving out any possibilities
> of taking this further.
>
> Regarding performance: Today places that require this tend to be already
> paying the cost via GroupByKey / Reshuffle operations, since that was a
> simple way to induce determinism in batch Dataflow* (doesn't work for most
> other runners nor for streaming Dataflow). This change will replace a
> hard-coded implementation strategy with a requirement that may be fulfilled
> in the most efficient way available.
>
> Thoughts?
>
> Kenn (w/ lots of consult from colleagues, especially Ben)
>
> * There is some overlap with the reshuffle/redistribute discussion because
> of this historical situation, but I would like to leave that broader
> discussion out of this correctness issue.
>