Re: Flink Iterations Ordering

[Aljoscha Krettek <al...@apache.org>]

Hi,
as I understand it the order of elements will not be preserved across
iteration supersets. But maybe some-one else knows more.

Cheers,
Aljoscha

On Thu, 28 Apr 2016 at 00:23 David Kim <da...@braintreepayments.com>
wrote:

> Hello all,
>
> I read the documentation at [1] on iterations and had a question on
> whether an assumption is safe to make.
>
> As partial solutions are continuously looping through the step function,
> when new elements are added as iteration inputs will the insertion order of
> all of the elements be preserved?
>
> Example:
>
> Current partial solutions going through step function are: A, B, C.
>
> At a later time we add (in order) elements D, E.
>
> Will the iteration result always be A,B,C,D,E?
>
> References:
>
> [1] *https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html
> <https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html>*
>

Re: Flink Iterations Ordering

[Stephan Ewen <se...@apache.org>]

Hi!

The order in which the elements arrive in an iteration HEAD is the order in
which the last operator in the loop (the TAIL) produces them. If that is a
deterministic ordering (because of a sorted reduce, for example), then you
should be able to rely on the order.

Otherwise, the order of elements can always change every time elements are
re-distributed across tasks (for example during a key-based shuffle, a
broadcast, or a change of parallelism).

Greetings,
Stephan


On Mon, May 2, 2016 at 2:50 PM, Aljoscha Krettek <al...@apache.org>
wrote:

> Hi,
> as I understand it the order of elements will not be preserved across
> iteration supersets. But maybe some-one else knows more.
>
> Cheers,
> Aljoscha
>
> On Thu, 28 Apr 2016 at 00:23 David Kim <da...@braintreepayments.com>
> wrote:
>
>> Hello all,
>>
>> I read the documentation at [1] on iterations and had a question on
>> whether an assumption is safe to make.
>>
>> As partial solutions are continuously looping through the step function,
>> when new elements are added as iteration inputs will the insertion order of
>> all of the elements be preserved?
>>
>> Example:
>>
>> Current partial solutions going through step function are: A, B, C.
>>
>> At a later time we add (in order) elements D, E.
>>
>> Will the iteration result always be A,B,C,D,E?
>>
>> References:
>>
>> [1] *https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html
>> <https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html>*
>>
>