Regarding the field ordering after Select.Flattened transform

[Tao Li <ta...@zillow.com>]

Hi community,

I have been experimenting with Select.Flattened transform and noticed that the field order in the flattened schema is not consistent with the order of the top level fields from the original schema. For example, in the original schema, we have field “foo” as the first field and it has a nested field “bar”. After applying the flattening transform, the new field “foo.bar” becomes the last field in the flattened schema. Seems like the order of the new fields are not that deterministic in the flattened schema. Is this an expected behavior? Don’t we guarantee any ordering of the flattened fields (e.g. being consistent with the original order)? Thanks!

Re: Regarding the field ordering after Select.Flattened transform

[Brian Hulette <bh...@google.com>]

Gotcha. It would certainly look cleaner, I'd be open to such a change as
long as it doesn't complicate the code or impact performance, but I don't
think we should make any guarantees.

+Reuven Lax <re...@google.com> WDYT?

Brian

On Wed, Jan 20, 2021 at 10:14 AM Tao Li <ta...@zillow.com> wrote:

> Hi Brian,
>
>
>
> Thanks for your quick response. I totally agree that the we should not
> rely on any assumption on the field order and we can always specify the
> order of the flattened fields as we want. There is no blocker issue for me
> with the current behavior, but I am just wondering if may be convenient in
> some use cases if we can just keep the order (roughly) consistent with the
> order of the parent fields from the original schema.
>
>
>
> *From: *Brian Hulette <bh...@google.com>
> *Reply-To: *"user@beam.apache.org" <us...@beam.apache.org>
> *Date: *Wednesday, January 20, 2021 at 9:42 AM
> *To: *user <us...@beam.apache.org>
> *Subject: *Re: Regarding the field ordering after Select.Flattened
> transform
>
>
>
> This does seem like an odd choice, I suspect this was just a matter of
> convenience of implementation since the javadoc makes no claims about field
> order.
>
> In general schema transforms don't take care to maintain a particular
> field order and I'd recommend against relying on it. Instead fields should
> be addressed by name, either with Row.getValue(Sring), or by mapping to a
> user type. Is there a reason you want to rely on a particular field order?
> Maybe when writing to certain IOs field order could be important.
>
>
>
> On Tue, Jan 19, 2021 at 1:36 PM Tao Li <ta...@zillow.com> wrote:
>
> Hi community,
>
>
>
> I have been experimenting with Select.Flattened transform and noticed that
> the field order in the flattened schema is not consistent with the order of
> the top level fields from the original schema. For example, in the original
> schema, we have field “foo” as the first field and it has a nested field
> “bar”. After applying the flattening transform, the new field “foo.bar”
> becomes the last field in the flattened schema. Seems like the order of the
> new fields are not that deterministic in the flattened schema. Is this an
> expected behavior? Don’t we guarantee any ordering of the flattened fields
> (e.g. being consistent with the original order)? Thanks!
>
>

Re: Regarding the field ordering after Select.Flattened transform

[Tao Li <ta...@zillow.com>]

Hi Brian,

Thanks for your quick response. I totally agree that the we should not rely on any assumption on the field order and we can always specify the order of the flattened fields as we want. There is no blocker issue for me with the current behavior, but I am just wondering if may be convenient in some use cases if we can just keep the order (roughly) consistent with the order of the parent fields from the original schema.

From: Brian Hulette <bh...@google.com>
Reply-To: "user@beam.apache.org" <us...@beam.apache.org>
Date: Wednesday, January 20, 2021 at 9:42 AM
To: user <us...@beam.apache.org>
Subject: Re: Regarding the field ordering after Select.Flattened transform

This does seem like an odd choice, I suspect this was just a matter of convenience of implementation since the javadoc makes no claims about field order.

In general schema transforms don't take care to maintain a particular field order and I'd recommend against relying on it. Instead fields should be addressed by name, either with Row.getValue(Sring), or by mapping to a user type. Is there a reason you want to rely on a particular field order? Maybe when writing to certain IOs field order could be important.

On Tue, Jan 19, 2021 at 1:36 PM Tao Li <ta...@zillow.com>> wrote:
Hi community,

I have been experimenting with Select.Flattened transform and noticed that the field order in the flattened schema is not consistent with the order of the top level fields from the original schema. For example, in the original schema, we have field “foo” as the first field and it has a nested field “bar”. After applying the flattening transform, the new field “foo.bar” becomes the last field in the flattened schema. Seems like the order of the new fields are not that deterministic in the flattened schema. Is this an expected behavior? Don’t we guarantee any ordering of the flattened fields (e.g. being consistent with the original order)? Thanks!

Re: Regarding the field ordering after Select.Flattened transform

[Brian Hulette <bh...@google.com>]

This does seem like an odd choice, I suspect this was just a matter of
convenience of implementation since the javadoc makes no claims about field
order.

In general schema transforms don't take care to maintain a particular field
order and I'd recommend against relying on it. Instead fields should be
addressed by name, either with Row.getValue(Sring), or by mapping to a user
type. Is there a reason you want to rely on a particular field order? Maybe
when writing to certain IOs field order could be important.

On Tue, Jan 19, 2021 at 1:36 PM Tao Li <ta...@zillow.com> wrote:

> Hi community,
>
>
>
> I have been experimenting with Select.Flattened transform and noticed that
> the field order in the flattened schema is not consistent with the order of
> the top level fields from the original schema. For example, in the original
> schema, we have field “foo” as the first field and it has a nested field
> “bar”. After applying the flattening transform, the new field “foo.bar”
> becomes the last field in the flattened schema. Seems like the order of the
> new fields are not that deterministic in the flattened schema. Is this an
> expected behavior? Don’t we guarantee any ordering of the flattened fields
> (e.g. being consistent with the original order)? Thanks!
>