You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by Raymond Xu <xu...@gmail.com> on 2021/01/16 19:37:56 UTC

[DISCUSS] Support multiple ordering fields

Just want to discuss a small improvement for setting ordering fields.
For
- property `hoodie.payload.ordering.field` and
- deltastreamer --source-ordering-field
I think it can be useful to support multiple fields (configured via a
comma-separated list) to determine the order in some cases. This would need
another config to set the Comparable implementation, say
hoodie.payload.ordering.comparable.class to allow custom logic for doing
comparison.

Any suggestions? Thanks.

Re: [DISCUSS] Support multiple ordering fields

Posted by Danny Chan <da...@apache.org>.
It is reasonable to support multiple ordering fields. The ordering fields
semantics is very similar with the SQL PRIMARY KEY, the PRIMARY KEY did
support multiple fields.

A little different with PRIMARY KEY is that, current ordering field also
supports nested fields, E.G. "a.b.c", which the PRIMARY KEY definition can
not cover.

Best,
Danny Chan

Raymond Xu <xu...@gmail.com> 于2021年2月5日周五 下午6:48写道:

> No worries Vinoth. Thank you for the feedback. I have created
> https://issues.apache.org/jira/browse/HUDI-1588
> Anyone interested please feel free to pick it up. I would be happy to do it
> but not having cycles at the moment.
>
> On Sun, Jan 31, 2021 at 6:32 PM Vinoth Chandar <vi...@apache.org> wrote:
>
> > (apologies for the delay here)
> >
> > Yes. 0.7.0 now has created a `HoodiePayloadConfig` and its also passed
> into
> > the HoodieRecordPayload methods used in ordering/precombining.
> > So, we can clearly document each payload class with its options and it
> > should be possible for users to configure this just like any
> > normal configs. This is probably much simpler.
> >
> > Any takers for this work? :)
> >
> > On Wed, Jan 20, 2021 at 5:49 PM Raymond Xu <xu...@gmail.com>
> > wrote:
> >
> > > Thank you for the feedbacks. I'm also in favor of simplifying the delta
> > > streamer's usage.
> > >
> > > @vinoth, building on what you said, would it be simpler to merge the
> > > ordering field config with the payload class config?
> > > i.e., to remove both --source-ordering-field and payload ordering field
> > > config, then define the ordering field within the payload class.
> > > People who use fields other than `ts` will need to supply a payload
> class
> > > and override it.
> > > People who already supply a payload class will just override it in the
> > > existing custom class.
> > > WDYT?
> > >
> > >
> > > On Tue, Jan 19, 2021 at 10:40 AM Pratyaksh Sharma <
> pratyaksh13@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > We can use transformer to have a combination of multiple ordering
> > fields.
> > > > However custom Comparable implementation is not possible in that
> case.
> > So
> > > > overall a +1 from my side as well.
> > > >
> > > > On Tue, Jan 19, 2021 at 1:58 PM 刘金辉 <96...@qq.com> wrote:
> > > >
> > > > > +1,&nbsp;Currently we have encountered such scenarios and look
> > forward
> > > to
> > > > > supporting
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > ------------------&nbsp;原始邮件&nbsp;------------------
> > > > > 发件人:
> > > > >                                                   "dev"
> > > > >                                                                 <
> > > > > danny0405@apache.org&gt;;
> > > > > 发送时间:&nbsp;2021年1月19日(星期二) 下午4:25
> > > > > 收件人:&nbsp;"dev"<dev@hudi.apache.org&gt;;
> > > > >
> > > > > 主题:&nbsp;Re: [DISCUSS] Support multiple ordering fields
> > > > >
> > > > >
> > > > >
> > > > > &gt; Wondering if we should just take a bunch of payload configs
> and
> > > > > deprecate
> > > > > these flags
> > > > >
> > > > > I have the same feeling, there are already so many config options
> in
> > > > > Hoodie, the maintain work for developers or users is hard.
> > > > >
> > > > > Vinoth Chandar <vinoth@apache.org&gt; 于2021年1月18日周一 下午11:40写道:
> > > > >
> > > > > &gt; +1 as well.
> > > > > &gt;
> > > > > &gt; Slightly orthogonal point.
> > > > > &gt;
> > > > > &gt; Wondering if we should just take a bunch of payload configs
> and
> > > > > deprecate
> > > > > &gt; these flags?
> > > > > &gt; What I mean is : --source-ordering-field ultimately is used by
> > the
> > > > > &gt; HoodieAvroPayload class (or its family).
> > > > > &gt; Our utilities expose these as flags for convenience, but its
> > just
> > > > > more work
> > > > > &gt; to maintain.
> > > > > &gt;
> > > > > &gt; On Mon, Jan 18, 2021 at 7:09 AM vino yang <
> > yanghua1127@gmail.com
> > > > &gt;
> > > > > wrote:
> > > > > &gt;
> > > > > &gt; &gt; +1,
> > > > > &gt; &gt;
> > > > > &gt; &gt; We have found that such flexibility is needed in some
> > > > scenarios.
> > > > > &gt; &gt;
> > > > > &gt; &gt; Best,
> > > > > &gt; &gt; Vino
> > > > > &gt; &gt;
> > > > > &gt; &gt; Raymond Xu <xu.shiyan.raymond@gmail.com&gt;
> 于2021年1月17日周日
> > > > > 上午3:38写道:
> > > > > &gt; &gt;
> > > > > &gt; &gt; &gt; Just want to discuss a small improvement for setting
> > > > > ordering fields.
> > > > > &gt; &gt; &gt; For
> > > > > &gt; &gt; &gt; - property `hoodie.payload.ordering.field` and
> > > > > &gt; &gt; &gt; - deltastreamer --source-ordering-field
> > > > > &gt; &gt; &gt; I think it can be useful to support multiple fields
> > > > > (configured via a
> > > > > &gt; &gt; &gt; comma-separated list) to determine the order in some
> > > > cases.
> > > > > This would
> > > > > &gt; &gt; need
> > > > > &gt; &gt; &gt; another config to set the Comparable implementation,
> > say
> > > > > &gt; &gt; &gt; hoodie.payload.ordering.comparable.class to allow
> > custom
> > > > > logic for
> > > > > &gt; doing
> > > > > &gt; &gt; &gt; comparison.
> > > > > &gt; &gt; &gt;
> > > > > &gt; &gt; &gt; Any suggestions? Thanks.
> > > > > &gt; &gt; &gt;
> > > > > &gt; &gt;
> > > > > &gt;
> > > >
> > >
> >
>

Re: [DISCUSS] Support multiple ordering fields

Posted by Raymond Xu <xu...@gmail.com>.
No worries Vinoth. Thank you for the feedback. I have created
https://issues.apache.org/jira/browse/HUDI-1588
Anyone interested please feel free to pick it up. I would be happy to do it
but not having cycles at the moment.

On Sun, Jan 31, 2021 at 6:32 PM Vinoth Chandar <vi...@apache.org> wrote:

> (apologies for the delay here)
>
> Yes. 0.7.0 now has created a `HoodiePayloadConfig` and its also passed into
> the HoodieRecordPayload methods used in ordering/precombining.
> So, we can clearly document each payload class with its options and it
> should be possible for users to configure this just like any
> normal configs. This is probably much simpler.
>
> Any takers for this work? :)
>
> On Wed, Jan 20, 2021 at 5:49 PM Raymond Xu <xu...@gmail.com>
> wrote:
>
> > Thank you for the feedbacks. I'm also in favor of simplifying the delta
> > streamer's usage.
> >
> > @vinoth, building on what you said, would it be simpler to merge the
> > ordering field config with the payload class config?
> > i.e., to remove both --source-ordering-field and payload ordering field
> > config, then define the ordering field within the payload class.
> > People who use fields other than `ts` will need to supply a payload class
> > and override it.
> > People who already supply a payload class will just override it in the
> > existing custom class.
> > WDYT?
> >
> >
> > On Tue, Jan 19, 2021 at 10:40 AM Pratyaksh Sharma <pratyaksh13@gmail.com
> >
> > wrote:
> >
> > > Hi
> > >
> > > We can use transformer to have a combination of multiple ordering
> fields.
> > > However custom Comparable implementation is not possible in that case.
> So
> > > overall a +1 from my side as well.
> > >
> > > On Tue, Jan 19, 2021 at 1:58 PM 刘金辉 <96...@qq.com> wrote:
> > >
> > > > +1,&nbsp;Currently we have encountered such scenarios and look
> forward
> > to
> > > > supporting
> > > >
> > > >
> > > >
> > > >
> > > > ------------------&nbsp;原始邮件&nbsp;------------------
> > > > 发件人:
> > > >                                                   "dev"
> > > >                                                                 <
> > > > danny0405@apache.org&gt;;
> > > > 发送时间:&nbsp;2021年1月19日(星期二) 下午4:25
> > > > 收件人:&nbsp;"dev"<dev@hudi.apache.org&gt;;
> > > >
> > > > 主题:&nbsp;Re: [DISCUSS] Support multiple ordering fields
> > > >
> > > >
> > > >
> > > > &gt; Wondering if we should just take a bunch of payload configs and
> > > > deprecate
> > > > these flags
> > > >
> > > > I have the same feeling, there are already so many config options in
> > > > Hoodie, the maintain work for developers or users is hard.
> > > >
> > > > Vinoth Chandar <vinoth@apache.org&gt; 于2021年1月18日周一 下午11:40写道:
> > > >
> > > > &gt; +1 as well.
> > > > &gt;
> > > > &gt; Slightly orthogonal point.
> > > > &gt;
> > > > &gt; Wondering if we should just take a bunch of payload configs and
> > > > deprecate
> > > > &gt; these flags?
> > > > &gt; What I mean is : --source-ordering-field ultimately is used by
> the
> > > > &gt; HoodieAvroPayload class (or its family).
> > > > &gt; Our utilities expose these as flags for convenience, but its
> just
> > > > more work
> > > > &gt; to maintain.
> > > > &gt;
> > > > &gt; On Mon, Jan 18, 2021 at 7:09 AM vino yang <
> yanghua1127@gmail.com
> > > &gt;
> > > > wrote:
> > > > &gt;
> > > > &gt; &gt; +1,
> > > > &gt; &gt;
> > > > &gt; &gt; We have found that such flexibility is needed in some
> > > scenarios.
> > > > &gt; &gt;
> > > > &gt; &gt; Best,
> > > > &gt; &gt; Vino
> > > > &gt; &gt;
> > > > &gt; &gt; Raymond Xu <xu.shiyan.raymond@gmail.com&gt; 于2021年1月17日周日
> > > > 上午3:38写道:
> > > > &gt; &gt;
> > > > &gt; &gt; &gt; Just want to discuss a small improvement for setting
> > > > ordering fields.
> > > > &gt; &gt; &gt; For
> > > > &gt; &gt; &gt; - property `hoodie.payload.ordering.field` and
> > > > &gt; &gt; &gt; - deltastreamer --source-ordering-field
> > > > &gt; &gt; &gt; I think it can be useful to support multiple fields
> > > > (configured via a
> > > > &gt; &gt; &gt; comma-separated list) to determine the order in some
> > > cases.
> > > > This would
> > > > &gt; &gt; need
> > > > &gt; &gt; &gt; another config to set the Comparable implementation,
> say
> > > > &gt; &gt; &gt; hoodie.payload.ordering.comparable.class to allow
> custom
> > > > logic for
> > > > &gt; doing
> > > > &gt; &gt; &gt; comparison.
> > > > &gt; &gt; &gt;
> > > > &gt; &gt; &gt; Any suggestions? Thanks.
> > > > &gt; &gt; &gt;
> > > > &gt; &gt;
> > > > &gt;
> > >
> >
>

Re: [DISCUSS] Support multiple ordering fields

Posted by Vinoth Chandar <vi...@apache.org>.
(apologies for the delay here)

Yes. 0.7.0 now has created a `HoodiePayloadConfig` and its also passed into
the HoodieRecordPayload methods used in ordering/precombining.
So, we can clearly document each payload class with its options and it
should be possible for users to configure this just like any
normal configs. This is probably much simpler.

Any takers for this work? :)

On Wed, Jan 20, 2021 at 5:49 PM Raymond Xu <xu...@gmail.com>
wrote:

> Thank you for the feedbacks. I'm also in favor of simplifying the delta
> streamer's usage.
>
> @vinoth, building on what you said, would it be simpler to merge the
> ordering field config with the payload class config?
> i.e., to remove both --source-ordering-field and payload ordering field
> config, then define the ordering field within the payload class.
> People who use fields other than `ts` will need to supply a payload class
> and override it.
> People who already supply a payload class will just override it in the
> existing custom class.
> WDYT?
>
>
> On Tue, Jan 19, 2021 at 10:40 AM Pratyaksh Sharma <pr...@gmail.com>
> wrote:
>
> > Hi
> >
> > We can use transformer to have a combination of multiple ordering fields.
> > However custom Comparable implementation is not possible in that case. So
> > overall a +1 from my side as well.
> >
> > On Tue, Jan 19, 2021 at 1:58 PM 刘金辉 <96...@qq.com> wrote:
> >
> > > +1,&nbsp;Currently we have encountered such scenarios and look forward
> to
> > > supporting
> > >
> > >
> > >
> > >
> > > ------------------&nbsp;原始邮件&nbsp;------------------
> > > 发件人:
> > >                                                   "dev"
> > >                                                                 <
> > > danny0405@apache.org&gt;;
> > > 发送时间:&nbsp;2021年1月19日(星期二) 下午4:25
> > > 收件人:&nbsp;"dev"<dev@hudi.apache.org&gt;;
> > >
> > > 主题:&nbsp;Re: [DISCUSS] Support multiple ordering fields
> > >
> > >
> > >
> > > &gt; Wondering if we should just take a bunch of payload configs and
> > > deprecate
> > > these flags
> > >
> > > I have the same feeling, there are already so many config options in
> > > Hoodie, the maintain work for developers or users is hard.
> > >
> > > Vinoth Chandar <vinoth@apache.org&gt; 于2021年1月18日周一 下午11:40写道:
> > >
> > > &gt; +1 as well.
> > > &gt;
> > > &gt; Slightly orthogonal point.
> > > &gt;
> > > &gt; Wondering if we should just take a bunch of payload configs and
> > > deprecate
> > > &gt; these flags?
> > > &gt; What I mean is : --source-ordering-field ultimately is used by the
> > > &gt; HoodieAvroPayload class (or its family).
> > > &gt; Our utilities expose these as flags for convenience, but its just
> > > more work
> > > &gt; to maintain.
> > > &gt;
> > > &gt; On Mon, Jan 18, 2021 at 7:09 AM vino yang <yanghua1127@gmail.com
> > &gt;
> > > wrote:
> > > &gt;
> > > &gt; &gt; +1,
> > > &gt; &gt;
> > > &gt; &gt; We have found that such flexibility is needed in some
> > scenarios.
> > > &gt; &gt;
> > > &gt; &gt; Best,
> > > &gt; &gt; Vino
> > > &gt; &gt;
> > > &gt; &gt; Raymond Xu <xu.shiyan.raymond@gmail.com&gt; 于2021年1月17日周日
> > > 上午3:38写道:
> > > &gt; &gt;
> > > &gt; &gt; &gt; Just want to discuss a small improvement for setting
> > > ordering fields.
> > > &gt; &gt; &gt; For
> > > &gt; &gt; &gt; - property `hoodie.payload.ordering.field` and
> > > &gt; &gt; &gt; - deltastreamer --source-ordering-field
> > > &gt; &gt; &gt; I think it can be useful to support multiple fields
> > > (configured via a
> > > &gt; &gt; &gt; comma-separated list) to determine the order in some
> > cases.
> > > This would
> > > &gt; &gt; need
> > > &gt; &gt; &gt; another config to set the Comparable implementation, say
> > > &gt; &gt; &gt; hoodie.payload.ordering.comparable.class to allow custom
> > > logic for
> > > &gt; doing
> > > &gt; &gt; &gt; comparison.
> > > &gt; &gt; &gt;
> > > &gt; &gt; &gt; Any suggestions? Thanks.
> > > &gt; &gt; &gt;
> > > &gt; &gt;
> > > &gt;
> >
>

Re: [DISCUSS] Support multiple ordering fields

Posted by Raymond Xu <xu...@gmail.com>.
Thank you for the feedbacks. I'm also in favor of simplifying the delta
streamer's usage.

@vinoth, building on what you said, would it be simpler to merge the
ordering field config with the payload class config?
i.e., to remove both --source-ordering-field and payload ordering field
config, then define the ordering field within the payload class.
People who use fields other than `ts` will need to supply a payload class
and override it.
People who already supply a payload class will just override it in the
existing custom class.
WDYT?


On Tue, Jan 19, 2021 at 10:40 AM Pratyaksh Sharma <pr...@gmail.com>
wrote:

> Hi
>
> We can use transformer to have a combination of multiple ordering fields.
> However custom Comparable implementation is not possible in that case. So
> overall a +1 from my side as well.
>
> On Tue, Jan 19, 2021 at 1:58 PM 刘金辉 <96...@qq.com> wrote:
>
> > +1,&nbsp;Currently we have encountered such scenarios and look forward to
> > supporting
> >
> >
> >
> >
> > ------------------&nbsp;原始邮件&nbsp;------------------
> > 发件人:
> >                                                   "dev"
> >                                                                 <
> > danny0405@apache.org&gt;;
> > 发送时间:&nbsp;2021年1月19日(星期二) 下午4:25
> > 收件人:&nbsp;"dev"<dev@hudi.apache.org&gt;;
> >
> > 主题:&nbsp;Re: [DISCUSS] Support multiple ordering fields
> >
> >
> >
> > &gt; Wondering if we should just take a bunch of payload configs and
> > deprecate
> > these flags
> >
> > I have the same feeling, there are already so many config options in
> > Hoodie, the maintain work for developers or users is hard.
> >
> > Vinoth Chandar <vinoth@apache.org&gt; 于2021年1月18日周一 下午11:40写道:
> >
> > &gt; +1 as well.
> > &gt;
> > &gt; Slightly orthogonal point.
> > &gt;
> > &gt; Wondering if we should just take a bunch of payload configs and
> > deprecate
> > &gt; these flags?
> > &gt; What I mean is : --source-ordering-field ultimately is used by the
> > &gt; HoodieAvroPayload class (or its family).
> > &gt; Our utilities expose these as flags for convenience, but its just
> > more work
> > &gt; to maintain.
> > &gt;
> > &gt; On Mon, Jan 18, 2021 at 7:09 AM vino yang <yanghua1127@gmail.com
> &gt;
> > wrote:
> > &gt;
> > &gt; &gt; +1,
> > &gt; &gt;
> > &gt; &gt; We have found that such flexibility is needed in some
> scenarios.
> > &gt; &gt;
> > &gt; &gt; Best,
> > &gt; &gt; Vino
> > &gt; &gt;
> > &gt; &gt; Raymond Xu <xu.shiyan.raymond@gmail.com&gt; 于2021年1月17日周日
> > 上午3:38写道:
> > &gt; &gt;
> > &gt; &gt; &gt; Just want to discuss a small improvement for setting
> > ordering fields.
> > &gt; &gt; &gt; For
> > &gt; &gt; &gt; - property `hoodie.payload.ordering.field` and
> > &gt; &gt; &gt; - deltastreamer --source-ordering-field
> > &gt; &gt; &gt; I think it can be useful to support multiple fields
> > (configured via a
> > &gt; &gt; &gt; comma-separated list) to determine the order in some
> cases.
> > This would
> > &gt; &gt; need
> > &gt; &gt; &gt; another config to set the Comparable implementation, say
> > &gt; &gt; &gt; hoodie.payload.ordering.comparable.class to allow custom
> > logic for
> > &gt; doing
> > &gt; &gt; &gt; comparison.
> > &gt; &gt; &gt;
> > &gt; &gt; &gt; Any suggestions? Thanks.
> > &gt; &gt; &gt;
> > &gt; &gt;
> > &gt;
>

Re: [DISCUSS] Support multiple ordering fields

Posted by Pratyaksh Sharma <pr...@gmail.com>.
Hi

We can use transformer to have a combination of multiple ordering fields.
However custom Comparable implementation is not possible in that case. So
overall a +1 from my side as well.

On Tue, Jan 19, 2021 at 1:58 PM 刘金辉 <96...@qq.com> wrote:

> +1,&nbsp;Currently we have encountered such scenarios and look forward to
> supporting
>
>
>
>
> ------------------&nbsp;原始邮件&nbsp;------------------
> 发件人:
>                                                   "dev"
>                                                                 <
> danny0405@apache.org&gt;;
> 发送时间:&nbsp;2021年1月19日(星期二) 下午4:25
> 收件人:&nbsp;"dev"<dev@hudi.apache.org&gt;;
>
> 主题:&nbsp;Re: [DISCUSS] Support multiple ordering fields
>
>
>
> &gt; Wondering if we should just take a bunch of payload configs and
> deprecate
> these flags
>
> I have the same feeling, there are already so many config options in
> Hoodie, the maintain work for developers or users is hard.
>
> Vinoth Chandar <vinoth@apache.org&gt; 于2021年1月18日周一 下午11:40写道:
>
> &gt; +1 as well.
> &gt;
> &gt; Slightly orthogonal point.
> &gt;
> &gt; Wondering if we should just take a bunch of payload configs and
> deprecate
> &gt; these flags?
> &gt; What I mean is : --source-ordering-field ultimately is used by the
> &gt; HoodieAvroPayload class (or its family).
> &gt; Our utilities expose these as flags for convenience, but its just
> more work
> &gt; to maintain.
> &gt;
> &gt; On Mon, Jan 18, 2021 at 7:09 AM vino yang <yanghua1127@gmail.com&gt;
> wrote:
> &gt;
> &gt; &gt; +1,
> &gt; &gt;
> &gt; &gt; We have found that such flexibility is needed in some scenarios.
> &gt; &gt;
> &gt; &gt; Best,
> &gt; &gt; Vino
> &gt; &gt;
> &gt; &gt; Raymond Xu <xu.shiyan.raymond@gmail.com&gt; 于2021年1月17日周日
> 上午3:38写道:
> &gt; &gt;
> &gt; &gt; &gt; Just want to discuss a small improvement for setting
> ordering fields.
> &gt; &gt; &gt; For
> &gt; &gt; &gt; - property `hoodie.payload.ordering.field` and
> &gt; &gt; &gt; - deltastreamer --source-ordering-field
> &gt; &gt; &gt; I think it can be useful to support multiple fields
> (configured via a
> &gt; &gt; &gt; comma-separated list) to determine the order in some cases.
> This would
> &gt; &gt; need
> &gt; &gt; &gt; another config to set the Comparable implementation, say
> &gt; &gt; &gt; hoodie.payload.ordering.comparable.class to allow custom
> logic for
> &gt; doing
> &gt; &gt; &gt; comparison.
> &gt; &gt; &gt;
> &gt; &gt; &gt; Any suggestions? Thanks.
> &gt; &gt; &gt;
> &gt; &gt;
> &gt;

回复: [DISCUSS] Support multiple ordering fields

Posted by 刘金辉 <96...@qq.com>.
+1,&nbsp;Currently we have encountered such scenarios and look forward to supporting




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "dev"                                                                                    <danny0405@apache.org&gt;;
发送时间:&nbsp;2021年1月19日(星期二) 下午4:25
收件人:&nbsp;"dev"<dev@hudi.apache.org&gt;;

主题:&nbsp;Re: [DISCUSS] Support multiple ordering fields



&gt; Wondering if we should just take a bunch of payload configs and deprecate
these flags

I have the same feeling, there are already so many config options in
Hoodie, the maintain work for developers or users is hard.

Vinoth Chandar <vinoth@apache.org&gt; 于2021年1月18日周一 下午11:40写道:

&gt; +1 as well.
&gt;
&gt; Slightly orthogonal point.
&gt;
&gt; Wondering if we should just take a bunch of payload configs and deprecate
&gt; these flags?
&gt; What I mean is : --source-ordering-field ultimately is used by the
&gt; HoodieAvroPayload class (or its family).
&gt; Our utilities expose these as flags for convenience, but its just more work
&gt; to maintain.
&gt;
&gt; On Mon, Jan 18, 2021 at 7:09 AM vino yang <yanghua1127@gmail.com&gt; wrote:
&gt;
&gt; &gt; +1,
&gt; &gt;
&gt; &gt; We have found that such flexibility is needed in some scenarios.
&gt; &gt;
&gt; &gt; Best,
&gt; &gt; Vino
&gt; &gt;
&gt; &gt; Raymond Xu <xu.shiyan.raymond@gmail.com&gt; 于2021年1月17日周日 上午3:38写道:
&gt; &gt;
&gt; &gt; &gt; Just want to discuss a small improvement for setting ordering fields.
&gt; &gt; &gt; For
&gt; &gt; &gt; - property `hoodie.payload.ordering.field` and
&gt; &gt; &gt; - deltastreamer --source-ordering-field
&gt; &gt; &gt; I think it can be useful to support multiple fields (configured via a
&gt; &gt; &gt; comma-separated list) to determine the order in some cases. This would
&gt; &gt; need
&gt; &gt; &gt; another config to set the Comparable implementation, say
&gt; &gt; &gt; hoodie.payload.ordering.comparable.class to allow custom logic for
&gt; doing
&gt; &gt; &gt; comparison.
&gt; &gt; &gt;
&gt; &gt; &gt; Any suggestions? Thanks.
&gt; &gt; &gt;
&gt; &gt;
&gt;

Re: [DISCUSS] Support multiple ordering fields

Posted by Danny Chan <da...@apache.org>.
> Wondering if we should just take a bunch of payload configs and deprecate
these flags

I have the same feeling, there are already so many config options in
Hoodie, the maintain work for developers or users is hard.

Vinoth Chandar <vi...@apache.org> 于2021年1月18日周一 下午11:40写道:

> +1 as well.
>
> Slightly orthogonal point.
>
> Wondering if we should just take a bunch of payload configs and deprecate
> these flags?
> What I mean is : --source-ordering-field ultimately is used by the
> HoodieAvroPayload class (or its family).
> Our utilities expose these as flags for convenience, but its just more work
> to maintain.
>
> On Mon, Jan 18, 2021 at 7:09 AM vino yang <ya...@gmail.com> wrote:
>
> > +1,
> >
> > We have found that such flexibility is needed in some scenarios.
> >
> > Best,
> > Vino
> >
> > Raymond Xu <xu...@gmail.com> 于2021年1月17日周日 上午3:38写道:
> >
> > > Just want to discuss a small improvement for setting ordering fields.
> > > For
> > > - property `hoodie.payload.ordering.field` and
> > > - deltastreamer --source-ordering-field
> > > I think it can be useful to support multiple fields (configured via a
> > > comma-separated list) to determine the order in some cases. This would
> > need
> > > another config to set the Comparable implementation, say
> > > hoodie.payload.ordering.comparable.class to allow custom logic for
> doing
> > > comparison.
> > >
> > > Any suggestions? Thanks.
> > >
> >
>

Re: [DISCUSS] Support multiple ordering fields

Posted by Vinoth Chandar <vi...@apache.org>.
+1 as well.

Slightly orthogonal point.

Wondering if we should just take a bunch of payload configs and deprecate
these flags?
What I mean is : --source-ordering-field ultimately is used by the
HoodieAvroPayload class (or its family).
Our utilities expose these as flags for convenience, but its just more work
to maintain.

On Mon, Jan 18, 2021 at 7:09 AM vino yang <ya...@gmail.com> wrote:

> +1,
>
> We have found that such flexibility is needed in some scenarios.
>
> Best,
> Vino
>
> Raymond Xu <xu...@gmail.com> 于2021年1月17日周日 上午3:38写道:
>
> > Just want to discuss a small improvement for setting ordering fields.
> > For
> > - property `hoodie.payload.ordering.field` and
> > - deltastreamer --source-ordering-field
> > I think it can be useful to support multiple fields (configured via a
> > comma-separated list) to determine the order in some cases. This would
> need
> > another config to set the Comparable implementation, say
> > hoodie.payload.ordering.comparable.class to allow custom logic for doing
> > comparison.
> >
> > Any suggestions? Thanks.
> >
>

Re: [DISCUSS] Support multiple ordering fields

Posted by vino yang <ya...@gmail.com>.
+1,

We have found that such flexibility is needed in some scenarios.

Best,
Vino

Raymond Xu <xu...@gmail.com> 于2021年1月17日周日 上午3:38写道:

> Just want to discuss a small improvement for setting ordering fields.
> For
> - property `hoodie.payload.ordering.field` and
> - deltastreamer --source-ordering-field
> I think it can be useful to support multiple fields (configured via a
> comma-separated list) to determine the order in some cases. This would need
> another config to set the Comparable implementation, say
> hoodie.payload.ordering.comparable.class to allow custom logic for doing
> comparison.
>
> Any suggestions? Thanks.
>