[Feature Request] Support "faking" hudi commit time with the value of some field in the record

[Joseph Thaidigsman <jt...@slack-corp.com.INVALID>]

Hello,

We have a use-case where we have persisted the full CDC changelog for some
tables in s3 and want to be able to bootstrap hudi tables with the
changelog data and then be able to time-travel the hudi table to get
snapshot views of the table on dates prior to bootstrapping. In our
changelog, we have the timestamp associated with the
inserts/updates/deletes, so the data to achieve this is present. If we had
a live consumer processing those events in real-time and writing them to a
hudi table, then we would be able to achieve this, but because we are
instead creating the hudi table from a single batch job, we are unable to
achieve it despite processing the same exact data, since time-travel is all
based on the hudi commit time.

Aside from our specific use-case for bootstrapping tables, this would be
useful for real-time CDC consumers as well.  Currently, there is no way to
guarantee the accuracy of the time-travel operation as it relates to
reflecting the state of the upstream database table at a given point in
time. For example, say you have some downstream batch pipelines that want
to perform some aggregations based on production database tables at a fixed
point each day. In the case of lag or outage on the consumer-side, when the
consumer restarts, we have a large gap in hudi commit time and are unable
to time-travel to the exact moment that the downstream pipelines expect to
reflect the database table state.

If the hudi writer instead supported picking some field from the CDC record
as the value for the hudi commit time, then the consumer could process the
events at any time and the time-travel functionality would be the same
regardless of consumption time. This would make the writer idempotent in a
way that it currently lacks, guaranteeing consistent results for downstream
pipelines.

Original Slack Thread:
https://apache-hudi.slack.com/archives/C4D716NPQ/p1690583690053259

Re: [Feature Request] Support "faking" hudi commit time with the value of some field in the record

[Vinoth Chandar <vi...@apache.org>]

(sorry for the late reply)

Hi - the commit time can be a logical time as well, a lot of tests work
this way. There may be some table features (e.g time based cleaning) that
may not work, but those are more convenience ones anyway.

I assume, the consumer would process all events at the required source
timestamp boundary to achieve this?

I am happy to chat/help scope the changes more.



On Wed, Aug 2, 2023 at 1:17 PM Joseph Thaidigsman
<jt...@slack-corp.com.invalid> wrote:

> Hello,
>
> We have a use-case where we have persisted the full CDC changelog for some
> tables in s3 and want to be able to bootstrap hudi tables with the
> changelog data and then be able to time-travel the hudi table to get
> snapshot views of the table on dates prior to bootstrapping. In our
> changelog, we have the timestamp associated with the
> inserts/updates/deletes, so the data to achieve this is present. If we had
> a live consumer processing those events in real-time and writing them to a
> hudi table, then we would be able to achieve this, but because we are
> instead creating the hudi table from a single batch job, we are unable to
> achieve it despite processing the same exact data, since time-travel is all
> based on the hudi commit time.
>
> Aside from our specific use-case for bootstrapping tables, this would be
> useful for real-time CDC consumers as well.  Currently, there is no way to
> guarantee the accuracy of the time-travel operation as it relates to
> reflecting the state of the upstream database table at a given point in
> time. For example, say you have some downstream batch pipelines that want
> to perform some aggregations based on production database tables at a fixed
> point each day. In the case of lag or outage on the consumer-side, when the
> consumer restarts, we have a large gap in hudi commit time and are unable
> to time-travel to the exact moment that the downstream pipelines expect to
> reflect the database table state.
>
> If the hudi writer instead supported picking some field from the CDC record
> as the value for the hudi commit time, then the consumer could process the
> events at any time and the time-travel functionality would be the same
> regardless of consumption time. This would make the writer idempotent in a
> way that it currently lacks, guaranteeing consistent results for downstream
> pipelines.
>
> Original Slack Thread:
> https://apache-hudi.slack.com/archives/C4D716NPQ/p1690583690053259
>