You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "shanzhongkai (via GitHub)" <gi...@apache.org> on 2025/05/01 17:19:39 UTC
[I] [SUPPORT] In the scene of using Flink to write (upsert) hudi, if index.bootstrap.enabled=false(default), does it mean that the primary key within the partition cannot be guaranteed to be unique? [hudi]
shanzhongkai opened a new issue, #13244:
URL: https://github.com/apache/hudi/issues/13244
**Describe the problem you faced**
In the scene of using Flink to write (upsert) hudi, if index.bootstrap.enabled=false(default), does it mean that the primary key within the partition cannot be guaranteed to be unique?
**Environment Description**
* Hudi version : newest
**Additional context**

I saw that there is a logic when generating a Pipeline that only adds the bootstrap operator if **index. bootstrap. enabled** equals true. The bootstrap operator is a key step in deduplicating historical data. Does this mean that when **index.bootstrap.enabled**=false, performing an upsert operation will break the uniqueness of the primary key?If so, why is it designed like this? thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] In the scene of using Flink to write (upsert) hudi, if index.bootstrap.enabled=false(default), does it mean that the primary key within the partition cannot be guaranteed to be unique? [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #13244:
URL: https://github.com/apache/hudi/issues/13244#issuecomment-2846748370
yes, another choice is to use the `BUCKET` index, which does not need state for index storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] In the scene of using Flink to write (upsert) hudi, if index.bootstrap.enabled=false(default), does it mean that the primary key within the partition cannot be guaranteed to be unique? [hudi]
Posted by "shanzhongkai (via GitHub)" <gi...@apache.org>.
shanzhongkai closed issue #13244: [SUPPORT] In the scene of using Flink to write (upsert) hudi, if index.bootstrap.enabled=false(default), does it mean that the primary key within the partition cannot be guaranteed to be unique?
URL: https://github.com/apache/hudi/issues/13244
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] In the scene of using Flink to write (upsert) hudi, if index.bootstrap.enabled=false(default), does it mean that the primary key within the partition cannot be guaranteed to be unique? [hudi]
Posted by "shanzhongkai (via GitHub)" <gi...@apache.org>.
shanzhongkai commented on issue #13244:
URL: https://github.com/apache/hudi/issues/13244#issuecomment-2846301184
> > does it mean that the primary key within the partition cannot be guaranteed to be unique
>
> for single partition, the primary key is always unique.
I have observed that if the index bootstrap is not enabled, no index information will be loaded into the state. This means that if I want to write some data to the **date=2020** partition, since the index information of the baseFile already exists in date=**2020** will not be loaded, the upsert data in **date=2020** will not be able to obtain duplicate information and cannot guarantee uniqueness. Is there any problem with my understanding?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] In the scene of using Flink to write (upsert) hudi, if index.bootstrap.enabled=false(default), does it mean that the primary key within the partition cannot be guaranteed to be unique? [hudi]
Posted by "shanzhongkai (via GitHub)" <gi...@apache.org>.
shanzhongkai commented on issue #13244:
URL: https://github.com/apache/hudi/issues/13244#issuecomment-2847161032
Understood, thanks for answer~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] In the scene of using Flink to write (upsert) hudi, if index.bootstrap.enabled=false(default), does it mean that the primary key within the partition cannot be guaranteed to be unique? [hudi]
Posted by "shanzhongkai (via GitHub)" <gi...@apache.org>.
shanzhongkai commented on issue #13244:
URL: https://github.com/apache/hudi/issues/13244#issuecomment-2846576913
To add, for example, in the case of a stateless restart of a job, the index in state will be lost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] In the scene of using Flink to write (upsert) hudi, if index.bootstrap.enabled=false(default), does it mean that the primary key within the partition cannot be guaranteed to be unique? [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #13244:
URL: https://github.com/apache/hudi/issues/13244#issuecomment-2846053456
> does it mean that the primary key within the partition cannot be guaranteed to be unique
Not really, the index bootstrap is used to keep uniqueness for records scattered in multiple partitions, for single partition, the primary key is always unique.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org