You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "lookingUpAtTheSky (via GitHub)" <gi...@apache.org> on 2025/03/27 09:33:59 UTC
[I] Iceberg potential metadata conflicts [iceberg]
lookingUpAtTheSky opened a new issue, #12660:
URL: https://github.com/apache/iceberg/issues/12660
### Query engine
Spark
### Question
When HiveCatalog is used, the new metadata file uses the current version plus 1 and then combines the newly generated UUID. In high-concurrency scenarios, there is a possibility that iceberg data consistency errors occur due to UUID conflicts. Have you considered solving this problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg potential metadata conflicts [iceberg]
Posted by "lookingUpAtTheSky (via GitHub)" <gi...@apache.org>.
lookingUpAtTheSky commented on issue #12660:
URL: https://github.com/apache/iceberg/issues/12660#issuecomment-2763208727
Ok, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg potential metadata conflicts [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat closed issue #12660: Iceberg potential metadata conflicts
URL: https://github.com/apache/iceberg/issues/12660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg potential metadata conflicts [iceberg]
Posted by "ajantha-bhat (via GitHub)" <gi...@apache.org>.
ajantha-bhat commented on issue #12660:
URL: https://github.com/apache/iceberg/issues/12660#issuecomment-2758221748
I agree that UUID conflicts are impossible with limited concurrency (say few hundred commits at a time).
https://stackoverflow.com/questions/24876188/how-big-is-the-chance-to-get-a-java-uuid-randomuuid-collision
I also analyzed long back for snapshot id conflicts (here UUID was truncated / XORed)
https://github.com/apache/iceberg/pull/4747#issuecomment-1126806592
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg potential metadata conflicts [iceberg]
Posted by "pvary (via GitHub)" <gi...@apache.org>.
pvary commented on issue #12660:
URL: https://github.com/apache/iceberg/issues/12660#issuecomment-2757577064
Theoretically the UUID could conflict, but the probability is very low. See: https://en.wikipedia.org/w/index.php?title=Universally_unique_identifier&oldid=755882275#Random_UUID_probability_of_duplicates
> To put these numbers into perspective, the annual risk of a given person being hit by a meteorite is estimated to be one chance in 17 billion,[[5]](https://en.wikipedia.org/w/index.php?title=Universally_unique_identifier&oldid=755882275#cite_note-5) which means the probability is about 0.00000000006 (6 × 10−11), equivalent to the odds of creating a few tens of trillions of UUIDs in a year and having one duplicate. In other words, only after generating 1 billion UUIDs every second for approximately 100 years would the probability of creating a single duplicate reach 50%.
If you have faced the issue, then you might have some entropy issues with the random generator of the system.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org