You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "parisni (via GitHub)" <gi...@apache.org> on 2023/05/04 14:30:37 UTC
[GitHub] [hudi] parisni opened a new issue, #8634: [SUPPORT] OCC writes + hive sync results in missing partitions in metastore
parisni opened a new issue, #8634:
URL: https://github.com/apache/hudi/issues/8634
Hudi > 0.10 (OCC feature)
When running multiple concurrent ingress process in a given table, which ads new partitions then it leads sometimes to missing partitions.
This happens when some ingress are faster than others.
For eg, say we have 2 ingress processes:
```
p1 starts at commit1 and will add part1
p2 starts at commit2 and will add part2
p2 finishes and sync the metastore to adds part2 and set metastore_last_commit = commit2
p1 finishes and consider the metastore as already synced because commit1 < metastore_last_commit
```
A way to mitigate this would be to force sync the current commit even if current_commit < metastore_last_commit
We also should consider to not throw an error when a partition already exists (which is already the case for glue metastore)
Thought ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed issue #8634: [SUPPORT] OCC writes + hive sync results in missing partitions in metastore
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #8634: [SUPPORT] OCC writes + hive sync results in missing partitions in metastore
URL: https://github.com/apache/hudi/issues/8634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8634: [SUPPORT] OCC writes + hive sync results in missing partitions in metastore
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8634:
URL: https://github.com/apache/hudi/issues/8634#issuecomment-1535669758
Another issue related: https://github.com/apache/hudi/issues/8500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8634: [SUPPORT] OCC writes + hive sync results in missing partitions in metastore
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8634:
URL: https://github.com/apache/hudi/issues/8634#issuecomment-1535668529
We just introduces instant completion time support: https://github.com/apache/hudi/pull/7627, it could help for your multi-writer commits meta sync.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org