You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/16 14:09:39 UTC
[GitHub] [hudi] XuQianJin-Stars opened a new pull request, #6963: [WIP][HUDI-5040] flink multi writer for bucket index
XuQianJin-Stars opened a new pull request, #6963:
URL: https://github.com/apache/hudi/pull/6963
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
**Risk level: none | low | medium | high**
_Choose one. If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on pull request #6963: [WIP][HUDI-5040] flink multi writer for bucket index
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on PR #6963:
URL: https://github.com/apache/hudi/pull/6963#issuecomment-1280424464
nice work！
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wulei0302 commented on a diff in pull request #6963: [WIP][HUDI-5040] flink multi writer for bucket index
Posted by GitBox <gi...@apache.org>.
wulei0302 commented on code in PR #6963:
URL: https://github.com/apache/hudi/pull/6963#discussion_r1021110814
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -164,20 +166,31 @@ private void init(HoodieRecord record) {
String baseInstantTime;
String baseFile = "";
List<String> logFiles = new ArrayList<>();
+ Option<HoodieInstant> maxCompleteInstant = hoodieTable.getMetaClient().getActiveTimeline().getWriteTimeline()
+ .filterMorCompactionInstants().lastInstant();
+ if (maxCompleteInstant.isPresent()) {
+ if (fileSlice.isPresent()) {
+ baseInstantTime = fileSlice.get().getBaseInstantTime();
+ } else {
+ baseInstantTime = instantTime;
+ // Handle log file only case. This is necessary for the concurrent clustering and writer case (e.g., consistent hashing bucket index).
+ // NOTE: flink engine use instantTime to mark operation type, check BaseFlinkCommitActionExecutor::execute
+ if (record.getCurrentLocation() != null && HoodieInstantTimeGenerator.isValidInstantTime(record.getCurrentLocation().getInstantTime())) {
+ baseInstantTime = record.getCurrentLocation().getInstantTime();
+ }
+ }
+ } else {
+ String instantTime = HoodieActiveTimeline.createNewInstantTime();
+ baseInstantTime = instantTime.substring(0, instantTime.length() - 9) + String.format("%09d", 0);
Review Comment:
What if the instant time for concurrent writing just spans 24:00. For example,
instant_ 1 is 20221113235900000,
instant_ 2 is 20221114000100000,
and then instant_ 2 Finish first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org