You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "stayrascal (via GitHub)" <gi...@apache.org> on 2024/02/24 15:01:19 UTC
[PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
stayrascal opened a new pull request, #10744:
URL: https://github.com/apache/hudi/pull/10744
As [Hudi-7441](https://issues.apache.org/jira/browse/HUDI-7441) mentioned, a normal flink streaming reading hudi case should not include hive dependency, but since the `IncrementalInputSplits#getReadPartitions` method will call `
HoodieInputFormatUtils.getWritePartitionPaths` method, and `HoodieInputFormatUtils` imported hive classes, so we have to include hive dependency in the use case without hive integration.
### Change Logs
Move the `IncrementalInputSplits#getReadPartitions` method to `HoodieCommitMetadataUtils#getReadPartitions`
### Impact
No
### Risk level (write none, low medium or high below)
Low
### Documentation Update
No
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1962480290
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22604",
"triggerID" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c903b39d758f9ee365ad93d6dd3b2ff745eee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22604)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "stayrascal (via GitHub)" <gi...@apache.org>.
stayrascal commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1968069640
@danny0405 do we need to cheery pick to 0.x or 0.14 branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1972293330
Sure, my fault, this is a critical fix for flink streaming read, let's cherry pick it into the branch-0.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1963303343
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22604",
"triggerID" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"triggerType" : "PUSH"
}, {
"hash" : "ba1ae3434440bf73d4a32465de630ceffaffcc97",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22632",
"triggerID" : "ba1ae3434440bf73d4a32465de630ceffaffcc97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ba1ae3434440bf73d4a32465de630ceffaffcc97 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22632)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1972294347
Can you also update our site to remove the key requirements: https://hudi.apache.org/docs/flink-quick-start-guide#streaming-query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on code in PR #10744:
URL: https://github.com/apache/hudi/pull/10744#discussion_r1501877329
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieCommitMetadataUtils.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class HoodieCommitMetadataUtils {
+ /**
+ * Returns all the incremental write partition paths as a set with the given commits metadata.
+ *
+ * @param metadataList The commits metadata
+ * @return the partition path set
+ */
+ public static Set<String> getWritePartitionPaths(List<HoodieCommitMetadata> metadataList) {
Review Comment:
Instead of a new Utils file, can we add this method to org.apache.hudi.metadata.HoodieTableMetadataUtil ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1962403493
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c903b39d758f9ee365ad93d6dd3b2ff745eee74 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1968101197
I don't think so, it is just a code refactoring right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "stayrascal (via GitHub)" <gi...@apache.org>.
stayrascal commented on code in PR #10744:
URL: https://github.com/apache/hudi/pull/10744#discussion_r1501953899
##########
hudi-common/src/main/java/org/apache/hudi/common/util/HoodieCommitMetadataUtils.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.util;
+
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+
+import java.util.Collection;
+import java.util.List;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+public class HoodieCommitMetadataUtils {
+ /**
+ * Returns all the incremental write partition paths as a set with the given commits metadata.
+ *
+ * @param metadataList The commits metadata
+ * @return the partition path set
+ */
+ public static Set<String> getWritePartitionPaths(List<HoodieCommitMetadata> metadataList) {
Review Comment:
it's a good idea, thx for reviewing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1962405249
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22604",
"triggerID" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c903b39d758f9ee365ad93d6dd3b2ff745eee74 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22604)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "stayrascal (via GitHub)" <gi...@apache.org>.
stayrascal commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1968175524
> I don't think so, it is just a code refactoring right?
It's also solve the problem that we have to include hive dependency jar(e.g. flink-hive-connector) during flink streaming read a normal hudi table, even through the flink job doesn't do any integration with hive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1963176882
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22604",
"triggerID" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"triggerType" : "PUSH"
}, {
"hash" : "ba1ae3434440bf73d4a32465de630ceffaffcc97",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ba1ae3434440bf73d4a32465de630ceffaffcc97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c903b39d758f9ee365ad93d6dd3b2ff745eee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22604)
* ba1ae3434440bf73d4a32465de630ceffaffcc97 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar merged PR #10744:
URL: https://github.com/apache/hudi/pull/10744
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7441] Move `getWritePartitionPaths` method to common module to … [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10744:
URL: https://github.com/apache/hudi/pull/10744#issuecomment-1963183995
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22604",
"triggerID" : "2c903b39d758f9ee365ad93d6dd3b2ff745eee74",
"triggerType" : "PUSH"
}, {
"hash" : "ba1ae3434440bf73d4a32465de630ceffaffcc97",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22632",
"triggerID" : "ba1ae3434440bf73d4a32465de630ceffaffcc97",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2c903b39d758f9ee365ad93d6dd3b2ff745eee74 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22604)
* ba1ae3434440bf73d4a32465de630ceffaffcc97 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22632)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org