You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2024/12/09 12:21:14 UTC
[PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
TheR1sing3un opened a new pull request, #12451:
URL: https://github.com/apache/hudi/pull/12451
When perform clustering to adjust bucket num for consistent-bucket, it is unnecessary to go through all the records to find the partitions to which the mapping relationship is to be loaded, because clustering plan already knows which partitions are required.
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
1. improve consistent-bucket resizing performance by reducing unnecessary record collecting
### Impact
_Describe any public API or user-facing feature change or any performance impact._
none
### Risk level (write none, low medium or high below)
none
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
none
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12451:
URL: https://github.com/apache/hudi/pull/12451#discussion_r1889636893
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/ConsistentHashingBucketInsertPartitioner.java:
##########
@@ -29,5 +29,14 @@ public interface ConsistentHashingBucketInsertPartitioner {
* @param partition partition to set Consistent Hashing nodes
* @param nodes nodes from clustering plan
*/
+ /**
+ * Set pending consistent hashing for partition, only used in executing clustering
+ * When call this method, the bulk insert will directly use the pending metadata as the consistent hash metadata for writing data to after-resizing buckets.
+ * Used in the case of executing bulk insert.
+ * NOTE: This method should be called before the bulk insert operation, and will skip building identifiers from records, just use the pending metadata.
+ * For which not calling this method, the bulk insert will use the committed metadata as the bucket metadata and disallow writing data to the pending-resizing buckets.
+ * @param partition partition to set Consistent Hashing nodes
+ * @param nodes nodes from clustering plan
+ */
Review Comment:
> yeah, if the SE/DE works well, maybe we just put the metadata in the partitioner constructor.
Done~ I've refactored this part of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12451:
URL: https://github.com/apache/hudi/pull/12451#discussion_r1896355013
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/RDDConsistentBucketBulkInsertPartitioner.java:
##########
@@ -49,25 +48,48 @@
/**
* A partitioner for (consistent hashing) bucket index used in bulk_insert
*/
-public class RDDConsistentBucketBulkInsertPartitioner<T> extends RDDBucketIndexPartitioner<T> implements ConsistentHashingBucketInsertPartitioner {
+public class RDDConsistentBucketBulkInsertPartitioner<T> extends RDDBucketIndexPartitioner<T> {
- private final Map<String, List<ConsistentHashingNode>> hashingChildrenNodes;
+ private Map<String/*partition*/, List<ConsistentHashingNode/*pending resizing related child nodes*/>> hashingChildrenNodes;
+
+ // mark if this partitioner is used for writing to uncommitted buckets. Only for case that clustering service executes bucket resizing.
+ private boolean isExecutingClustering = false;
Review Comment:
> The flag can be eliminated.
Removed~
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/ConsistentBucketIndexBulkInsertPartitionerWithRows.java:
##########
@@ -54,14 +52,16 @@
/**
* Bulk_insert partitioner of Spark row using consistent hashing bucket index.
*/
-public class ConsistentBucketIndexBulkInsertPartitionerWithRows
- extends BucketSortBulkInsertPartitioner<Dataset<Row>> implements ConsistentHashingBucketInsertPartitioner {
+public class ConsistentBucketIndexBulkInsertPartitionerWithRows extends BucketSortBulkInsertPartitioner<Dataset<Row>> {
private final String indexKeyFields;
private final List<String> fileIdPfxList = new ArrayList<>();
- private final Map<String, List<ConsistentHashingNode>> hashingChildrenNodes;
+ private Map<String/*partition*/, List<ConsistentHashingNode>/*pending resizing related child nodes*/> hashingChildrenNodes;
+
+ // mark if this partitioner is used for writing to uncommitted buckets. Only for case that clustering service executes bucket resizing.
+ private boolean isExecutingClustering = false;
Review Comment:
> The flag can be eliminated.
Removed~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2550486840
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
}, {
"hash" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480",
"triggerID" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f457d79a2628589f00a02bc9cf965d76a0520b43 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2560615732
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
}, {
"hash" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480",
"triggerID" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"triggerType" : "PUSH"
}, {
"hash" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2506",
"triggerID" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"triggerType" : "PUSH"
}, {
"hash" : "f224eadf25cb4feeb162f405cbd8a5e8250c791f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f224eadf25cb4feeb162f405cbd8a5e8250c791f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ededfcf1096006e5a8378142dd5a41db5f02472c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2506)
* f224eadf25cb4feeb162f405cbd8a5e8250c791f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2528000372
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 16d1021628a482733e49d21c742a754638f5aabb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2553152683
@danny0405 Hi, code has been updated according to your previous review, let's continue to review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12451:
URL: https://github.com/apache/hudi/pull/12451#discussion_r1889607418
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/ConsistentHashingBucketInsertPartitioner.java:
##########
@@ -29,5 +29,14 @@ public interface ConsistentHashingBucketInsertPartitioner {
* @param partition partition to set Consistent Hashing nodes
* @param nodes nodes from clustering plan
*/
+ /**
+ * Set pending consistent hashing for partition, only used in executing clustering
+ * When call this method, the bulk insert will directly use the pending metadata as the consistent hash metadata for writing data to after-resizing buckets.
+ * Used in the case of executing bulk insert.
+ * NOTE: This method should be called before the bulk insert operation, and will skip building identifiers from records, just use the pending metadata.
+ * For which not calling this method, the bulk insert will use the committed metadata as the bucket metadata and disallow writing data to the pending-resizing buckets.
+ * @param partition partition to set Consistent Hashing nodes
+ * @param nodes nodes from clustering plan
+ */
Review Comment:
yeah, if the SE/DE works well, maybe we just put the metadata in the partitioner constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12451:
URL: https://github.com/apache/hudi/pull/12451#discussion_r1877717384
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/ConsistentHashingBucketInsertPartitioner.java:
##########
@@ -29,5 +29,14 @@ public interface ConsistentHashingBucketInsertPartitioner {
* @param partition partition to set Consistent Hashing nodes
* @param nodes nodes from clustering plan
*/
+ /**
+ * Set pending consistent hashing for partition, only used in executing clustering
+ * When call this method, the bulk insert will directly use the pending metadata as the consistent hash metadata for writing data to after-resizing buckets.
+ * Used in the case of executing bulk insert.
+ * NOTE: This method should be called before the bulk insert operation, and will skip building identifiers from records, just use the pending metadata.
+ * For which not calling this method, the bulk insert will use the committed metadata as the bucket metadata and disallow writing data to the pending-resizing buckets.
+ * @param partition partition to set Consistent Hashing nodes
+ * @param nodes nodes from clustering plan
+ */
Review Comment:
> Does one partitioner instance works for only one partition during the whole life-cycle?
This is certainly true for consistent-bucket clustering.
<img width="1094" alt="image" src="https://github.com/user-attachments/assets/8a1e03c8-e226-4061-aa54-bde5e40f4e84">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2527784024
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 16d1021628a482733e49d21c742a754638f5aabb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2527780581
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 16d1021628a482733e49d21c742a754638f5aabb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12451:
URL: https://github.com/apache/hudi/pull/12451#discussion_r1881573645
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/ConsistentHashingBucketInsertPartitioner.java:
##########
@@ -29,5 +29,14 @@ public interface ConsistentHashingBucketInsertPartitioner {
* @param partition partition to set Consistent Hashing nodes
* @param nodes nodes from clustering plan
*/
+ /**
+ * Set pending consistent hashing for partition, only used in executing clustering
+ * When call this method, the bulk insert will directly use the pending metadata as the consistent hash metadata for writing data to after-resizing buckets.
+ * Used in the case of executing bulk insert.
+ * NOTE: This method should be called before the bulk insert operation, and will skip building identifiers from records, just use the pending metadata.
+ * For which not calling this method, the bulk insert will use the committed metadata as the bucket metadata and disallow writing data to the pending-resizing buckets.
+ * @param partition partition to set Consistent Hashing nodes
+ * @param nodes nodes from clustering plan
+ */
Review Comment:
> Then shall we move the instantiation of the metadata once in the constructor? Not sure if there are some SE/DE issue.
What you means is that we remove the method: `ConsistentHashBucketInsertPartitioner::addHashingChildrenNodes`? And then in the clustering scenario, the partitioner is instantiated directly by carrying these metadata as construct parameters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12451:
URL: https://github.com/apache/hudi/pull/12451#discussion_r1881561262
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/ConsistentHashingBucketInsertPartitioner.java:
##########
@@ -29,5 +29,14 @@ public interface ConsistentHashingBucketInsertPartitioner {
* @param partition partition to set Consistent Hashing nodes
* @param nodes nodes from clustering plan
*/
+ /**
+ * Set pending consistent hashing for partition, only used in executing clustering
+ * When call this method, the bulk insert will directly use the pending metadata as the consistent hash metadata for writing data to after-resizing buckets.
+ * Used in the case of executing bulk insert.
+ * NOTE: This method should be called before the bulk insert operation, and will skip building identifiers from records, just use the pending metadata.
+ * For which not calling this method, the bulk insert will use the committed metadata as the bucket metadata and disallow writing data to the pending-resizing buckets.
+ * @param partition partition to set Consistent Hashing nodes
+ * @param nodes nodes from clustering plan
+ */
Review Comment:
Then shall we move the instantiation of the metadata once in the constructor? Not sure if there are some SE/DE issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2560616803
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
}, {
"hash" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480",
"triggerID" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"triggerType" : "PUSH"
}, {
"hash" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2506",
"triggerID" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"triggerType" : "PUSH"
}, {
"hash" : "f224eadf25cb4feeb162f405cbd8a5e8250c791f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2566",
"triggerID" : "f224eadf25cb4feeb162f405cbd8a5e8250c791f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ededfcf1096006e5a8378142dd5a41db5f02472c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2506)
* f224eadf25cb4feeb162f405cbd8a5e8250c791f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2566)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2550374152
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
}, {
"hash" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480",
"triggerID" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 16d1021628a482733e49d21c742a754638f5aabb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337)
* f457d79a2628589f00a02bc9cf965d76a0520b43 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #12451:
URL: https://github.com/apache/hudi/pull/12451#discussion_r1881573645
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/ConsistentHashingBucketInsertPartitioner.java:
##########
@@ -29,5 +29,14 @@ public interface ConsistentHashingBucketInsertPartitioner {
* @param partition partition to set Consistent Hashing nodes
* @param nodes nodes from clustering plan
*/
+ /**
+ * Set pending consistent hashing for partition, only used in executing clustering
+ * When call this method, the bulk insert will directly use the pending metadata as the consistent hash metadata for writing data to after-resizing buckets.
+ * Used in the case of executing bulk insert.
+ * NOTE: This method should be called before the bulk insert operation, and will skip building identifiers from records, just use the pending metadata.
+ * For which not calling this method, the bulk insert will use the committed metadata as the bucket metadata and disallow writing data to the pending-resizing buckets.
+ * @param partition partition to set Consistent Hashing nodes
+ * @param nodes nodes from clustering plan
+ */
Review Comment:
> Then shall we move the instantiation of the metadata once in the constructor? Not sure if there are some SE/DE issue.
What you means is that we remove the method: `ConsistentHashBucketInsertPartitioner::addHashingChildrenNodes`, and in the clustering scenario, the partitioner is instantiated directly by carrying these metadata as construct parameters?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2550372835
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
}, {
"hash" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 16d1021628a482733e49d21c742a754638f5aabb Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337)
* f457d79a2628589f00a02bc9cf965d76a0520b43 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2552786173
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
}, {
"hash" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480",
"triggerID" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"triggerType" : "PUSH"
}, {
"hash" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2506",
"triggerID" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ededfcf1096006e5a8378142dd5a41db5f02472c Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2506)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12451:
URL: https://github.com/apache/hudi/pull/12451#discussion_r1896320829
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/ConsistentBucketIndexBulkInsertPartitionerWithRows.java:
##########
@@ -54,14 +52,16 @@
/**
* Bulk_insert partitioner of Spark row using consistent hashing bucket index.
*/
-public class ConsistentBucketIndexBulkInsertPartitionerWithRows
- extends BucketSortBulkInsertPartitioner<Dataset<Row>> implements ConsistentHashingBucketInsertPartitioner {
+public class ConsistentBucketIndexBulkInsertPartitionerWithRows extends BucketSortBulkInsertPartitioner<Dataset<Row>> {
private final String indexKeyFields;
private final List<String> fileIdPfxList = new ArrayList<>();
- private final Map<String, List<ConsistentHashingNode>> hashingChildrenNodes;
+ private Map<String/*partition*/, List<ConsistentHashingNode>/*pending resizing related child nodes*/> hashingChildrenNodes;
+
+ // mark if this partitioner is used for writing to uncommitted buckets. Only for case that clustering service executes bucket resizing.
+ private boolean isExecutingClustering = false;
Review Comment:
The flag can be eliminated.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/execution/bulkinsert/RDDConsistentBucketBulkInsertPartitioner.java:
##########
@@ -49,25 +48,48 @@
/**
* A partitioner for (consistent hashing) bucket index used in bulk_insert
*/
-public class RDDConsistentBucketBulkInsertPartitioner<T> extends RDDBucketIndexPartitioner<T> implements ConsistentHashingBucketInsertPartitioner {
+public class RDDConsistentBucketBulkInsertPartitioner<T> extends RDDBucketIndexPartitioner<T> {
- private final Map<String, List<ConsistentHashingNode>> hashingChildrenNodes;
+ private Map<String/*partition*/, List<ConsistentHashingNode/*pending resizing related child nodes*/>> hashingChildrenNodes;
+
+ // mark if this partitioner is used for writing to uncommitted buckets. Only for case that clustering service executes bucket resizing.
+ private boolean isExecutingClustering = false;
Review Comment:
The flag can be eliminated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #12451:
URL: https://github.com/apache/hudi/pull/12451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #12451:
URL: https://github.com/apache/hudi/pull/12451#discussion_r1877709074
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/ConsistentHashingBucketInsertPartitioner.java:
##########
@@ -29,5 +29,14 @@ public interface ConsistentHashingBucketInsertPartitioner {
* @param partition partition to set Consistent Hashing nodes
* @param nodes nodes from clustering plan
*/
+ /**
+ * Set pending consistent hashing for partition, only used in executing clustering
+ * When call this method, the bulk insert will directly use the pending metadata as the consistent hash metadata for writing data to after-resizing buckets.
+ * Used in the case of executing bulk insert.
+ * NOTE: This method should be called before the bulk insert operation, and will skip building identifiers from records, just use the pending metadata.
+ * For which not calling this method, the bulk insert will use the committed metadata as the bucket metadata and disallow writing data to the pending-resizing buckets.
+ * @param partition partition to set Consistent Hashing nodes
+ * @param nodes nodes from clustering plan
+ */
Review Comment:
Does one partitioner instance works for only one partition during the whole life-cycle?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2560671529
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
}, {
"hash" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480",
"triggerID" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"triggerType" : "PUSH"
}, {
"hash" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2506",
"triggerID" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"triggerType" : "PUSH"
}, {
"hash" : "f224eadf25cb4feeb162f405cbd8a5e8250c791f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2566",
"triggerID" : "f224eadf25cb4feeb162f405cbd8a5e8250c791f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f224eadf25cb4feeb162f405cbd8a5e8250c791f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2566)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2552702654
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
}, {
"hash" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480",
"triggerID" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"triggerType" : "PUSH"
}, {
"hash" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2506",
"triggerID" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f457d79a2628589f00a02bc9cf965d76a0520b43 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480)
* ededfcf1096006e5a8378142dd5a41db5f02472c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2506)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-8678] feat: improve consistent-bucket resizing performance by reducing unnecessary record collecting [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #12451:
URL: https://github.com/apache/hudi/pull/12451#issuecomment-2552701388
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "16d1021628a482733e49d21c742a754638f5aabb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2337",
"triggerID" : "16d1021628a482733e49d21c742a754638f5aabb",
"triggerType" : "PUSH"
}, {
"hash" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480",
"triggerID" : "f457d79a2628589f00a02bc9cf965d76a0520b43",
"triggerType" : "PUSH"
}, {
"hash" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ededfcf1096006e5a8378142dd5a41db5f02472c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f457d79a2628589f00a02bc9cf965d76a0520b43 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=2480)
* ededfcf1096006e5a8378142dd5a41db5f02472c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org