You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "zpinto (via GitHub)" <gi...@apache.org> on 2023/09/20 01:09:20 UTC

[GitHub] [helix] zpinto opened a new pull request, #2622: Support persist instance info collected by CloudInstanceInformationProcessor during auto-reg

zpinto opened a new pull request, #2622:
URL: https://github.com/apache/helix/pull/2622

   *TODO: This PR currently includes commits from https://github.com/apache/helix/pull/2609 as it has not been merged yet, rebase off master once merged*
   
   Add support to persist all instance information collected by CloudInstanceInformationProcessor in CloudInstanceInformation object. Add ability for CloudInstanceInformationProcessor to produce full DOMAIN field instead of appending _instanceName unless last character in CloudInstanceInformation.CloudInstanceField.FAULT_DOMAIN is '='.
   
   ### Description
   
   Typically the information collected by the CloudInstanceInformationProcessor is used for constructing the DOMAIN field in InstanceConfig. There are some usecases where a customer wants to override the DOMAIN field from what was originally generated from auto-registration. When this happens, all the collected topology information is lost. In order to prevent losing this data, we will persist it in a new field called INSTANCE_INFO_MAP.
   
   Possible usecase steps:
   1. node comes online for first time using auto-reg and defaultInstanceConfig that has HELIX_ENABLED = false
   2. All topology information is collected with CloudInstanceInformationProcessor and returned as CloudInstanceInformation
   3. All information in CloudInstanceInformation is persisted into INSTANCE_INFO_MAP
   4. DOMAIN is generated from CloudInstanceInformation.CloudInstanceField.FAULT_DOMAIN field
   5. node has joined the cluster in disabled state
   6. customers external hook can now run to override the DOMAIN field with whatever they want and set HELIX_ENABLED = true
   
   ### Tests
   
   - [x] TestInstanceConfig updated to test new INSTANCE_INFO_MAP field
   - [x] TestParticipantAutoJoin updated to verify that INSTANCE_INFO_MAP is populated and DOMAIN is constructed without appending _instanceName in case that CloudInstanceInformationProcessor constructs the full domain(signified when there is not = at the end of CloudInstanceInformation.CloudInstanceField.FAULT_DOMAIN)
   
   ```
   ➜  helix git:(support_persist_instance_info) mvn test -o -Dtest=TestInstanceConfig -pl=helix-core 
   ...
   [INFO] -------------------------------------------------------
   [INFO]  T E S T S
   [INFO] -------------------------------------------------------
   [INFO] Running org.apache.helix.model.TestInstanceConfig
   [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.012 s - in org.apache.helix.model.TestInstanceConfig
   [INFO] 
   [INFO] Results:
   [INFO] 
   [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0
   [INFO] 
   [INFO] 
   [INFO] --- jacoco:0.8.6:report (generate-code-coverage-report) @ helix-core ---
   [INFO] Loading execution data file /Users/zapinto/Documents/git/zpinto/helix/helix-core/target/jacoco.exec
   [INFO] Analyzed bundle 'Apache Helix :: Core' with 947 classes
   [INFO] ------------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time:  9.512 s
   [INFO] Finished at: 2023-09-19T18:05:33-07:00
   [INFO] ------------------------------------------------------------------------
   
   
   ➜  helix git:(support_persist_instance_info) mvn test -o -Dtest=TestInstanceAutoJoin -pl=helix-core
   ...
   [INFO] -------------------------------------------------------
   [INFO]  T E S T S
   [INFO] -------------------------------------------------------
   [INFO] Running org.apache.helix.integration.paticipant.TestInstanceAutoJoin
   [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.07 s - in org.apache.helix.integration.paticipant.TestInstanceAutoJoin
   [INFO] 
   [INFO] Results:
   [INFO] 
   [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0
   [INFO] 
   [INFO] 
   [INFO] --- jacoco:0.8.6:report (generate-code-coverage-report) @ helix-core ---
   [INFO] Loading execution data file /Users/zapinto/Documents/git/zpinto/helix/helix-core/target/jacoco.exec
   [INFO] Analyzed bundle 'Apache Helix :: Core' with 947 classes
   [INFO] ------------------------------------------------------------------------
   [INFO] BUILD SUCCESS
   [INFO] ------------------------------------------------------------------------
   [INFO] Total time:  43.920 s
   [INFO] Finished at: 2023-09-19T18:04:52-07:00
   [INFO] ------------------------------------------------------------------------
   
   ```
   
   ### Changes that Break Backward Compatibility (Optional)
   
   NA
   
   ### Commits
   
   - My commits all reference appropriate Apache Helix GitHub issues in their subject lines. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)":
     1. Subject is separated from body by a blank line
     1. Subject is limited to 50 characters (not including Jira issue reference)
     1. Subject does not end with a period
     1. Subject uses the imperative mood ("add", not "adding")
     1. Body wraps at 72 characters
     1. Body explains "what" and "why", not "how"
   
   ### Code Quality
   
   - My diff has been formatted using helix-style.xml 
   (helix-style-intellij.xml if IntelliJ IDE is used)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] zpinto commented on pull request #2622: Support persist instance info collected by CloudInstanceInformationProcessor during auto-reg

Posted by "zpinto (via GitHub)" <gi...@apache.org>.
zpinto commented on PR #2622:
URL: https://github.com/apache/helix/pull/2622#issuecomment-1728541654

   This PR is ready to be merged.
   
   Final Commit Message:
   Add support to persist all instance information collected by CloudInstanceInformationProcessor in CloudInstanceInformation object. Add ability for CloudInstanceInformationProcessor to produce full DOMAIN field instead of appending _instanceName unless last character in CloudInstanceInformation.CloudInstanceField.FAULT_DOMAIN is '='.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu commented on a diff in pull request #2622: Support persist instance info collected by CloudInstanceInformationProcessor during auto-reg

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on code in PR #2622:
URL: https://github.com/apache/helix/pull/2622#discussion_r1332115279


##########
helix-core/src/main/java/org/apache/helix/manager/zk/ParticipantManager.java:
##########
@@ -208,25 +209,33 @@ private void joinCluster() {
     InstanceConfig instanceConfig;
     if (!ZKUtil.isInstanceSetup(_zkclient, _clusterName, _instanceName, _instanceType)) {
       if (!autoJoin) {
-        throw new HelixException("Initial cluster structure is not set up for instance: "
-            + _instanceName + ", instanceType: " + _instanceType);
+        throw new HelixException(
+            "Initial cluster structure is not set up for instance: " + _instanceName
+                + ", instanceType: " + _instanceType);
       }
+
+      InstanceConfig.Builder instanceConfigBuilder =
+          _helixManagerProperty.getDefaultInstanceConfigBuilder();
       if (!autoRegistration) {
         LOG.info(_instanceName + " is auto-joining cluster: " + _clusterName);
-        instanceConfig =
-            _helixManagerProperty.getDefaultInstanceConfigBuilder().build(_instanceName);
+        instanceConfig = instanceConfigBuilder.build(_instanceName);
       } else {
         LOG.info(_instanceName + " is auto-registering cluster: " + _clusterName);
         CloudInstanceInformation cloudInstanceInformation = getCloudInstanceInformation();
-        String domain = cloudInstanceInformation.get(
-            CloudInstanceInformation.CloudInstanceField.FAULT_DOMAIN.name()) + _instanceName;
-        instanceConfig =
-            _helixManagerProperty.getDefaultInstanceConfigBuilder().build(_instanceName);
-        instanceConfig.setDomain(domain);
+        if (cloudInstanceInformation instanceof CloudInstanceInformationV2) {
+          CloudInstanceInformationV2 cloudInstanceInformationV2 =
+              (CloudInstanceInformationV2) cloudInstanceInformation;
+          cloudInstanceInformationV2.getAll().forEach(instanceConfigBuilder::addInstanceInfo);
+        }
+
+        String cloudInstanceInformationFaultDomain = cloudInstanceInformation.get(
+            CloudInstanceInformation.CloudInstanceField.FAULT_DOMAIN.name());
+        instanceConfig = instanceConfigBuilder.setDomain(
+            cloudInstanceInformationFaultDomain.endsWith("=") ? cloudInstanceInformationFaultDomain

Review Comment:
   It looks like a hard code string match. Could you add some comment/sample etc? thx.



##########
helix-core/src/main/java/org/apache/helix/api/cloud/CloudInstanceInformationV2.java:
##########
@@ -0,0 +1,32 @@
+package org.apache.helix.api.cloud;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import com.google.common.collect.ImmutableMap;
+
+public interface CloudInstanceInformationV2 extends CloudInstanceInformation {

Review Comment:
   Considering adding comments on what/why V2?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org


[GitHub] [helix] xyuanlu merged pull request #2622: Support persist instance info collected by CloudInstanceInformationProcessor during auto-reg

Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu merged PR #2622:
URL: https://github.com/apache/helix/pull/2622


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org