You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "coldestlin (via GitHub)" <gi...@apache.org> on 2024/05/13 13:55:13 UTC

[PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

coldestlin opened a new pull request, #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993

   <!--Thanks very much for contributing to Apache DolphinScheduler, we are happy that you want to help us improve DolphinScheduler! -->
   
   ## Purpose of the pull request
   
   <!--(For example: This pull request adds checkstyle plugin).-->
   
   fix https://github.com/apache/dolphinscheduler/issues/15992
   
   ## Brief change log
   
   <!--*(for example:)*
   - *Add maven-checkstyle-plugin to root pom.xml*
   -->
   
   ## Verify this pull request
   
   <!--*(Please pick either of the following options)*-->
   
   This pull request is code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   <!--*(example:)*
   - *Added dolphinscheduler-dao tests for end-to-end.*
   - *Added CronUtilsTest to verify the change.*
   - *Manually verified the change by testing locally.* -->
   
   (or)
   
   If your pull request contain incompatible change, you should also add it to `docs/docs/en/guide/upgrede/incompatible.md`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on code in PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993#discussion_r1599681595


##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/MasterPriorityQueue.java:
##########
@@ -80,15 +80,23 @@ public void clear() {
 
     private void refreshMasterList() {
         hostIndexMap.clear();
-        Iterator<Server> iterator = queue.iterator();
         int index = 0;
-        while (iterator.hasNext()) {
-            Server server = iterator.next();
+        for (Server server: getOrderedElements()) {
             String addr = NetUtils.getAddr(server.getHost(), server.getPort());
             hostIndexMap.put(addr, index);
             index += 1;
         }
+    }
 
+    /**
+     * get ordered collection of priority queue
+     *
+     * @return ordered collection
+     */
+    private Server[] getOrderedElements() {

Review Comment:
   ```suggestion
       Server[] getServers() {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "coldestlin (via GitHub)" <gi...@apache.org>.
coldestlin commented on code in PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993#discussion_r1599694213


##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/MasterPriorityQueue.java:
##########
@@ -80,15 +80,23 @@ public void clear() {
 
     private void refreshMasterList() {
         hostIndexMap.clear();
-        Iterator<Server> iterator = queue.iterator();
         int index = 0;
-        while (iterator.hasNext()) {
-            Server server = iterator.next();
+        for (Server server: getOrderedElements()) {
             String addr = NetUtils.getAddr(server.getHost(), server.getPort());
             hostIndexMap.put(addr, index);
             index += 1;
         }
+    }
 
+    /**
+     * get ordered collection of priority queue
+     *
+     * @return ordered collection
+     */
+    private Server[] getOrderedElements() {

Review Comment:
   Accepted



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993#issuecomment-2111504022

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15993) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [9 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15993&resolved=false&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15993&metric=new_accepted_issues&view=list)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15993&resolved=false&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15993&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15993&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15993)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993#issuecomment-2110249077

   Please fix the code style by `mvn spotless:apply`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993#issuecomment-2111504432

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15993) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [9 New issues](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=15993&resolved=false&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15993&metric=new_accepted_issues&view=list)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=15993&resolved=false&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [100.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15993&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=15993&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=15993)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "coldestlin (via GitHub)" <gi...@apache.org>.
coldestlin commented on code in PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993#discussion_r1599614118


##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/MasterPriorityQueue.java:
##########
@@ -80,15 +81,14 @@ public void clear() {
 
     private void refreshMasterList() {
         hostIndexMap.clear();
-        Iterator<Server> iterator = queue.iterator();
+        Server[] nQueue = queue.toArray(new Server[0]);
+        Arrays.sort(nQueue, new ServerComparator());
         int index = 0;
-        while (iterator.hasNext()) {
-            Server server = iterator.next();
+        for (Server server: nQueue) {
             String addr = NetUtils.getAddr(server.getHost(), server.getPort());
             hostIndexMap.put(addr, index);
             index += 1;
         }
-
     }

Review Comment:
   modified, and UT added, please have a look



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "coldestlin (via GitHub)" <gi...@apache.org>.
coldestlin commented on code in PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993#discussion_r1599692321


##########
dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/queue/MasterPriorityQueueTest.java:
##########
@@ -0,0 +1,92 @@
+package org.apache.dolphinscheduler.service.queue;
+
+import org.apache.dolphinscheduler.common.model.Server;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Arrays;
+
+import java.util.Date;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Assertions;
+
+public class MasterPriorityQueueTest {
+
+
+    @Test
+    public void getOrderedCollection () throws NoSuchMethodException, InvocationTargetException, IllegalAccessException {
+
+        MasterPriorityQueue queue = new MasterPriorityQueue();
+        Method getOrderedElementsMethod = MasterPriorityQueue.class.getDeclaredMethod("getOrderedElements");
+        getOrderedElementsMethod.setAccessible(true);
+
+        // Test empty queue
+        Server[] emptyElements = (Server[]) getOrderedElementsMethod.invoke(queue);
+        Assertions.assertArrayEquals(emptyElements, new Server[]{});
+
+        // Test queue with fabricated servers
+        queue.putAll(getServerList());
+        Server[] orderElements = (Server[]) getOrderedElementsMethod.invoke(queue);
+        Assertions.assertEquals(extractServerIds(orderElements), Arrays.asList(4, 2, 1, 3));
+
+    }

Review Comment:
   Got it, modified. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on code in PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993#discussion_r1599681595


##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/MasterPriorityQueue.java:
##########
@@ -80,15 +80,23 @@ public void clear() {
 
     private void refreshMasterList() {
         hostIndexMap.clear();
-        Iterator<Server> iterator = queue.iterator();
         int index = 0;
-        while (iterator.hasNext()) {
-            Server server = iterator.next();
+        for (Server server: getOrderedElements()) {
             String addr = NetUtils.getAddr(server.getHost(), server.getPort());
             hostIndexMap.put(addr, index);
             index += 1;
         }
+    }
 
+    /**
+     * get ordered collection of priority queue
+     *
+     * @return ordered collection
+     */
+    private Server[] getOrderedElements() {

Review Comment:
   ```suggestion
       Server[] getOrderedElements() {
   ```



##########
dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/queue/MasterPriorityQueueTest.java:
##########
@@ -0,0 +1,92 @@
+package org.apache.dolphinscheduler.service.queue;
+
+import org.apache.dolphinscheduler.common.model.Server;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.util.Arrays;
+
+import java.util.Date;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.Assertions;
+
+public class MasterPriorityQueueTest {
+
+
+    @Test
+    public void getOrderedCollection () throws NoSuchMethodException, InvocationTargetException, IllegalAccessException {
+
+        MasterPriorityQueue queue = new MasterPriorityQueue();
+        Method getOrderedElementsMethod = MasterPriorityQueue.class.getDeclaredMethod("getOrderedElements");
+        getOrderedElementsMethod.setAccessible(true);
+
+        // Test empty queue
+        Server[] emptyElements = (Server[]) getOrderedElementsMethod.invoke(queue);
+        Assertions.assertArrayEquals(emptyElements, new Server[]{});
+
+        // Test queue with fabricated servers
+        queue.putAll(getServerList());
+        Server[] orderElements = (Server[]) getOrderedElementsMethod.invoke(queue);
+        Assertions.assertEquals(extractServerIds(orderElements), Arrays.asList(4, 2, 1, 3));
+
+    }

Review Comment:
   Don't use reflection to test private methods



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993#issuecomment-2111499319

   ## [Codecov](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15993?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   All modified and coverable lines are covered by tests :white_check_mark:
   > Project coverage is 40.44%. Comparing base [(`b474563`)](https://app.codecov.io/gh/apache/dolphinscheduler/commit/b474563d523a40d17f4ddf544fb43603c41ca4d0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`b2ebfb0`)](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15993?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 6 commits behind head on dev.
   
   > :exclamation: Current head b2ebfb0 differs from pull request most recent head bf1381d. Consider uploading reports for the commit bf1381d to get more accurate results
   
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev   #15993      +/-   ##
   ============================================
   + Coverage     39.91%   40.44%   +0.52%     
   - Complexity     5083     5184     +101     
   ============================================
     Files          1370     1379       +9     
     Lines         45649    46060     +411     
     Branches       4869     4900      +31     
   ============================================
   + Hits          18223    18627     +404     
   + Misses        25527    25506      -21     
   - Partials       1899     1927      +28     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/dolphinscheduler/pull/15993?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun merged PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Fix-15992] maintain slot index by iterate MasterPriorityQueue orderly [dolphinscheduler]

Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on code in PR #15993:
URL: https://github.com/apache/dolphinscheduler/pull/15993#discussion_r1599368431


##########
dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/queue/MasterPriorityQueue.java:
##########
@@ -80,15 +81,14 @@ public void clear() {
 
     private void refreshMasterList() {
         hostIndexMap.clear();
-        Iterator<Server> iterator = queue.iterator();
+        Server[] nQueue = queue.toArray(new Server[0]);
+        Arrays.sort(nQueue, new ServerComparator());
         int index = 0;
-        while (iterator.hasNext()) {
-            Server server = iterator.next();
+        for (Server server: nQueue) {
             String addr = NetUtils.getAddr(server.getHost(), server.getPort());
             hostIndexMap.put(addr, index);
             index += 1;
         }
-
     }

Review Comment:
   It's needed to add a private method to get the Elements from queue, and add UT for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org