You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Di Li <di...@ca.ibm.com> on 2017/06/01 20:35:31 UTC
Review Request 59727: AMBARI-21161 Able to safe guard Ambari server
against circular dependencies in role command order
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59727/
-----------------------------------------------------------
Review request for Ambari and Tim Thorpe.
Bugs: AMBARI-21161
https://issues.apache.org/jira/browse/AMBARI-21161
Repository: ambari
Description
-------
It is important to safe guard against cycle graphs, when Ambari supports mpacks, custom services and service level role command order. Cycle graphs indicate circular dependencies that can cause Ambari enter an infinite loop while building stages, causing Ambari server to take up 100% CPU and large amount of memory.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 024a5e1
ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java 271ee89
Diff: https://reviews.apache.org/r/59727/diff/1/
Testing
-------
unit testins. patch a trunk cluster with code change and install HDP cluster, intentionally add circular reference to make NN start depends on DN start and DN start depends on NN start. Verify UI shows an error message when try to start HDFS service, and does not really kick off the HDFS start. (without the fix, Ambari server will try to start HDFS and gets itself into an infinite loop and uses up 100% CPU and large amount of RAM)
Thanks,
Di Li
Re: Review Request 59727: AMBARI-21161 Able to safe guard Ambari
server against circular dependencies in role command order
Posted by Tim Thorpe <tt...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59727/#review177169
-----------------------------------------------------------
Ship it!
Ship It!
- Tim Thorpe
On June 6, 2017, 8:23 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59727/
> -----------------------------------------------------------
>
> (Updated June 6, 2017, 8:23 p.m.)
>
>
> Review request for Ambari and Tim Thorpe.
>
>
> Bugs: AMBARI-21161
> https://issues.apache.org/jira/browse/AMBARI-21161
>
>
> Repository: ambari
>
>
> Description
> -------
>
> It is important to safe guard against cycle graphs, when Ambari supports mpacks, custom services and service level role command order. Cycle graphs indicate circular dependencies that can cause Ambari enter an infinite loop while building stages, causing Ambari server to take up 100% CPU and large amount of memory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 024a5e1
> ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java 271ee89
> ambari-server/src/test/resources/stacks/HDP/2.0.6.1/role_command_order.json 6154004
>
>
> Diff: https://reviews.apache.org/r/59727/diff/3/
>
>
> Testing
> -------
>
> unit tests. patch a trunk cluster with code change and install HDP cluster, intentionally add circular reference to make NN start depends on DN start and DN start depends on NN start. Verify UI shows an error message when try to start HDFS service, and does not really kick off the HDFS start. (without the fix, Ambari server will try to start HDFS and gets itself into an infinite loop and uses up 100% CPU and large amount of RAM)
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 59727: AMBARI-21161 Able to safe guard Ambari
server against circular dependencies in role command order
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59727/#review177107
-----------------------------------------------------------
Ship it!
Ship It!
- Alejandro Fernandez
On June 6, 2017, 8:23 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59727/
> -----------------------------------------------------------
>
> (Updated June 6, 2017, 8:23 p.m.)
>
>
> Review request for Ambari and Tim Thorpe.
>
>
> Bugs: AMBARI-21161
> https://issues.apache.org/jira/browse/AMBARI-21161
>
>
> Repository: ambari
>
>
> Description
> -------
>
> It is important to safe guard against cycle graphs, when Ambari supports mpacks, custom services and service level role command order. Cycle graphs indicate circular dependencies that can cause Ambari enter an infinite loop while building stages, causing Ambari server to take up 100% CPU and large amount of memory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 024a5e1
> ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java 271ee89
> ambari-server/src/test/resources/stacks/HDP/2.0.6.1/role_command_order.json 6154004
>
>
> Diff: https://reviews.apache.org/r/59727/diff/3/
>
>
> Testing
> -------
>
> unit tests. patch a trunk cluster with code change and install HDP cluster, intentionally add circular reference to make NN start depends on DN start and DN start depends on NN start. Verify UI shows an error message when try to start HDFS service, and does not really kick off the HDFS start. (without the fix, Ambari server will try to start HDFS and gets itself into an infinite loop and uses up 100% CPU and large amount of RAM)
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 59727: AMBARI-21161 Able to safe guard Ambari
server against circular dependencies in role command order
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59727/
-----------------------------------------------------------
(Updated June 6, 2017, 8:23 p.m.)
Review request for Ambari and Tim Thorpe.
Bugs: AMBARI-21161
https://issues.apache.org/jira/browse/AMBARI-21161
Repository: ambari
Description
-------
It is important to safe guard against cycle graphs, when Ambari supports mpacks, custom services and service level role command order. Cycle graphs indicate circular dependencies that can cause Ambari enter an infinite loop while building stages, causing Ambari server to take up 100% CPU and large amount of memory.
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 024a5e1
ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java 271ee89
ambari-server/src/test/resources/stacks/HDP/2.0.6.1/role_command_order.json 6154004
Diff: https://reviews.apache.org/r/59727/diff/3/
Changes: https://reviews.apache.org/r/59727/diff/2-3/
Testing
-------
unit tests. patch a trunk cluster with code change and install HDP cluster, intentionally add circular reference to make NN start depends on DN start and DN start depends on NN start. Verify UI shows an error message when try to start HDFS service, and does not really kick off the HDFS start. (without the fix, Ambari server will try to start HDFS and gets itself into an infinite loop and uses up 100% CPU and large amount of RAM)
Thanks,
Di Li
Re: Review Request 59727: AMBARI-21161 Able to safe guard Ambari
server against circular dependencies in role command order
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59727/
-----------------------------------------------------------
(Updated June 6, 2017, 7:02 p.m.)
Review request for Ambari and Tim Thorpe.
Bugs: AMBARI-21161
https://issues.apache.org/jira/browse/AMBARI-21161
Repository: ambari
Description
-------
It is important to safe guard against cycle graphs, when Ambari supports mpacks, custom services and service level role command order. Cycle graphs indicate circular dependencies that can cause Ambari enter an infinite loop while building stages, causing Ambari server to take up 100% CPU and large amount of memory.
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 024a5e1
ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java 271ee89
ambari-server/src/test/resources/stacks/HDP/2.0.6.1/role_command_order.json 6154004
Diff: https://reviews.apache.org/r/59727/diff/2/
Changes: https://reviews.apache.org/r/59727/diff/1-2/
Testing
-------
unit tests. patch a trunk cluster with code change and install HDP cluster, intentionally add circular reference to make NN start depends on DN start and DN start depends on NN start. Verify UI shows an error message when try to start HDFS service, and does not really kick off the HDFS start. (without the fix, Ambari server will try to start HDFS and gets itself into an infinite loop and uses up 100% CPU and large amount of RAM)
Thanks,
Di Li
Re: Review Request 59727: AMBARI-21161 Able to safe guard Ambari
server against circular dependencies in role command order
Posted by Di Li <di...@ca.ibm.com>.
> On June 1, 2017, 8:58 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java
> > Lines 342 (patched)
> > <https://reviews.apache.org/r/59727/diff/1/?file=1738985#file1738985line342>
> >
> > Btw, this will only find cycles between two adjacent nodes. It will not find cycles involved with 3 nodes, e.g.,
> >
> > A -> B
> > B -> C
> > C -> A
> >
> > This needs a 2-pass depth-first traversal or Tarjan's algorithm.
My tests showed that Ambari is able to break long chains of dependencies to short chains that represents cycle graphs of two nodes. This behavior makes it easier to detect circular dependencies via current logic.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59727/#review176659
-----------------------------------------------------------
On June 1, 2017, 8:35 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59727/
> -----------------------------------------------------------
>
> (Updated June 1, 2017, 8:35 p.m.)
>
>
> Review request for Ambari and Tim Thorpe.
>
>
> Bugs: AMBARI-21161
> https://issues.apache.org/jira/browse/AMBARI-21161
>
>
> Repository: ambari
>
>
> Description
> -------
>
> It is important to safe guard against cycle graphs, when Ambari supports mpacks, custom services and service level role command order. Cycle graphs indicate circular dependencies that can cause Ambari enter an infinite loop while building stages, causing Ambari server to take up 100% CPU and large amount of memory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 024a5e1
> ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java 271ee89
>
>
> Diff: https://reviews.apache.org/r/59727/diff/1/
>
>
> Testing
> -------
>
> unit tests. patch a trunk cluster with code change and install HDP cluster, intentionally add circular reference to make NN start depends on DN start and DN start depends on NN start. Verify UI shows an error message when try to start HDFS service, and does not really kick off the HDFS start. (without the fix, Ambari server will try to start HDFS and gets itself into an infinite loop and uses up 100% CPU and large amount of RAM)
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 59727: AMBARI-21161 Able to safe guard Ambari
server against circular dependencies in role command order
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On June 1, 2017, 8:58 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java
> > Lines 342 (patched)
> > <https://reviews.apache.org/r/59727/diff/1/?file=1738985#file1738985line342>
> >
> > Btw, this will only find cycles between two adjacent nodes. It will not find cycles involved with 3 nodes, e.g.,
> >
> > A -> B
> > B -> C
> > C -> A
> >
> > This needs a 2-pass depth-first traversal or Tarjan's algorithm.
>
> Di Li wrote:
> My tests showed that Ambari is able to break long chains of dependencies to short chains that represents cycle graphs of two nodes. This behavior makes it easier to detect circular dependencies via current logic.
Let's add a unit test for it, Storm depends on Kafka which depends on ZK. So if we then add ZK depends on Storm, that would create a cycle.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59727/#review176659
-----------------------------------------------------------
On June 6, 2017, 7:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59727/
> -----------------------------------------------------------
>
> (Updated June 6, 2017, 7:02 p.m.)
>
>
> Review request for Ambari and Tim Thorpe.
>
>
> Bugs: AMBARI-21161
> https://issues.apache.org/jira/browse/AMBARI-21161
>
>
> Repository: ambari
>
>
> Description
> -------
>
> It is important to safe guard against cycle graphs, when Ambari supports mpacks, custom services and service level role command order. Cycle graphs indicate circular dependencies that can cause Ambari enter an infinite loop while building stages, causing Ambari server to take up 100% CPU and large amount of memory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 024a5e1
> ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java 271ee89
> ambari-server/src/test/resources/stacks/HDP/2.0.6.1/role_command_order.json 6154004
>
>
> Diff: https://reviews.apache.org/r/59727/diff/2/
>
>
> Testing
> -------
>
> unit tests. patch a trunk cluster with code change and install HDP cluster, intentionally add circular reference to make NN start depends on DN start and DN start depends on NN start. Verify UI shows an error message when try to start HDFS service, and does not really kick off the HDFS start. (without the fix, Ambari server will try to start HDFS and gets itself into an infinite loop and uses up 100% CPU and large amount of RAM)
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 59727: AMBARI-21161 Able to safe guard Ambari
server against circular dependencies in role command order
Posted by Di Li <di...@ca.ibm.com>.
> On June 1, 2017, 8:58 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java
> > Lines 342 (patched)
> > <https://reviews.apache.org/r/59727/diff/1/?file=1738985#file1738985line342>
> >
> > Btw, this will only find cycles between two adjacent nodes. It will not find cycles involved with 3 nodes, e.g.,
> >
> > A -> B
> > B -> C
> > C -> A
> >
> > This needs a 2-pass depth-first traversal or Tarjan's algorithm.
>
> Di Li wrote:
> My tests showed that Ambari is able to break long chains of dependencies to short chains that represents cycle graphs of two nodes. This behavior makes it easier to detect circular dependencies via current logic.
>
> Alejandro Fernandez wrote:
> Let's add a unit test for it, Storm depends on Kafka which depends on ZK. So if we then add ZK depends on Storm, that would create a cycle.
I have two unit test cases for checking depedencies between ZK/YARN/HBase (didn't use Storm/Kafka/ZK, don't think it matters for unit testing) for the STOP action, and ZK/HBase for the UPGRADE action.
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59727/#review176659
-----------------------------------------------------------
On June 6, 2017, 7:02 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59727/
> -----------------------------------------------------------
>
> (Updated June 6, 2017, 7:02 p.m.)
>
>
> Review request for Ambari and Tim Thorpe.
>
>
> Bugs: AMBARI-21161
> https://issues.apache.org/jira/browse/AMBARI-21161
>
>
> Repository: ambari
>
>
> Description
> -------
>
> It is important to safe guard against cycle graphs, when Ambari supports mpacks, custom services and service level role command order. Cycle graphs indicate circular dependencies that can cause Ambari enter an infinite loop while building stages, causing Ambari server to take up 100% CPU and large amount of memory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 024a5e1
> ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java 271ee89
> ambari-server/src/test/resources/stacks/HDP/2.0.6.1/role_command_order.json 6154004
>
>
> Diff: https://reviews.apache.org/r/59727/diff/2/
>
>
> Testing
> -------
>
> unit tests. patch a trunk cluster with code change and install HDP cluster, intentionally add circular reference to make NN start depends on DN start and DN start depends on NN start. Verify UI shows an error message when try to start HDFS service, and does not really kick off the HDFS start. (without the fix, Ambari server will try to start HDFS and gets itself into an infinite loop and uses up 100% CPU and large amount of RAM)
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 59727: AMBARI-21161 Able to safe guard Ambari
server against circular dependencies in role command order
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59727/#review176659
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java
Lines 342 (patched)
<https://reviews.apache.org/r/59727/#comment250075>
Btw, this will only find cycles between two adjacent nodes. It will not find cycles involved with 3 nodes, e.g.,
A -> B
B -> C
C -> A
This needs a 2-pass depth-first traversal or Tarjan's algorithm.
ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java
Lines 356 (patched)
<https://reviews.apache.org/r/59727/#comment250073>
Nitpick here, add spaces before and after "+", or use String.format
- Alejandro Fernandez
On June 1, 2017, 8:35 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59727/
> -----------------------------------------------------------
>
> (Updated June 1, 2017, 8:35 p.m.)
>
>
> Review request for Ambari and Tim Thorpe.
>
>
> Bugs: AMBARI-21161
> https://issues.apache.org/jira/browse/AMBARI-21161
>
>
> Repository: ambari
>
>
> Description
> -------
>
> It is important to safe guard against cycle graphs, when Ambari supports mpacks, custom services and service level role command order. Cycle graphs indicate circular dependencies that can cause Ambari enter an infinite loop while building stages, causing Ambari server to take up 100% CPU and large amount of memory.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 024a5e1
> ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java 271ee89
>
>
> Diff: https://reviews.apache.org/r/59727/diff/1/
>
>
> Testing
> -------
>
> unit tests. patch a trunk cluster with code change and install HDP cluster, intentionally add circular reference to make NN start depends on DN start and DN start depends on NN start. Verify UI shows an error message when try to start HDFS service, and does not really kick off the HDFS start. (without the fix, Ambari server will try to start HDFS and gets itself into an infinite loop and uses up 100% CPU and large amount of RAM)
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 59727: AMBARI-21161 Able to safe guard Ambari
server against circular dependencies in role command order
Posted by Di Li <di...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59727/
-----------------------------------------------------------
(Updated June 1, 2017, 8:35 p.m.)
Review request for Ambari and Tim Thorpe.
Bugs: AMBARI-21161
https://issues.apache.org/jira/browse/AMBARI-21161
Repository: ambari
Description
-------
It is important to safe guard against cycle graphs, when Ambari supports mpacks, custom services and service level role command order. Cycle graphs indicate circular dependencies that can cause Ambari enter an infinite loop while building stages, causing Ambari server to take up 100% CPU and large amount of memory.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/stageplanner/RoleGraph.java 024a5e1
ambari-server/src/test/java/org/apache/ambari/server/stageplanner/TestStagePlanner.java 271ee89
Diff: https://reviews.apache.org/r/59727/diff/1/
Testing (updated)
-------
unit tests. patch a trunk cluster with code change and install HDP cluster, intentionally add circular reference to make NN start depends on DN start and DN start depends on NN start. Verify UI shows an error message when try to start HDFS service, and does not really kick off the HDFS start. (without the fix, Ambari server will try to start HDFS and gets itself into an infinite loop and uses up 100% CPU and large amount of RAM)
Thanks,
Di Li