You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Sebastian Toader <st...@hortonworks.com> on 2017/07/07 17:00:03 UTC
Review Request 60711: Assigning hosts concurrently to same config
group may fail with
"org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException:
Config group already exist'
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60711/
-----------------------------------------------------------
Review request for Ambari, Attila Magyar, Laszlo Puskas, Robert Levas, and Sandor Magyari.
Bugs: AMBARI-21427
https://issues.apache.org/jira/browse/AMBARI-21427
Repository: ambari
Description
-------
PROBLEM:
Assigning hosts concurrently to the same config group may fail due to a race condition causing the config group to be created twice:
```
05 Jul 2017 18:50:44,742 INFO [pool-5-thread-2] RegisterWithConfigGroupTask:45 - HostRequest: Executing CONFIGURE task for host: w1.internal
05 Jul 2017 18:50:44,884 ERROR [pool-5-thread-2] AmbariContext:655 - Failed to create new configuration group: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
05 Jul 2017 18:50:44,885 ERROR [pool-5-thread-2] AmbariContext:342 - Unable to register config group for host:
java.lang.RuntimeException: Failed to create new configuration group: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
at org.apache.ambari.server.topology.AmbariContext.createConfigGroupsAndRegisterHost(AmbariContext.java:656)
at org.apache.ambari.server.topology.AmbariContext.registerHostWithConfigGroup(AmbariContext.java:339)
at org.apache.ambari.server.topology.tasks.RegisterWithConfigGroupTask.runTask(RegisterWithConfigGroupTask.java:46)
at org.apache.ambari.server.topology.tasks.TopologyHostTask.run(TopologyHostTask.java:51)
at org.apache.ambari.server.topology.HostOfferResponse$1.run(HostOfferResponse.java:83)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:282)
at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.createResources(ConfigGroupResourceProvider.java:276)
at org.apache.ambari.server.topology.AmbariContext.createConfigGroupsAndRegisterHost(AmbariContext.java:653)
... 7 more
```
SOLUTION:
Synchronize the verification of the config group existence and creation of a config group with the same name.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java 0467b9b
Diff: https://reviews.apache.org/r/60711/diff/1/
Testing
-------
Manual testing:
Created a cluster using Blueprint with configurations specified in the host groups to trigger config group creation.
ambari.properties included ```topology.task.creation.parallel=true``` to trigger parallel host assignment to config groups.
Unit tests:
Failed tests:
RestMetricsPropertyProviderTest.testRestMetricsPropertyProviderAsAdministrator:222->testPopulateResourcesMany:457 expected:<400> but was:<121>
Tests run: 4797, Failures: 1, Errors: 0, Skipped: 35
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ........................................ SUCCESS [ 53.583 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.012 s]
[INFO] Ambari Views ....................................... SUCCESS [ 5.241 s]
[INFO] ambari-utility ..................................... SUCCESS [ 7.855 s]
[INFO] ambari-metrics ..................................... SUCCESS [ 1.334 s]
[INFO] Ambari Metrics Common .............................. SUCCESS [ 10.795 s]
[INFO] Ambari Service Advisor ............................. SUCCESS [ 0.886 s]
[INFO] Ambari Server ...................................... FAILURE [ 01:05 h]
[INFO] ------------------------------------------------------------------------
One metrics test failed and is not related to this change.
Thanks,
Sebastian Toader
Re: Review Request 60711: Assigning hosts concurrently to same config
group may fail with
"org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException:
Config group already exist'
Posted by Attila Magyar <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60711/#review179897
-----------------------------------------------------------
Ship it!
Ship It!
- Attila Magyar
On July 7, 2017, 5 p.m., Sebastian Toader wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60711/
> -----------------------------------------------------------
>
> (Updated July 7, 2017, 5 p.m.)
>
>
> Review request for Ambari, Attila Magyar, Laszlo Puskas, Robert Levas, and Sandor Magyari.
>
>
> Bugs: AMBARI-21427
> https://issues.apache.org/jira/browse/AMBARI-21427
>
>
> Repository: ambari
>
>
> Description
> -------
>
> PROBLEM:
> Assigning hosts concurrently to the same config group may fail due to a race condition causing the config group to be created twice:
>
> ```
> 05 Jul 2017 18:50:44,742 INFO [pool-5-thread-2] RegisterWithConfigGroupTask:45 - HostRequest: Executing CONFIGURE task for host: w1.internal
>
> 05 Jul 2017 18:50:44,884 ERROR [pool-5-thread-2] AmbariContext:655 - Failed to create new configuration group: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
> 05 Jul 2017 18:50:44,885 ERROR [pool-5-thread-2] AmbariContext:342 - Unable to register config group for host:
> java.lang.RuntimeException: Failed to create new configuration group: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
> at org.apache.ambari.server.topology.AmbariContext.createConfigGroupsAndRegisterHost(AmbariContext.java:656)
> at org.apache.ambari.server.topology.AmbariContext.registerHostWithConfigGroup(AmbariContext.java:339)
> at org.apache.ambari.server.topology.tasks.RegisterWithConfigGroupTask.runTask(RegisterWithConfigGroupTask.java:46)
> at org.apache.ambari.server.topology.tasks.TopologyHostTask.run(TopologyHostTask.java:51)
> at org.apache.ambari.server.topology.HostOfferResponse$1.run(HostOfferResponse.java:83)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
> at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:282)
> at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.createResources(ConfigGroupResourceProvider.java:276)
> at org.apache.ambari.server.topology.AmbariContext.createConfigGroupsAndRegisterHost(AmbariContext.java:653)
> ... 7 more
> ```
>
> SOLUTION:
> Synchronize the verification of the config group existence and creation of a config group with the same name.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java 0467b9b
>
>
> Diff: https://reviews.apache.org/r/60711/diff/1/
>
>
> Testing
> -------
>
> Manual testing:
> Created a cluster using Blueprint with configurations specified in the host groups to trigger config group creation.
> ambari.properties included ```topology.task.creation.parallel=true``` to trigger parallel host assignment to config groups.
>
>
> Unit tests:
> Failed tests:
> RestMetricsPropertyProviderTest.testRestMetricsPropertyProviderAsAdministrator:222->testPopulateResourcesMany:457 expected:<400> but was:<121>
>
> Tests run: 4797, Failures: 1, Errors: 0, Skipped: 35
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 53.583 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.012 s]
> [INFO] Ambari Views ....................................... SUCCESS [ 5.241 s]
> [INFO] ambari-utility ..................................... SUCCESS [ 7.855 s]
> [INFO] ambari-metrics ..................................... SUCCESS [ 1.334 s]
> [INFO] Ambari Metrics Common .............................. SUCCESS [ 10.795 s]
> [INFO] Ambari Service Advisor ............................. SUCCESS [ 0.886 s]
> [INFO] Ambari Server ...................................... FAILURE [ 01:05 h]
> [INFO] ------------------------------------------------------------------------
>
>
> One metrics test failed and is not related to this change.
>
>
> Thanks,
>
> Sebastian Toader
>
>
Re: Review Request 60711: Assigning hosts concurrently to same config
group may fail with
"org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException:
Config group already exist'
Posted by Sebastian Toader <st...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60711/
-----------------------------------------------------------
(Updated July 10, 2017, 10:44 a.m.)
Review request for Ambari, Attila Magyar, Laszlo Puskas, Robert Levas, and Sandor Magyari.
Changes
-------
Rebased on trunk
Bugs: AMBARI-21427
https://issues.apache.org/jira/browse/AMBARI-21427
Repository: ambari
Description
-------
PROBLEM:
Assigning hosts concurrently to the same config group may fail due to a race condition causing the config group to be created twice:
```
05 Jul 2017 18:50:44,742 INFO [pool-5-thread-2] RegisterWithConfigGroupTask:45 - HostRequest: Executing CONFIGURE task for host: w1.internal
05 Jul 2017 18:50:44,884 ERROR [pool-5-thread-2] AmbariContext:655 - Failed to create new configuration group: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
05 Jul 2017 18:50:44,885 ERROR [pool-5-thread-2] AmbariContext:342 - Unable to register config group for host:
java.lang.RuntimeException: Failed to create new configuration group: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
at org.apache.ambari.server.topology.AmbariContext.createConfigGroupsAndRegisterHost(AmbariContext.java:656)
at org.apache.ambari.server.topology.AmbariContext.registerHostWithConfigGroup(AmbariContext.java:339)
at org.apache.ambari.server.topology.tasks.RegisterWithConfigGroupTask.runTask(RegisterWithConfigGroupTask.java:46)
at org.apache.ambari.server.topology.tasks.TopologyHostTask.run(TopologyHostTask.java:51)
at org.apache.ambari.server.topology.HostOfferResponse$1.run(HostOfferResponse.java:83)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:282)
at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.createResources(ConfigGroupResourceProvider.java:276)
at org.apache.ambari.server.topology.AmbariContext.createConfigGroupsAndRegisterHost(AmbariContext.java:653)
... 7 more
```
SOLUTION:
Synchronize the verification of the config group existence and creation of a config group with the same name.
Diffs (updated)
-----
ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java 9b64edc
Diff: https://reviews.apache.org/r/60711/diff/2/
Changes: https://reviews.apache.org/r/60711/diff/1-2/
Testing
-------
Manual testing:
Created a cluster using Blueprint with configurations specified in the host groups to trigger config group creation.
ambari.properties included ```topology.task.creation.parallel=true``` to trigger parallel host assignment to config groups.
Unit tests:
Failed tests:
RestMetricsPropertyProviderTest.testRestMetricsPropertyProviderAsAdministrator:222->testPopulateResourcesMany:457 expected:<400> but was:<121>
Tests run: 4797, Failures: 1, Errors: 0, Skipped: 35
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ........................................ SUCCESS [ 53.583 s]
[INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.012 s]
[INFO] Ambari Views ....................................... SUCCESS [ 5.241 s]
[INFO] ambari-utility ..................................... SUCCESS [ 7.855 s]
[INFO] ambari-metrics ..................................... SUCCESS [ 1.334 s]
[INFO] Ambari Metrics Common .............................. SUCCESS [ 10.795 s]
[INFO] Ambari Service Advisor ............................. SUCCESS [ 0.886 s]
[INFO] Ambari Server ...................................... FAILURE [ 01:05 h]
[INFO] ------------------------------------------------------------------------
One metrics test failed and is not related to this change.
Thanks,
Sebastian Toader
Re: Review Request 60711: Assigning hosts concurrently to same config
group may fail with
"org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException:
Config group already exist'
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60711/#review179896
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Levas
On July 7, 2017, 1 p.m., Sebastian Toader wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60711/
> -----------------------------------------------------------
>
> (Updated July 7, 2017, 1 p.m.)
>
>
> Review request for Ambari, Attila Magyar, Laszlo Puskas, Robert Levas, and Sandor Magyari.
>
>
> Bugs: AMBARI-21427
> https://issues.apache.org/jira/browse/AMBARI-21427
>
>
> Repository: ambari
>
>
> Description
> -------
>
> PROBLEM:
> Assigning hosts concurrently to the same config group may fail due to a race condition causing the config group to be created twice:
>
> ```
> 05 Jul 2017 18:50:44,742 INFO [pool-5-thread-2] RegisterWithConfigGroupTask:45 - HostRequest: Executing CONFIGURE task for host: w1.internal
>
> 05 Jul 2017 18:50:44,884 ERROR [pool-5-thread-2] AmbariContext:655 - Failed to create new configuration group: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
> 05 Jul 2017 18:50:44,885 ERROR [pool-5-thread-2] AmbariContext:342 - Unable to register config group for host:
> java.lang.RuntimeException: Failed to create new configuration group: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
> at org.apache.ambari.server.topology.AmbariContext.createConfigGroupsAndRegisterHost(AmbariContext.java:656)
> at org.apache.ambari.server.topology.AmbariContext.registerHostWithConfigGroup(AmbariContext.java:339)
> at org.apache.ambari.server.topology.tasks.RegisterWithConfigGroupTask.runTask(RegisterWithConfigGroupTask.java:46)
> at org.apache.ambari.server.topology.tasks.TopologyHostTask.run(TopologyHostTask.java:51)
> at org.apache.ambari.server.topology.HostOfferResponse$1.run(HostOfferResponse.java:83)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.ambari.server.controller.spi.ResourceAlreadyExistsException: Config group already exists with the same name and tag, clusterName = mycluster, groupName = myblueprint:group1, tag = HDFS
> at org.apache.ambari.server.controller.internal.AbstractResourceProvider.createResources(AbstractResourceProvider.java:282)
> at org.apache.ambari.server.controller.internal.ConfigGroupResourceProvider.createResources(ConfigGroupResourceProvider.java:276)
> at org.apache.ambari.server.topology.AmbariContext.createConfigGroupsAndRegisterHost(AmbariContext.java:653)
> ... 7 more
> ```
>
> SOLUTION:
> Synchronize the verification of the config group existence and creation of a config group with the same name.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/topology/AmbariContext.java 0467b9b
>
>
> Diff: https://reviews.apache.org/r/60711/diff/1/
>
>
> Testing
> -------
>
> Manual testing:
> Created a cluster using Blueprint with configurations specified in the host groups to trigger config group creation.
> ambari.properties included ```topology.task.creation.parallel=true``` to trigger parallel host assignment to config groups.
>
>
> Unit tests:
> Failed tests:
> RestMetricsPropertyProviderTest.testRestMetricsPropertyProviderAsAdministrator:222->testPopulateResourcesMany:457 expected:<400> but was:<121>
>
> Tests run: 4797, Failures: 1, Errors: 0, Skipped: 35
>
> [INFO] ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Ambari Main ........................................ SUCCESS [ 53.583 s]
> [INFO] Apache Ambari Project POM .......................... SUCCESS [ 0.012 s]
> [INFO] Ambari Views ....................................... SUCCESS [ 5.241 s]
> [INFO] ambari-utility ..................................... SUCCESS [ 7.855 s]
> [INFO] ambari-metrics ..................................... SUCCESS [ 1.334 s]
> [INFO] Ambari Metrics Common .............................. SUCCESS [ 10.795 s]
> [INFO] Ambari Service Advisor ............................. SUCCESS [ 0.886 s]
> [INFO] Ambari Server ...................................... FAILURE [ 01:05 h]
> [INFO] ------------------------------------------------------------------------
>
>
> One metrics test failed and is not related to this change.
>
>
> Thanks,
>
> Sebastian Toader
>
>