You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Jonathan Hurley <jh...@hortonworks.com> on 2016/03/29 21:32:43 UTC
Review Request 45442: Orphaned Host Alerts Cause Stale Alert
Notifications After Removing Hosts
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45442/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and Nate Cole.
Bugs: AMBARI-15620
https://issues.apache.org/jira/browse/AMBARI-15620
Repository: ambari
Description
-------
Host-level alerts from {{AMBARI}}/{{AMBARI_AGENT}} are orphaned after removing a host because they are always considered valid.
STR
- Deploy cluster
- Add/Remove nodes a few times
- Removed all aded nodes
{code}
There are 4 stale alerts from 4 host(s): amb-roll-workflow1458640758-5.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-2.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-3.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-4.novalocal[Host Disk Usage (3h 52m)]
{code}
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java 8dc8e1e
ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostDAO.java ebd29e3
ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java a1ebaba
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java 6c68d0e
ambari-server/src/test/java/org/apache/ambari/server/state/alerts/AlertReceivedListenerTest.java 136a756
Diff: https://reviews.apache.org/r/45442/diff/
Testing
-------
mvn clean test
Thanks,
Jonathan Hurley
Re: Review Request 45442: Orphaned Host Alerts Cause Stale Alert
Notifications After Removing Hosts
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On March 29, 2016, 3:46 p.m., Nate Cole wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java, lines 404-405
> > <https://reviews.apache.org/r/45442/diff/1/?file=1318274#file1318274line404>
> >
> > Would anyone need any other detail? There's a lot to trigger this: "Unable to process alert for ... due to ..." Also, is it really in-error, or warning?
Good point about error/warning; I think not finding the cluster is probably an error. The rest can be warnings. I'll change it.
As for the message, I'll change it to state a bit more context as to why the cluster/host is not valid.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45442/#review125940
-----------------------------------------------------------
On March 29, 2016, 3:32 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45442/
> -----------------------------------------------------------
>
> (Updated March 29, 2016, 3:32 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-15620
> https://issues.apache.org/jira/browse/AMBARI-15620
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Host-level alerts from {{AMBARI}}/{{AMBARI_AGENT}} are orphaned after removing a host because they are always considered valid.
>
> STR
> - Deploy cluster
> - Add/Remove nodes a few times
> - Removed all aded nodes
>
> {code}
> There are 4 stale alerts from 4 host(s): amb-roll-workflow1458640758-5.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-2.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-3.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-4.novalocal[Host Disk Usage (3h 52m)]
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java 8dc8e1e
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostDAO.java ebd29e3
> ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java a1ebaba
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java 6c68d0e
> ambari-server/src/test/java/org/apache/ambari/server/state/alerts/AlertReceivedListenerTest.java 136a756
>
> Diff: https://reviews.apache.org/r/45442/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 45442: Orphaned Host Alerts Cause Stale Alert
Notifications After Removing Hosts
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45442/#review125940
-----------------------------------------------------------
Ship it!
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java (lines 404 - 405)
<https://reviews.apache.org/r/45442/#comment188868>
Would anyone need any other detail? There's a lot to trigger this: "Unable to process alert for ... due to ..." Also, is it really in-error, or warning?
ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java (lines 392 - 397)
<https://reviews.apache.org/r/45442/#comment188869>
I have no idea how this hostClusterMap relationship came to be :)
- Nate Cole
On March 29, 2016, 3:32 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45442/
> -----------------------------------------------------------
>
> (Updated March 29, 2016, 3:32 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-15620
> https://issues.apache.org/jira/browse/AMBARI-15620
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Host-level alerts from {{AMBARI}}/{{AMBARI_AGENT}} are orphaned after removing a host because they are always considered valid.
>
> STR
> - Deploy cluster
> - Add/Remove nodes a few times
> - Removed all aded nodes
>
> {code}
> There are 4 stale alerts from 4 host(s): amb-roll-workflow1458640758-5.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-2.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-3.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-4.novalocal[Host Disk Usage (3h 52m)]
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java 8dc8e1e
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostDAO.java ebd29e3
> ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java a1ebaba
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java 6c68d0e
> ambari-server/src/test/java/org/apache/ambari/server/state/alerts/AlertReceivedListenerTest.java 136a756
>
> Diff: https://reviews.apache.org/r/45442/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 45442: Orphaned Host Alerts Cause Stale Alert
Notifications After Removing Hosts
Posted by Alejandro Fernandez <af...@hortonworks.com>.
> On March 31, 2016, 1:13 a.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java, line 403
> > <https://reviews.apache.org/r/45442/diff/1/?file=1318274#file1318274line403>
> >
> > This will acquire a read lock every time. If performance suffers, perhaps we can relax that constraint a bit since alerts have a bit more leeway
>
> Jonathan Hurley wrote:
> I'm fine relaxing this a bit. In fact, I think we're negating the benefits of the ConcurrentHashMap here.
+1 on removing the readlock
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45442/#review126231
-----------------------------------------------------------
On March 29, 2016, 7:32 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45442/
> -----------------------------------------------------------
>
> (Updated March 29, 2016, 7:32 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-15620
> https://issues.apache.org/jira/browse/AMBARI-15620
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Host-level alerts from {{AMBARI}}/{{AMBARI_AGENT}} are orphaned after removing a host because they are always considered valid.
>
> STR
> - Deploy cluster
> - Add/Remove nodes a few times
> - Removed all aded nodes
>
> {code}
> There are 4 stale alerts from 4 host(s): amb-roll-workflow1458640758-5.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-2.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-3.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-4.novalocal[Host Disk Usage (3h 52m)]
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java 8dc8e1e
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostDAO.java ebd29e3
> ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java a1ebaba
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java 6c68d0e
> ambari-server/src/test/java/org/apache/ambari/server/state/alerts/AlertReceivedListenerTest.java 136a756
>
> Diff: https://reviews.apache.org/r/45442/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 45442: Orphaned Host Alerts Cause Stale Alert
Notifications After Removing Hosts
Posted by Jonathan Hurley <jh...@hortonworks.com>.
> On March 30, 2016, 9:13 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java, line 403
> > <https://reviews.apache.org/r/45442/diff/1/?file=1318274#file1318274line403>
> >
> > This will acquire a read lock every time. If performance suffers, perhaps we can relax that constraint a bit since alerts have a bit more leeway
I'm fine relaxing this a bit. In fact, I think we're negating the benefits of the ConcurrentHashMap here.
- Jonathan
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45442/#review126231
-----------------------------------------------------------
On March 29, 2016, 3:32 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45442/
> -----------------------------------------------------------
>
> (Updated March 29, 2016, 3:32 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-15620
> https://issues.apache.org/jira/browse/AMBARI-15620
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Host-level alerts from {{AMBARI}}/{{AMBARI_AGENT}} are orphaned after removing a host because they are always considered valid.
>
> STR
> - Deploy cluster
> - Add/Remove nodes a few times
> - Removed all aded nodes
>
> {code}
> There are 4 stale alerts from 4 host(s): amb-roll-workflow1458640758-5.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-2.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-3.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-4.novalocal[Host Disk Usage (3h 52m)]
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java 8dc8e1e
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostDAO.java ebd29e3
> ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java a1ebaba
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java 6c68d0e
> ambari-server/src/test/java/org/apache/ambari/server/state/alerts/AlertReceivedListenerTest.java 136a756
>
> Diff: https://reviews.apache.org/r/45442/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Jonathan Hurley
>
>
Re: Review Request 45442: Orphaned Host Alerts Cause Stale Alert
Notifications After Removing Hosts
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45442/#review126231
-----------------------------------------------------------
ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java (line 403)
<https://reviews.apache.org/r/45442/#comment189148>
This will acquire a read lock every time. If performance suffers, perhaps we can relax that constraint a bit since alerts have a bit more leeway
- Alejandro Fernandez
On March 29, 2016, 7:32 p.m., Jonathan Hurley wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45442/
> -----------------------------------------------------------
>
> (Updated March 29, 2016, 7:32 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Dmitro Lisnichenko, and Nate Cole.
>
>
> Bugs: AMBARI-15620
> https://issues.apache.org/jira/browse/AMBARI-15620
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Host-level alerts from {{AMBARI}}/{{AMBARI_AGENT}} are orphaned after removing a host because they are always considered valid.
>
> STR
> - Deploy cluster
> - Add/Remove nodes a few times
> - Removed all aded nodes
>
> {code}
> There are 4 stale alerts from 4 host(s): amb-roll-workflow1458640758-5.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-2.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-3.novalocal[Host Disk Usage (3h 52m)], amb-roll-workflow1458640758-4.novalocal[Host Disk Usage (3h 52m)]
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/events/listeners/alerts/AlertReceivedListener.java 8dc8e1e
> ambari-server/src/main/java/org/apache/ambari/server/orm/dao/HostDAO.java ebd29e3
> ambari-server/src/main/java/org/apache/ambari/server/state/Clusters.java a1ebaba
> ambari-server/src/main/java/org/apache/ambari/server/state/cluster/ClustersImpl.java 6c68d0e
> ambari-server/src/test/java/org/apache/ambari/server/state/alerts/AlertReceivedListenerTest.java 136a756
>
> Diff: https://reviews.apache.org/r/45442/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Jonathan Hurley
>
>