You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Di Li <lo...@yahoo.com.INVALID> on 2016/02/23 19:13:44 UTC
Ru/EU not updating alert definitions in Ambari server db?
Hello folks,
Do RU and EU handle updating alert definitions stored in the Ambari server database?
Based on what I tested, alert definitions, especially the paths to alert_***.py Python scripts are not updated by neither RU and EU. meaning if I changed the path to the alert_***.py scripts in a new release, the alert definitions are broken and certain components would show with a UNKNOWN alert state after I run RU/EU.
P.S
I know that, post upgrade, I can manually update the alert definitions via REST APIs.
Thanks.
Re: Ru/EU not updating alert definitions in Ambari server db?
Posted by Jonathan Hurley <jh...@hortonworks.com>.
All good questions. I think that the RU/EU aspect of it can't really be a part of the discussion; the orchestration done is really independent of the data stored in Ambari. When using proper inheritance, though, the example below will still work since the path still exists to the original script. If the stacks have inheritance, it will be picked up. You can also define the alerts anywhere within the stack. So if you needed to, you could store them in FLUME/alerts/alert_flume_whatever.py in order to apply to any flume alert.
The upgrade catalogs do touch alert definitions on upgrade. In most cases, we edit the content; in rare cases, we remove an outdated definition. So I think that this would normally be the right place to perform the actions. Aside from that, if you're changing locations of the scripts, I think the only course of action for alerts to also change their definitions. Ambari 2.4 is going to allow editing of script alerts via the UI - so that will certainly help for situations like these.
> On Feb 23, 2016, at 4:17 PM, Di Li <lo...@yahoo.com.INVALID> wrote:
>
> Hello folks,
>
> Yes, I would probably have had seen the error if I had looked at the UI (my bad, I didn't check until post-RU). and yes the issue (since it's about refactored stack structure) will be there the moment I upgrade the Ambari server.
>
> I wanted to know if there is someway to have the alert definitions updated during ambari server upgrade or RU/EU. I am keeping RU/EU in the loop because I may have different versions of the same service in the common service mapped to service in stack. By that, I mean
> common-services/FLUME/1.2.3.4.5 mapped to MyStack/1.0/FLUME
> common-services/FLUME/1.3.4.5.1 mapped to MyStack/2.0/FLUMEwhere two versions of FLUME have different alert_definition Python script.
>
> In this case,I 'd rather have RU/EU take care of updating the alert definition when the stack version is switched (from 1.0 to 2.0).
>
> Understandably this may not be something currently in place in Ambari upgrade nor RU/EU, so I figured I'd better check about this behavior on the mail list.
> Di
>
> From: Jonathan Hurley <jh...@hortonworks.com>
> To: Ambari <de...@ambari.apache.org>; Di Li <lo...@yahoo.com>
> Sent: Tuesday, February 23, 2016 3:43 PM
> Subject: Re: Ru/EU not updating alert definitions in Ambari server db?
>
> Based on this description, you should have seen the error before RU since you refactored your stacks and moved the script. RU and, for that matter, Ambari upgrade, has no built-in knowledge of your scripts. You can pretty much put them anywhere. So, if you change your directory structure around, you're going to also have to update your definitions to have the right path.
>
> This is why you should probably always put them in common services, and use inheritance between stacks.
>
>> On Feb 23, 2016, at 2:24 PM, Di Li <lo...@yahoo.com.INVALID> wrote:
>>
>> Hello folks,
>>
>> Thanks for the reply. I have my own stack called MyStack 1.0 and MyStack 2.0. I found the issue after I heavily refactored the MyStack directory structure.
>>
>> 1. In Ambari 2.1/MyStack 1.0, I have Flume's alert_flume_agent_status.py defined as MyStack/1.0/services/FLUME/package/alerts/alert_flume_agent_status.py
>> 2. In Ambari 2.2, I changed the structure of MyStack to make use of common services, so
>> a) The Flume directories are moved into common-services
>> b) MyStack 1.0 now contains a Flume dir with just a metainfo.xml file.
>> c) I also added MyStack 2.0 so I can run RU/EU. MyStack 2.0 also only contains a Flume dir with just a metainfo.xml file.
>> d) common-services has Flume agent alert Python script as FLUME/1.2.3.4.5/package/alerts/alert_flume_agent_status.py (note the custom version I used here...). As you can see, I changed the path to the alert_flume_agent_status.py...
>> 3. Perform RU
>>
>> -- Post RU, I saw Flume agent produced an "UNKNOWN" alert with an error message
>> Flume Agent Status
>> Unable to find 'MyStack/1.0/services/FLUME/package/alerts/alert_flume_agent_status.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
>>
>> -- I noticed tht in the Ambari server database, the Flume agent alert definition had the wrong patch to the Python script as shown below. It still has the old MyStack based path.
>> 9 | 2 | flume_agent_status | FLUME | FLUME_HANDLER |
>> ANY | Flume Agent Status | This host-level alert is triggered if any of the expect
>> ed flume agent processes are not available.
>> | 1 | 1 | SCRIPT | {"path":"MyStack/1.0/services/FLUME/package/alerts/alert_flu
>> me_agent_status.py","parameters":[{"name":"run.directory","display_name":"Run Directory","value":"/var/run/flume","de
>> scription":"The directory where flume agent processes will place their PID files.","type":"STRING"}],"type":"SCRIPT"}
>>
>> -- This prompted me to ask the question about whether the alert definition should have been updated by RU/EU (or ambari server upgrade maybe).
>>
>> -- I was able to update Flume agent's alert definition with the correct path via following REST API.
>> curl -u admin:admin -H 'X-Requested-By:ambari' -X PUT -d '{"AlertDefinition" : {"source" : {"path" : "FLUME/1.2.3.4.5/package/alerts/alert_flume_agent_status.py", "type" : "SCRIPT" }}}' http://myhost.mydomain.com:8080/api/v1/clusters/test/alert_definitions/9
>>
>>
>> From: Jonathan Hurley <jh...@hortonworks.com>
>> To: "dev@ambari.apache.org" <de...@ambari.apache.org>; Di Li <lo...@yahoo.com>
>> Sent: Tuesday, February 23, 2016 1:23 PM
>> Subject: Re: Ru/EU not updating alert definitions in Ambari server db?
>>
>> No, they are not updated; however, stack inheritance should allow you to reference the files from the new stack. We've performed many different upgrades both both inter-stack and intra-stack and have never seen any issues with alert definitions.
>>
>>> On Feb 23, 2016, at 1:13 PM, Di Li <lo...@yahoo.com.INVALID> wrote:
>>>
>>> Hello folks,
>>>
>>> Do RU and EU handle updating alert definitions stored in the Ambari server database?
>>>
>>> Based on what I tested, alert definitions, especially the paths to alert_***.py Python scripts are not updated by neither RU and EU. meaning if I changed the path to the alert_***.py scripts in a new release, the alert definitions are broken and certain components would show with a UNKNOWN alert state after I run RU/EU.
>>>
>>>
>>> P.S
>>> I know that, post upgrade, I can manually update the alert definitions via REST APIs.
>>>
>>> Thanks.
>>>
>>>
>>
>>
>
>
Re: Ru/EU not updating alert definitions in Ambari server db?
Posted by Di Li <lo...@yahoo.com.INVALID>.
Hello folks,
Yes, I would probably have had seen the error if I had looked at the UI (my bad, I didn't check until post-RU). and yes the issue (since it's about refactored stack structure) will be there the moment I upgrade the Ambari server.
I wanted to know if there is someway to have the alert definitions updated during ambari server upgrade or RU/EU. I am keeping RU/EU in the loop because I may have different versions of the same service in the common service mapped to service in stack. By that, I mean
common-services/FLUME/1.2.3.4.5 mapped to MyStack/1.0/FLUME
common-services/FLUME/1.3.4.5.1 mapped to MyStack/2.0/FLUMEwhere two versions of FLUME have different alert_definition Python script.
In this case,I 'd rather have RU/EU take care of updating the alert definition when the stack version is switched (from 1.0 to 2.0).
Understandably this may not be something currently in place in Ambari upgrade nor RU/EU, so I figured I'd better check about this behavior on the mail list.
Di
From: Jonathan Hurley <jh...@hortonworks.com>
To: Ambari <de...@ambari.apache.org>; Di Li <lo...@yahoo.com>
Sent: Tuesday, February 23, 2016 3:43 PM
Subject: Re: Ru/EU not updating alert definitions in Ambari server db?
Based on this description, you should have seen the error before RU since you refactored your stacks and moved the script. RU and, for that matter, Ambari upgrade, has no built-in knowledge of your scripts. You can pretty much put them anywhere. So, if you change your directory structure around, you're going to also have to update your definitions to have the right path.
This is why you should probably always put them in common services, and use inheritance between stacks.
> On Feb 23, 2016, at 2:24 PM, Di Li <lo...@yahoo.com.INVALID> wrote:
>
> Hello folks,
>
> Thanks for the reply. I have my own stack called MyStack 1.0 and MyStack 2.0. I found the issue after I heavily refactored the MyStack directory structure.
>
> 1. In Ambari 2.1/MyStack 1.0, I have Flume's alert_flume_agent_status.py defined as MyStack/1.0/services/FLUME/package/alerts/alert_flume_agent_status.py
> 2. In Ambari 2.2, I changed the structure of MyStack to make use of common services, so
> a) The Flume directories are moved into common-services
> b) MyStack 1.0 now contains a Flume dir with just a metainfo.xml file.
> c) I also added MyStack 2.0 so I can run RU/EU. MyStack 2.0 also only contains a Flume dir with just a metainfo.xml file.
> d) common-services has Flume agent alert Python script as FLUME/1.2.3.4.5/package/alerts/alert_flume_agent_status.py (note the custom version I used here...). As you can see, I changed the path to the alert_flume_agent_status.py...
> 3. Perform RU
>
> -- Post RU, I saw Flume agent produced an "UNKNOWN" alert with an error message
> Flume Agent Status
> Unable to find 'MyStack/1.0/services/FLUME/package/alerts/alert_flume_agent_status.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
>
> -- I noticed tht in the Ambari server database, the Flume agent alert definition had the wrong patch to the Python script as shown below. It still has the old MyStack based path.
> 9 | 2 | flume_agent_status | FLUME | FLUME_HANDLER |
> ANY | Flume Agent Status | This host-level alert is triggered if any of the expect
> ed flume agent processes are not available.
> | 1 | 1 | SCRIPT | {"path":"MyStack/1.0/services/FLUME/package/alerts/alert_flu
> me_agent_status.py","parameters":[{"name":"run.directory","display_name":"Run Directory","value":"/var/run/flume","de
> scription":"The directory where flume agent processes will place their PID files.","type":"STRING"}],"type":"SCRIPT"}
>
> -- This prompted me to ask the question about whether the alert definition should have been updated by RU/EU (or ambari server upgrade maybe).
>
> -- I was able to update Flume agent's alert definition with the correct path via following REST API.
> curl -u admin:admin -H 'X-Requested-By:ambari' -X PUT -d '{"AlertDefinition" : {"source" : {"path" : "FLUME/1.2.3.4.5/package/alerts/alert_flume_agent_status.py", "type" : "SCRIPT" }}}' http://myhost.mydomain.com:8080/api/v1/clusters/test/alert_definitions/9
>
>
> From: Jonathan Hurley <jh...@hortonworks.com>
> To: "dev@ambari.apache.org" <de...@ambari.apache.org>; Di Li <lo...@yahoo.com>
> Sent: Tuesday, February 23, 2016 1:23 PM
> Subject: Re: Ru/EU not updating alert definitions in Ambari server db?
>
> No, they are not updated; however, stack inheritance should allow you to reference the files from the new stack. We've performed many different upgrades both both inter-stack and intra-stack and have never seen any issues with alert definitions.
>
>> On Feb 23, 2016, at 1:13 PM, Di Li <lo...@yahoo.com.INVALID> wrote:
>>
>> Hello folks,
>>
>> Do RU and EU handle updating alert definitions stored in the Ambari server database?
>>
>> Based on what I tested, alert definitions, especially the paths to alert_***.py Python scripts are not updated by neither RU and EU. meaning if I changed the path to the alert_***.py scripts in a new release, the alert definitions are broken and certain components would show with a UNKNOWN alert state after I run RU/EU.
>>
>>
>> P.S
>> I know that, post upgrade, I can manually update the alert definitions via REST APIs.
>>
>> Thanks.
>>
>>
>
>
Re: Ru/EU not updating alert definitions in Ambari server db?
Posted by Jonathan Hurley <jh...@hortonworks.com>.
Based on this description, you should have seen the error before RU since you refactored your stacks and moved the script. RU and, for that matter, Ambari upgrade, has no built-in knowledge of your scripts. You can pretty much put them anywhere. So, if you change your directory structure around, you're going to also have to update your definitions to have the right path.
This is why you should probably always put them in common services, and use inheritance between stacks.
> On Feb 23, 2016, at 2:24 PM, Di Li <lo...@yahoo.com.INVALID> wrote:
>
> Hello folks,
>
> Thanks for the reply. I have my own stack called MyStack 1.0 and MyStack 2.0. I found the issue after I heavily refactored the MyStack directory structure.
>
> 1. In Ambari 2.1/MyStack 1.0, I have Flume's alert_flume_agent_status.py defined as MyStack/1.0/services/FLUME/package/alerts/alert_flume_agent_status.py
> 2. In Ambari 2.2, I changed the structure of MyStack to make use of common services, so
> a) The Flume directories are moved into common-services
> b) MyStack 1.0 now contains a Flume dir with just a metainfo.xml file.
> c) I also added MyStack 2.0 so I can run RU/EU. MyStack 2.0 also only contains a Flume dir with just a metainfo.xml file.
> d) common-services has Flume agent alert Python script as FLUME/1.2.3.4.5/package/alerts/alert_flume_agent_status.py (note the custom version I used here...). As you can see, I changed the path to the alert_flume_agent_status.py...
> 3. Perform RU
>
> -- Post RU, I saw Flume agent produced an "UNKNOWN" alert with an error message
> Flume Agent Status
> Unable to find 'MyStack/1.0/services/FLUME/package/alerts/alert_flume_agent_status.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
>
> -- I noticed tht in the Ambari server database, the Flume agent alert definition had the wrong patch to the Python script as shown below. It still has the old MyStack based path.
> 9 | 2 | flume_agent_status | FLUME | FLUME_HANDLER |
> ANY | Flume Agent Status | This host-level alert is triggered if any of the expect
> ed flume agent processes are not available.
> | 1 | 1 | SCRIPT | {"path":"MyStack/1.0/services/FLUME/package/alerts/alert_flu
> me_agent_status.py","parameters":[{"name":"run.directory","display_name":"Run Directory","value":"/var/run/flume","de
> scription":"The directory where flume agent processes will place their PID files.","type":"STRING"}],"type":"SCRIPT"}
>
> -- This prompted me to ask the question about whether the alert definition should have been updated by RU/EU (or ambari server upgrade maybe).
>
> -- I was able to update Flume agent's alert definition with the correct path via following REST API.
> curl -u admin:admin -H 'X-Requested-By:ambari' -X PUT -d '{"AlertDefinition" : {"source" : {"path" : "FLUME/1.2.3.4.5/package/alerts/alert_flume_agent_status.py", "type" : "SCRIPT" }}}' http://myhost.mydomain.com:8080/api/v1/clusters/test/alert_definitions/9
>
>
> From: Jonathan Hurley <jh...@hortonworks.com>
> To: "dev@ambari.apache.org" <de...@ambari.apache.org>; Di Li <lo...@yahoo.com>
> Sent: Tuesday, February 23, 2016 1:23 PM
> Subject: Re: Ru/EU not updating alert definitions in Ambari server db?
>
> No, they are not updated; however, stack inheritance should allow you to reference the files from the new stack. We've performed many different upgrades both both inter-stack and intra-stack and have never seen any issues with alert definitions.
>
>> On Feb 23, 2016, at 1:13 PM, Di Li <lo...@yahoo.com.INVALID> wrote:
>>
>> Hello folks,
>>
>> Do RU and EU handle updating alert definitions stored in the Ambari server database?
>>
>> Based on what I tested, alert definitions, especially the paths to alert_***.py Python scripts are not updated by neither RU and EU. meaning if I changed the path to the alert_***.py scripts in a new release, the alert definitions are broken and certain components would show with a UNKNOWN alert state after I run RU/EU.
>>
>>
>> P.S
>> I know that, post upgrade, I can manually update the alert definitions via REST APIs.
>>
>> Thanks.
>>
>>
>
>
Re: Ru/EU not updating alert definitions in Ambari server db?
Posted by Di Li <lo...@yahoo.com.INVALID>.
Hello folks,
Thanks for the reply. I have my own stack called MyStack 1.0 and MyStack 2.0. I found the issue after I heavily refactored the MyStack directory structure.
1. In Ambari 2.1/MyStack 1.0, I have Flume's alert_flume_agent_status.py defined as MyStack/1.0/services/FLUME/package/alerts/alert_flume_agent_status.py
2. In Ambari 2.2, I changed the structure of MyStack to make use of common services, so
a) The Flume directories are moved into common-services
b) MyStack 1.0 now contains a Flume dir with just a metainfo.xml file.
c) I also added MyStack 2.0 so I can run RU/EU. MyStack 2.0 also only contains a Flume dir with just a metainfo.xml file.
d) common-services has Flume agent alert Python script as FLUME/1.2.3.4.5/package/alerts/alert_flume_agent_status.py (note the custom version I used here...). As you can see, I changed the path to the alert_flume_agent_status.py...
3. Perform RU
-- Post RU, I saw Flume agent produced an "UNKNOWN" alert with an error message
Flume Agent Status
Unable to find 'MyStack/1.0/services/FLUME/package/alerts/alert_flume_agent_status.py' as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
-- I noticed tht in the Ambari server database, the Flume agent alert definition had the wrong patch to the Python script as shown below. It still has the old MyStack based path.
9 | 2 | flume_agent_status | FLUME | FLUME_HANDLER |
ANY | Flume Agent Status | This host-level alert is triggered if any of the expect
ed flume agent processes are not available.
| 1 | 1 | SCRIPT | {"path":"MyStack/1.0/services/FLUME/package/alerts/alert_flu
me_agent_status.py","parameters":[{"name":"run.directory","display_name":"Run Directory","value":"/var/run/flume","de
scription":"The directory where flume agent processes will place their PID files.","type":"STRING"}],"type":"SCRIPT"}
-- This prompted me to ask the question about whether the alert definition should have been updated by RU/EU (or ambari server upgrade maybe).
-- I was able to update Flume agent's alert definition with the correct path via following REST API.
curl -u admin:admin -H 'X-Requested-By:ambari' -X PUT -d '{"AlertDefinition" : {"source" : {"path" : "FLUME/1.2.3.4.5/package/alerts/alert_flume_agent_status.py", "type" : "SCRIPT" }}}' http://myhost.mydomain.com:8080/api/v1/clusters/test/alert_definitions/9
From: Jonathan Hurley <jh...@hortonworks.com>
To: "dev@ambari.apache.org" <de...@ambari.apache.org>; Di Li <lo...@yahoo.com>
Sent: Tuesday, February 23, 2016 1:23 PM
Subject: Re: Ru/EU not updating alert definitions in Ambari server db?
No, they are not updated; however, stack inheritance should allow you to reference the files from the new stack. We've performed many different upgrades both both inter-stack and intra-stack and have never seen any issues with alert definitions.
> On Feb 23, 2016, at 1:13 PM, Di Li <lo...@yahoo.com.INVALID> wrote:
>
> Hello folks,
>
> Do RU and EU handle updating alert definitions stored in the Ambari server database?
>
> Based on what I tested, alert definitions, especially the paths to alert_***.py Python scripts are not updated by neither RU and EU. meaning if I changed the path to the alert_***.py scripts in a new release, the alert definitions are broken and certain components would show with a UNKNOWN alert state after I run RU/EU.
>
>
> P.S
> I know that, post upgrade, I can manually update the alert definitions via REST APIs.
>
> Thanks.
>
>
Re: Ru/EU not updating alert definitions in Ambari server db?
Posted by Jonathan Hurley <jh...@hortonworks.com>.
No, they are not updated; however, stack inheritance should allow you to reference the files from the new stack. We've performed many different upgrades both both inter-stack and intra-stack and have never seen any issues with alert definitions.
> On Feb 23, 2016, at 1:13 PM, Di Li <lo...@yahoo.com.INVALID> wrote:
>
> Hello folks,
>
> Do RU and EU handle updating alert definitions stored in the Ambari server database?
>
> Based on what I tested, alert definitions, especially the paths to alert_***.py Python scripts are not updated by neither RU and EU. meaning if I changed the path to the alert_***.py scripts in a new release, the alert definitions are broken and certain components would show with a UNKNOWN alert state after I run RU/EU.
>
>
> P.S
> I know that, post upgrade, I can manually update the alert definitions via REST APIs.
>
> Thanks.
>
>
Re: Ru/EU not updating alert definitions in Ambari server db?
Posted by Alejandro Fernandez <af...@hortonworks.com>.
Typically, alert definitions are modified during Ambari Upgrade as opposed
to RU/EU since alert definitions are for the most part agnostic to the
individual versions inside the stack.
Can you provide an example of an alert that needs to change during a stack
upgrade?
Thanks,
Alejandro
On 2/23/16, 10:13 AM, "Di Li" <lo...@yahoo.com.INVALID> wrote:
>Hello folks,
>
>Do RU and EU handle updating alert definitions stored in the Ambari
>server database?
>
>Based on what I tested, alert definitions, especially the paths to
>alert_***.py Python scripts are not updated by neither RU and EU. meaning
>if I changed the path to the alert_***.py scripts in a new release, the
>alert definitions are broken and certain components would show with a
>UNKNOWN alert state after I run RU/EU.
>
>
>P.S
> I know that, post upgrade, I can manually update the alert
>definitions via REST APIs.
>
>Thanks.
>
>