You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Dmytro Sen <ds...@hortonworks.com> on 2014/01/16 14:45:44 UTC
Review Request 16965: Allow log4j properties to be applied via the API in
Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/
-----------------------------------------------------------
Review request for Ambari, Oleksandr Diachenko and Sumit Mohanty.
Bugs: AMBARI-3537
https://issues.apache.org/jira/browse/AMBARI-3537
Repository: ambari
Description
-------
New cluster deployment scenario:
- During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
- UI makes calls like
curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
- python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
- log4j.properties management is made by Ambari UI or API
Cluster upgrade scenario(not implemented yet):
- before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
- run ambari-server upgrade
After applying the patch, ambari server is able to manage log4j properties at the files below:
/etc/hadoop/conf/log4j.properties
/etc/hbase/conf/log4j.properties
/etc/hive/conf/hive-log4j.properties.template
/etc/hive/conf/hive-log4j.properties.template
/etc/oozie/conf/oozie-log4j.properties
/etc/pig/conf/log4j.properties
/etc/zookeeper/conf/log4j.properties
Diffs
-----
ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.4/hooks/before-START/scripts/params.py aabb406
ambari-server/src/main/resources/stacks/HDP/1.3.4/hooks/before-START/scripts/shared_initialization.py 7b406e1
ambari-server/src/main/resources/stacks/HDP/1.3.4/hooks/before-START/templates/log4j.properties.j2 577ad04
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/HBASE/package/scripts/params.py 95880cb
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/OOZIE/package/scripts/oozie.py 91da7ae
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/OOZIE/package/scripts/params.py 0466ad8
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/1.3.4/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
ambari-server/src/main/resources/stacks/HDP/2.0.8/hooks/before-START/scripts/params.py 712a5ab
ambari-server/src/main/resources/stacks/HDP/2.0.8/hooks/before-START/scripts/shared_initialization.py f2644aa
ambari-server/src/main/resources/stacks/HDP/2.0.8/hooks/before-START/templates/log4j.properties.j2 6c02292
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/HBASE/package/scripts/params.py 674b2d9
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/OOZIE/package/scripts/oozie.py 26a6d16
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/OOZIE/package/scripts/params.py 06f2e0e
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/2.0.8/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
Diff: https://reviews.apache.org/r/16965/diff/
Testing
-------
Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
Thanks,
Dmytro Sen
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Oleksandr Diachenko <od...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32491
-----------------------------------------------------------
Ship it!
Ship It!
- Oleksandr Diachenko
On Jan. 22, 2014, 2:12 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 22, 2014, 2:12 p.m.)
>
>
> Review request for Ambari, Oleksandr Diachenko and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/
-----------------------------------------------------------
(Updated Jan. 23, 2014, 5:59 p.m.)
Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
Changes
-------
Fixed according to the comments
Bugs: AMBARI-3537
https://issues.apache.org/jira/browse/AMBARI-3537
Repository: ambari
Description
-------
New cluster deployment scenario:
- During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
- UI makes calls like
curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
- python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
- log4j.properties management is made by Ambari UI or API
Cluster upgrade scenario(not implemented yet):
- before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
- run ambari-server upgrade
After applying the patch, ambari server is able to manage log4j properties at the files below:
/etc/hadoop/conf/log4j.properties
/etc/hbase/conf/log4j.properties
/etc/hive/conf/hive-log4j.properties.template
/etc/hive/conf/hive-log4j.properties.template
/etc/oozie/conf/oozie-log4j.properties
/etc/pig/conf/log4j.properties
/etc/zookeeper/conf/log4j.properties
Diffs (updated)
-----
ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 931ebf9
ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py f3c62df
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/metainfo.xml 4c610db
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/metainfo.xml 009acae
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/metainfo.xml 0a0f8fa
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/metainfo.xml 71783d7
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/metainfo.xml 487104d
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/metainfo.xml 9fb2c06
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/metainfo.xml 22c3eb8
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 63810fa
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/metainfo.xml 15d1045
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/metainfo.xml 8149bc2
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/metainfo.xml da752c9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/metainfo.xml a208c69
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py df4f796
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/metainfo.xml 2e7d493
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/metainfo.xml 2c2169c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/metainfo.xml af5b8cf
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
Diff: https://reviews.apache.org/r/16965/diff/
Testing
-------
Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
Thanks,
Dmytro Sen
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32716
-----------------------------------------------------------
Ship it!
LGTM +1
- Andrew Onischuk
On Jan. 24, 2014, 4:04 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 24, 2014, 4:04 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 931ebf9
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py f3c62df
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/metainfo.xml 4c610db
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/metainfo.xml 009acae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/metainfo.xml 0a0f8fa
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/metainfo.xml 71783d7
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/metainfo.xml 487104d
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/metainfo.xml 9fb2c06
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/metainfo.xml 22c3eb8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 63810fa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/metainfo.xml 15d1045
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/metainfo.xml 8149bc2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/metainfo.xml da752c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/metainfo.xml a208c69
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py df4f796
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/metainfo.xml 2e7d493
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/metainfo.xml 2c2169c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/metainfo.xml af5b8cf
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/
-----------------------------------------------------------
(Updated Jan. 24, 2014, 4:04 p.m.)
Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
Bugs: AMBARI-3537
https://issues.apache.org/jira/browse/AMBARI-3537
Repository: ambari
Description
-------
New cluster deployment scenario:
- During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
- UI makes calls like
curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
- python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
- log4j.properties management is made by Ambari UI or API
Cluster upgrade scenario(not implemented yet):
- before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
- run ambari-server upgrade
After applying the patch, ambari server is able to manage log4j properties at the files below:
/etc/hadoop/conf/log4j.properties
/etc/hbase/conf/log4j.properties
/etc/hive/conf/hive-log4j.properties.template
/etc/hive/conf/hive-log4j.properties.template
/etc/oozie/conf/oozie-log4j.properties
/etc/pig/conf/log4j.properties
/etc/zookeeper/conf/log4j.properties
Diffs (updated)
-----
ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 931ebf9
ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py f3c62df
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/metainfo.xml 4c610db
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/metainfo.xml 009acae
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/metainfo.xml 0a0f8fa
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/metainfo.xml 71783d7
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/metainfo.xml 487104d
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/metainfo.xml 9fb2c06
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/metainfo.xml 22c3eb8
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 63810fa
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/metainfo.xml 15d1045
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/metainfo.xml 8149bc2
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/metainfo.xml da752c9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/metainfo.xml a208c69
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py df4f796
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/metainfo.xml 2e7d493
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/metainfo.xml 2c2169c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/metainfo.xml af5b8cf
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
Diff: https://reviews.apache.org/r/16965/diff/
Testing
-------
Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
Thanks,
Dmytro Sen
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32643
-----------------------------------------------------------
Ship it!
LGTM.
- Sumit Mohanty
On Jan. 23, 2014, 5:59 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 5:59 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 931ebf9
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py f3c62df
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/metainfo.xml 4c610db
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/metainfo.xml 009acae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/metainfo.xml 0a0f8fa
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/metainfo.xml 71783d7
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/metainfo.xml 487104d
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/metainfo.xml 9fb2c06
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/metainfo.xml 22c3eb8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 63810fa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/metainfo.xml 15d1045
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/metainfo.xml 8149bc2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/metainfo.xml da752c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/metainfo.xml a208c69
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py df4f796
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/metainfo.xml 2e7d493
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/metainfo.xml 2c2169c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/metainfo.xml af5b8cf
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Dmytro Sen <ds...@hortonworks.com>.
> On Jan. 24, 2014, 11:40 a.m., Andrew Onischuk wrote:
> > shutil.copyfile(format("{params.hive_conf_dir}/{log4j_exec_filename}"),
> > params.hive_conf_dir)
> > Does that make any sense to copy to the same folder?
> >
> > Also, you can you content to make copying just like this, it's more readable then.
> > File(format("{params.hive_conf_dir}/{log4j_exec_filename}"),
> > mode = 0644,
> > group = params.user_group,
> > owner = params.hive_user,
> > content = StaticFile(dest)
> > )
Thanks for the hint ) Fixed my patch
- Dmytro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32708
-----------------------------------------------------------
On Jan. 24, 2014, 4:04 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 24, 2014, 4:04 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 931ebf9
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py f3c62df
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/metainfo.xml 4c610db
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/metainfo.xml 009acae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/metainfo.xml 0a0f8fa
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/metainfo.xml 71783d7
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/metainfo.xml 487104d
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/metainfo.xml 9fb2c06
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/metainfo.xml 22c3eb8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 63810fa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/metainfo.xml 15d1045
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/metainfo.xml 8149bc2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/metainfo.xml da752c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/metainfo.xml a208c69
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py df4f796
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/metainfo.xml 2e7d493
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/metainfo.xml 2c2169c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/metainfo.xml af5b8cf
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32708
-----------------------------------------------------------
shutil.copyfile(format("{params.hive_conf_dir}/{log4j_exec_filename}"),
params.hive_conf_dir)
Does that make any sense to copy to the same folder?
Also, you can you content to make copying just like this, it's more readable then.
File(format("{params.hive_conf_dir}/{log4j_exec_filename}"),
mode = 0644,
group = params.user_group,
owner = params.hive_user,
content = StaticFile(dest)
)
- Andrew Onischuk
On Jan. 23, 2014, 5:59 p.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 5:59 p.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 931ebf9
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py f3c62df
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/metainfo.xml 4c610db
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/metainfo.xml 009acae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/metainfo.xml 0a0f8fa
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/metainfo.xml 71783d7
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/metainfo.xml 487104d
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/metainfo.xml 9fb2c06
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/metainfo.xml 22c3eb8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 63810fa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/metainfo.xml 15d1045
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/metainfo.xml 8149bc2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/metainfo.xml da752c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/metainfo.xml a208c69
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py df4f796
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/metainfo.xml 2e7d493
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/metainfo.xml 2c2169c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/metainfo.xml af5b8cf
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/
-----------------------------------------------------------
(Updated Jan. 23, 2014, 5:59 p.m.)
Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
Bugs: AMBARI-3537
https://issues.apache.org/jira/browse/AMBARI-3537
Repository: ambari
Description
-------
New cluster deployment scenario:
- During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
- UI makes calls like
curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
- python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
- log4j.properties management is made by Ambari UI or API
Cluster upgrade scenario(not implemented yet):
- before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
- run ambari-server upgrade
After applying the patch, ambari server is able to manage log4j properties at the files below:
/etc/hadoop/conf/log4j.properties
/etc/hbase/conf/log4j.properties
/etc/hive/conf/hive-log4j.properties.template
/etc/hive/conf/hive-log4j.properties.template
/etc/oozie/conf/oozie-log4j.properties
/etc/pig/conf/log4j.properties
/etc/zookeeper/conf/log4j.properties
Diffs
-----
ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py 931ebf9
ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py f3c62df
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/metainfo.xml 4c610db
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/metainfo.xml 009acae
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/metainfo.xml 0a0f8fa
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/metainfo.xml 71783d7
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/metainfo.xml 487104d
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/metainfo.xml 9fb2c06
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/metainfo.xml 22c3eb8
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 63810fa
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/metainfo.xml 15d1045
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/metainfo.xml 8149bc2
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/metainfo.xml da752c9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/metainfo.xml a208c69
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py df4f796
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/metainfo.xml 2e7d493
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/metainfo.xml 2c2169c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/metainfo.xml af5b8cf
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
Diff: https://reviews.apache.org/r/16965/diff/
Testing
-------
Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
Thanks,
Dmytro Sen
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32609
-----------------------------------------------------------
property_map (rca_property_map) should be put into params, not into conreate script. Not to mess up parameter constans with code. Also then this part whould go to params too:
{code}
if params.rca_enabled:
params.log4j_props.update(property_map)
{code}
Also, if we don't have some log4j properties map sent from server, you are just changing the permission for file. Are all that files installed with the packages, or why you do so?
- Andrew Onischuk
On Jan. 23, 2014, 1:28 a.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 1:28 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32590
-----------------------------------------------------------
We should modify all the metainfo.xml to ensure that when a log4j config is modified then the corresponding service components must be marked as RESTART required.
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py
What is the file content when "else" of "if (params.log4j_props != None):" is being executed? Is it an empty file? We need not create an empty file.
If we should not create empty files then this change should be propagated across all the python files.
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py
The comments says "#oozie-log4j.properties" - should be hbase
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py
The log4j files should not be saved as *.templates. The template is only a reference to what the file should contain. They should simply be stored as *.properties.
Minor Nits (make the changes if you wish):
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py
def update_log4j_props()
Rename property "property_map" to "rca_property_map"
- Sumit Mohanty
On Jan. 23, 2014, 1:28 a.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 1:28 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32608
-----------------------------------------------------------
property_map (rca_property_map) should be put into params, not into conreate script. Not to mess up parameter constans with code. Also then this part whould go to params too:
{code}
if params.rca_enabled:
params.log4j_props.update(property_map)
{code}
Also, if we don't have some log4j properties map sent from server, you are just changing the permission for file. Are all that files installed with the packages, or why you do so?
- Andrew Onischuk
On Jan. 23, 2014, 1:28 a.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 1:28 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32607
-----------------------------------------------------------
property_map (rca_property_map) should be put into params, not into conreate script. Not to mess up parameter constans with code. Also then this part whould go to params too:
{code}
if params.rca_enabled:
params.log4j_props.update(property_map)
{code}
Also, if we don't have some log4j properties map sent from server, you are just changing the permission for file. Are all that files installed with the packages, or why you do so?
- Andrew Onischuk
On Jan. 23, 2014, 1:28 a.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 1:28 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Andrew Onischuk <ao...@hortonworks.com>.
> On Jan. 23, 2014, 11:55 a.m., Andrew Onischuk wrote:
> > property_map (rca_property_map) should be put into params, not into conreate script. Not to mess up parameter constans with code. Also then this part whould go to params too:
> > {code}
> > if params.rca_enabled:
> > params.log4j_props.update(property_map)
> > {code}
> >
> > Also, if we don't have some log4j properties map sent from server, you are just changing the permission for file. Are all that files installed with the packages, or why you do so?
Sorry, some bug
- Andrew
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32611
-----------------------------------------------------------
On Jan. 23, 2014, 1:28 a.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 1:28 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32611
-----------------------------------------------------------
property_map (rca_property_map) should be put into params, not into conreate script. Not to mess up parameter constans with code. Also then this part whould go to params too:
{code}
if params.rca_enabled:
params.log4j_props.update(property_map)
{code}
Also, if we don't have some log4j properties map sent from server, you are just changing the permission for file. Are all that files installed with the packages, or why you do so?
- Andrew Onischuk
On Jan. 23, 2014, 1:28 a.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 1:28 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/#review32610
-----------------------------------------------------------
property_map (rca_property_map) should be put into params, not into conreate script. Not to mess up parameter constans with code. Also then this part whould go to params too:
{code}
if params.rca_enabled:
params.log4j_props.update(property_map)
{code}
Also, if we don't have some log4j properties map sent from server, you are just changing the permission for file. Are all that files installed with the packages, or why you do so?
- Andrew Onischuk
On Jan. 23, 2014, 1:28 a.m., Dmytro Sen wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/16965/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2014, 1:28 a.m.)
>
>
> Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
>
>
> Bugs: AMBARI-3537
> https://issues.apache.org/jira/browse/AMBARI-3537
>
>
> Repository: ambari
>
>
> Description
> -------
>
>
> New cluster deployment scenario:
> - During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
> - UI makes calls like
> curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
> - python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
> - log4j.properties management is made by Ambari UI or API
>
> Cluster upgrade scenario(not implemented yet):
> - before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
> - run ambari-server upgrade
>
>
> After applying the patch, ambari server is able to manage log4j properties at the files below:
> /etc/hadoop/conf/log4j.properties
> /etc/hbase/conf/log4j.properties
> /etc/hive/conf/hive-log4j.properties.template
> /etc/hive/conf/hive-log4j.properties.template
> /etc/oozie/conf/oozie-log4j.properties
> /etc/pig/conf/log4j.properties
> /etc/zookeeper/conf/log4j.properties
>
>
> Diffs
> -----
>
> ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
> ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
> ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
> ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
> ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
> ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
> ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
>
> Diff: https://reviews.apache.org/r/16965/diff/
>
>
> Testing
> -------
>
> Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
>
>
> Thanks,
>
> Dmytro Sen
>
>
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/
-----------------------------------------------------------
(Updated Jan. 23, 2014, 1:28 a.m.)
Review request for Ambari, Andrew Onischuk, Oleksandr Diachenko, and Sumit Mohanty.
Bugs: AMBARI-3537
https://issues.apache.org/jira/browse/AMBARI-3537
Repository: ambari
Description
-------
New cluster deployment scenario:
- During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
- UI makes calls like
curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
- python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
- log4j.properties management is made by Ambari UI or API
Cluster upgrade scenario(not implemented yet):
- before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
- run ambari-server upgrade
After applying the patch, ambari server is able to manage log4j properties at the files below:
/etc/hadoop/conf/log4j.properties
/etc/hbase/conf/log4j.properties
/etc/hive/conf/hive-log4j.properties.template
/etc/hive/conf/hive-log4j.properties.template
/etc/oozie/conf/oozie-log4j.properties
/etc/pig/conf/log4j.properties
/etc/zookeeper/conf/log4j.properties
Diffs
-----
ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
Diff: https://reviews.apache.org/r/16965/diff/
Testing
-------
Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
Thanks,
Dmytro Sen
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/
-----------------------------------------------------------
(Updated Jan. 22, 2014, 2:12 p.m.)
Review request for Ambari, Oleksandr Diachenko and Sumit Mohanty.
Changes
-------
removed commented properties
Bugs: AMBARI-3537
https://issues.apache.org/jira/browse/AMBARI-3537
Repository: ambari
Description
-------
New cluster deployment scenario:
- During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
- UI makes calls like
curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
- python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
- log4j.properties management is made by Ambari UI or API
Cluster upgrade scenario(not implemented yet):
- before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
- run ambari-server upgrade
After applying the patch, ambari server is able to manage log4j properties at the files below:
/etc/hadoop/conf/log4j.properties
/etc/hbase/conf/log4j.properties
/etc/hive/conf/hive-log4j.properties.template
/etc/hive/conf/hive-log4j.properties.template
/etc/oozie/conf/oozie-log4j.properties
/etc/pig/conf/log4j.properties
/etc/zookeeper/conf/log4j.properties
Diffs (updated)
-----
ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
Diff: https://reviews.apache.org/r/16965/diff/
Testing
-------
Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
Thanks,
Dmytro Sen
Re: Review Request 16965: Allow log4j properties to be applied via the API
in Ambari for hadoop/oozie/hbase/hive/zookeeper/pig
Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16965/
-----------------------------------------------------------
(Updated Jan. 22, 2014, 10:34 a.m.)
Review request for Ambari, Oleksandr Diachenko and Sumit Mohanty.
Changes
-------
moved changes from HDP2.0.8 to 2.1.1 and from 1.3.4 to 1.3.3
Bugs: AMBARI-3537
https://issues.apache.org/jira/browse/AMBARI-3537
Repository: ambari
Description
-------
New cluster deployment scenario:
- During cluster deployment UI can read default values from "....\resources\stacks\HDP\2.0.8\services\HDFS\configuration\hdfs-log4j.xml", but user is able to modify them before deployment.
- UI makes calls like
curl -i -X PUT -d '{"Clusters":{"desired_configs":{"type":"hdfs-log4j","tag":"version1","properties":{"hadoop.root.logger":"INFO,console"}}}}' http://localhost:8080/api/v1/clusters/c1
- python scripts at the agent side store hdfs-log4j configs as log4j.properties in /etc/hadoop/conf/
- log4j.properties management is made by Ambari UI or API
Cluster upgrade scenario(not implemented yet):
- before running "ambari-server upgrade" command, user modifies *-log4j.xml files at the upgrade stack directory (like stacks\HDP\2.0.8\services\HDFS\configuration)
- run ambari-server upgrade
After applying the patch, ambari server is able to manage log4j properties at the files below:
/etc/hadoop/conf/log4j.properties
/etc/hbase/conf/log4j.properties
/etc/hive/conf/hive-log4j.properties.template
/etc/hive/conf/hive-log4j.properties.template
/etc/oozie/conf/oozie-log4j.properties
/etc/pig/conf/log4j.properties
/etc/zookeeper/conf/log4j.properties
Diffs (updated)
-----
ambari-agent/src/main/python/resource_management/libraries/providers/__init__.py c5e7bae
ambari-agent/src/main/python/resource_management/libraries/providers/properties_file.py PRE-CREATION
ambari-agent/src/main/python/resource_management/libraries/resources/__init__.py bad8470
ambari-agent/src/main/python/resource_management/libraries/resources/properties_file.py PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/params.py aabb406
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/scripts/shared_initialization.py 7b406e1
ambari-server/src/main/resources/stacks/HDP/1.3.3/hooks/before-START/templates/log4j.properties.j2 577ad04
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HBASE/package/scripts/params.py 95880cb
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/MAPREDUCE/configuration/mapreduce-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/oozie.py 91da7ae
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/scripts/params.py 0466ad8
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/OOZIE/package/templates/oozie-log4j.properties.j2 e4a2662
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/1.3.3/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/params.py 712a5ab
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/scripts/shared_initialization.py f2644aa
ambari-server/src/main/resources/stacks/HDP/2.1.1/hooks/before-START/templates/log4j.properties.j2 6c02292
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/configuration/hbase-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/hbase.py bd33463
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HBASE/package/scripts/params.py 674b2d9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HDFS/configuration/hdfs-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-exec-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/configuration/hive-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/hive.py b37ebb2
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/HIVE/package/scripts/params.py 0cf89be
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/configuration/oozie-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/oozie.py 1422d1e
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/OOZIE/package/scripts/params.py 1ad1412
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/configuration/pig-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/params.py 86e962c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/scripts/pig.py c2d7b02
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/PIG/package/templates/log4j.properties.j2 9ef6e2c
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/YARN/configuration/yarn-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/configuration/zookeeper-log4j.xml PRE-CREATION
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/params.py 9acc0c9
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/scripts/zookeeper.py c49eb22
ambari-server/src/main/resources/stacks/HDP/2.1.1/services/ZOOKEEPER/package/templates/log4j.properties.j2 db69564
Diff: https://reviews.apache.org/r/16965/diff/
Testing
-------
Manual testing on HDP1.3.4 and HDP2.0.8 on multinode cluster
Thanks,
Dmytro Sen