You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Andrew Onischuk <ao...@hortonworks.com> on 2015/12/14 15:14:52 UTC
Review Request 41348: Create a script to allow config merge during
Ambari takeover
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41348/
-----------------------------------------------------------
Review request for Ambari and Sid Wagle.
Bugs: AMBARI-14366
https://issues.apache.org/jira/browse/AMBARI-14366
Repository: ambari
Description
-------
_Background_:
* Paypal 900 node cluster Ambari takeover needs to address the problem of host based configs merged to create default Ambari configs.
* The scope of this task is to provide a script that reads a dir structure for all configs. **For now only site.xml** since env files cannot be parsed, we might need to handle yaml files in later version.
* It produces a output of all the problems / conflicts found along with a configuration.json which can be directly incorporated in a cluster **blueprint**
_Tasks_:
* Please find the attached patch with a partial script that you can start from (Really basic foundation patch).
* Assume that user has copied all the configs in a directory, example:
/tmp/my-master-configs/etc/hadoop/conf,
/tmp/my-master-configs/etc/hbase/conf,
/etc/my-slave-configs/hadoop/conf and so on. (There could be 5 core-site from
different master / slave / client)
* Parse these files and merge them into a map of configs that can be exported to json based on following rules:
* If propertyA is found in core-site1 and core-site2 with same value, add it to the map
* If propertyA is found in core-site1 only, add it to the map
* If propertyA is found in core-site1 and core-site2 with different value, add it to an output file with same name as the config-type, as a conflict.
Sample file (Choose appropriate formatting of choice) ::
core-site-conflicts.txt ::
|| Conflicts ||
| propertyA | <path-to-config-file> | value |
| propertyA | <path-to-config-file> | value |
* **Note**: Single run should generate all the conflicts, if no conflicts found generate the json file
* Provide ability to force output and also a verbose mode to log every step
Diffs
-----
ambari-server/src/main/resources/scripts/takeover_config_merge.py PRE-CREATION
Diff: https://reviews.apache.org/r/41348/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 41348: Create a script to allow config merge during
Ambari takeover
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41348/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 6:31 p.m.)
Review request for Ambari and Sid Wagle.
Bugs: AMBARI-14366
https://issues.apache.org/jira/browse/AMBARI-14366
Repository: ambari
Description
-------
_Background_:
* a lot of node cluster Ambari takeover needs to address the problem of host based configs merged to create default Ambari configs.
* The scope of this task is to provide a script that reads a dir structure for all configs. **For now only site.xml** since env files cannot be parsed, we might need to handle yaml files in later version.
* It produces a output of all the problems / conflicts found along with a configuration.json which can be directly incorporated in a cluster **blueprint**
_Tasks_:
* Please find the attached patch with a partial script that you can start from (Really basic foundation patch).
* Assume that user has copied all the configs in a directory, example:
/tmp/my-master-configs/etc/hadoop/conf,
/tmp/my-master-configs/etc/hbase/conf,
/etc/my-slave-configs/hadoop/conf and so on. (There could be 5 core-site from
different master / slave / client)
* Parse these files and merge them into a map of configs that can be exported to json based on following rules:
* If propertyA is found in core-site1 and core-site2 with same value, add it to the map
* If propertyA is found in core-site1 only, add it to the map
* If propertyA is found in core-site1 and core-site2 with different value, add it to an output file with same name as the config-type, as a conflict.
Sample file (Choose appropriate formatting of choice) ::
core-site-conflicts.txt ::
|| Conflicts ||
| propertyA | <path-to-config-file> | value |
| propertyA | <path-to-config-file> | value |
* **Note**: Single run should generate all the conflicts, if no conflicts found generate the json file
* Provide ability to force output and also a verbose mode to log every step
Diffs (updated)
-----
ambari-server/src/main/resources/scripts/takeover_config_merge.py PRE-CREATION
Diff: https://reviews.apache.org/r/41348/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 41348: Create a script to allow config merge during
Ambari takeover
Posted by Sid Wagle <sw...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41348/#review110256
-----------------------------------------------------------
Ship it!
ambari-server/src/main/resources/scripts/takeover_config_merge.py (line 167)
<https://reviews.apache.org/r/41348/#comment170066>
type in blueprint.
- Sid Wagle
On Dec. 14, 2015, 2:47 p.m., Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41348/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 2:47 p.m.)
>
>
> Review request for Ambari and Sid Wagle.
>
>
> Bugs: AMBARI-14366
> https://issues.apache.org/jira/browse/AMBARI-14366
>
>
> Repository: ambari
>
>
> Description
> -------
>
> _Background_:
>
> * a lot of node cluster Ambari takeover needs to address the problem of host based configs merged to create default Ambari configs.
> * The scope of this task is to provide a script that reads a dir structure for all configs. **For now only site.xml** since env files cannot be parsed, we might need to handle yaml files in later version.
> * It produces a output of all the problems / conflicts found along with a configuration.json which can be directly incorporated in a cluster **blueprint**
>
> _Tasks_:
>
> * Please find the attached patch with a partial script that you can start from (Really basic foundation patch).
> * Assume that user has copied all the configs in a directory, example:
> /tmp/my-master-configs/etc/hadoop/conf,
> /tmp/my-master-configs/etc/hbase/conf,
> /etc/my-slave-configs/hadoop/conf and so on. (There could be 5 core-site from
> different master / slave / client)
>
> * Parse these files and merge them into a map of configs that can be exported to json based on following rules:
> * If propertyA is found in core-site1 and core-site2 with same value, add it to the map
> * If propertyA is found in core-site1 only, add it to the map
> * If propertyA is found in core-site1 and core-site2 with different value, add it to an output file with same name as the config-type, as a conflict.
> Sample file (Choose appropriate formatting of choice) ::
>
>
> core-site-conflicts.txt ::
> || Conflicts ||
> | propertyA | <path-to-config-file> | value |
> | propertyA | <path-to-config-file> | value |
>
>
> * **Note**: Single run should generate all the conflicts, if no conflicts found generate the json file
> * Provide ability to force output and also a verbose mode to log every step
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/scripts/takeover_config_merge.py PRE-CREATION
>
> Diff: https://reviews.apache.org/r/41348/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Andrew Onischuk
>
>
Re: Review Request 41348: Create a script to allow config merge during
Ambari takeover
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41348/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 2:47 p.m.)
Review request for Ambari and Sid Wagle.
Bugs: AMBARI-14366
https://issues.apache.org/jira/browse/AMBARI-14366
Repository: ambari
Description
-------
_Background_:
* a lot of node cluster Ambari takeover needs to address the problem of host based configs merged to create default Ambari configs.
* The scope of this task is to provide a script that reads a dir structure for all configs. **For now only site.xml** since env files cannot be parsed, we might need to handle yaml files in later version.
* It produces a output of all the problems / conflicts found along with a configuration.json which can be directly incorporated in a cluster **blueprint**
_Tasks_:
* Please find the attached patch with a partial script that you can start from (Really basic foundation patch).
* Assume that user has copied all the configs in a directory, example:
/tmp/my-master-configs/etc/hadoop/conf,
/tmp/my-master-configs/etc/hbase/conf,
/etc/my-slave-configs/hadoop/conf and so on. (There could be 5 core-site from
different master / slave / client)
* Parse these files and merge them into a map of configs that can be exported to json based on following rules:
* If propertyA is found in core-site1 and core-site2 with same value, add it to the map
* If propertyA is found in core-site1 only, add it to the map
* If propertyA is found in core-site1 and core-site2 with different value, add it to an output file with same name as the config-type, as a conflict.
Sample file (Choose appropriate formatting of choice) ::
core-site-conflicts.txt ::
|| Conflicts ||
| propertyA | <path-to-config-file> | value |
| propertyA | <path-to-config-file> | value |
* **Note**: Single run should generate all the conflicts, if no conflicts found generate the json file
* Provide ability to force output and also a verbose mode to log every step
Diffs (updated)
-----
ambari-server/src/main/resources/scripts/takeover_config_merge.py PRE-CREATION
Diff: https://reviews.apache.org/r/41348/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 41348: Create a script to allow config merge during
Ambari takeover
Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41348/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 2:27 p.m.)
Review request for Ambari and Sid Wagle.
Bugs: AMBARI-14366
https://issues.apache.org/jira/browse/AMBARI-14366
Repository: ambari
Description (updated)
-------
_Background_:
* a lot of node cluster Ambari takeover needs to address the problem of host based configs merged to create default Ambari configs.
* The scope of this task is to provide a script that reads a dir structure for all configs. **For now only site.xml** since env files cannot be parsed, we might need to handle yaml files in later version.
* It produces a output of all the problems / conflicts found along with a configuration.json which can be directly incorporated in a cluster **blueprint**
_Tasks_:
* Please find the attached patch with a partial script that you can start from (Really basic foundation patch).
* Assume that user has copied all the configs in a directory, example:
/tmp/my-master-configs/etc/hadoop/conf,
/tmp/my-master-configs/etc/hbase/conf,
/etc/my-slave-configs/hadoop/conf and so on. (There could be 5 core-site from
different master / slave / client)
* Parse these files and merge them into a map of configs that can be exported to json based on following rules:
* If propertyA is found in core-site1 and core-site2 with same value, add it to the map
* If propertyA is found in core-site1 only, add it to the map
* If propertyA is found in core-site1 and core-site2 with different value, add it to an output file with same name as the config-type, as a conflict.
Sample file (Choose appropriate formatting of choice) ::
core-site-conflicts.txt ::
|| Conflicts ||
| propertyA | <path-to-config-file> | value |
| propertyA | <path-to-config-file> | value |
* **Note**: Single run should generate all the conflicts, if no conflicts found generate the json file
* Provide ability to force output and also a verbose mode to log every step
Diffs
-----
ambari-server/src/main/resources/scripts/takeover_config_merge.py PRE-CREATION
Diff: https://reviews.apache.org/r/41348/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk