You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Oliver Szabo <os...@hortonworks.com> on 2015/09/16 15:40:57 UTC
Review Request 38433: Config changes w/ blueprint provisioning are not
picked up
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38433/
-----------------------------------------------------------
Review request for Ambari, Robert Nettleton and Sumit Mohanty.
Bugs: AMBARI-13099
https://issues.apache.org/jira/browse/AMBARI-13099
Repository: ambari
Description
-------
Problem:
Installing a cluster with a blueprint that contains host group level configurations are not picked up during install time. Ambari UI complains to restart few services after installation which doesn't make sense.
In the provided BP there are host group level config like:
{
"hdfs-site" :
{ "dfs.datanode.data.dir" : "/hadoopfs/fs1/hdfs/datanode,/hadoopfs/fs2/hdfs/datanode" }
},
{
"yarn-site" :
{ "yarn.nodemanager.local-dirs" : "/hadoopfs/fs1/yarn/nodemanager,/hadoopfs/fs2/yarn/nodemanager", "yarn.nodemanager.log-dirs" : "/hadoopfs/fs1/yarn/nodemanager/log,/hadoopfs/fs2/yarn/nodemanager/log" }
}
and those directories are not in use, but the default ones. Compared to Ambari 2.1.0 there is nothing wrong with the bp.
Summary:
Restart can solve this issue, because *-site.xml are not updated, but correct configuration in the blueprint.
Cause:
In case of "forceRefreshConfigTagsBeforeExecution" parameter exists for createHostAction, the additional config parameters are skipped.
Solution:
Remove the code which add "forceRefreshConfigTagsBeforeExecution" parameter to the execution command.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java 20a6932
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java a7f206a
Diff: https://reviews.apache.org/r/38433/diff/
Testing
-------
Unit tests passed. (not contains any updated test)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:13 h
[INFO] Finished at: 2015-09-16T02:31:05+00:00
[INFO] Final Memory: 47M/520M
[INFO] ------------------------------------------------------------------------
Thanks,
Oliver Szabo
Re: Review Request 38433: Config changes w/ blueprint provisioning
are not picked up
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38433/#review99462
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Sept. 16, 2015, 3:40 p.m., Oliver Szabo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38433/
> -----------------------------------------------------------
>
> (Updated Sept. 16, 2015, 3:40 p.m.)
>
>
> Review request for Ambari, Robert Nettleton and Sumit Mohanty.
>
>
> Bugs: AMBARI-13099
> https://issues.apache.org/jira/browse/AMBARI-13099
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> Installing a cluster with a blueprint that contains host group level configurations are not picked up during install time. Ambari UI complains to restart few services after installation which doesn't make sense.
> In the provided BP there are host group level config like:
> {
> "hdfs-site" :
> { "dfs.datanode.data.dir" : "/hadoopfs/fs1/hdfs/datanode,/hadoopfs/fs2/hdfs/datanode" }
> },
> {
> "yarn-site" :
> { "yarn.nodemanager.local-dirs" : "/hadoopfs/fs1/yarn/nodemanager,/hadoopfs/fs2/yarn/nodemanager", "yarn.nodemanager.log-dirs" : "/hadoopfs/fs1/yarn/nodemanager/log,/hadoopfs/fs2/yarn/nodemanager/log" }
> }
> and those directories are not in use, but the default ones. Compared to Ambari 2.1.0 there is nothing wrong with the bp.
>
> Summary:
>
> Restart can solve this issue, because *-site.xml are not updated, but correct configuration in the blueprint.
>
> Cause:
>
> In case of "forceRefreshConfigTagsBeforeExecution" parameter exists for createHostAction, the additional config parameters are skipped.
>
> Solution:
>
> Remove the code which add "forceRefreshConfigTagsBeforeExecution" parameter to the execution command.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java 20a6932
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java a7f206a
>
> Diff: https://reviews.apache.org/r/38433/diff/
>
>
> Testing
> -------
>
> Unit tests passed. (not contains any updated test)
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:13 h
> [INFO] Finished at: 2015-09-16T02:31:05+00:00
> [INFO] Final Memory: 47M/520M
> [INFO] ------------------------------------------------------------------------
>
> functional testing:
>
> I used this gist for blueprint: https://gist.githubusercontent.com/oleewere/3dd7b4b605e8569fe0c4/raw/multi-node-hdfs-yarn
>
> Then I deploy this with 2.1.0 codebase. Service restart was not needed.
> I also try this on 2.1.1 codebase. Service restart was needed there. After recompile the project (with my change) I have overrided the ambari-server.jar in /usr/lib/ambari-server/. After the blueprint install I did not have to restart hdfs/yarn services.
>
> Reproduce test:
> - on boot2docker & docker environment
> 1.) for 2.1.0 testing: https://github.com/sequenceiq/docker-ambari/tree/2.1.0 (baseurl for repo: http://s3.amazonaws.com/dev.hortonworks.com/ambari/centos6/2.x/BUILDS/2.1.0-1470/)
> - source ambari-functions (. ambari-functions)
> - then type: amb-start-cluster
> - then start ambari shell: amb-shell
> - in the shell the following commands starts the deployment :
> - blueprint add --url https://gist.githubusercontent.com/oleewere/3dd7b4b605e8569fe0c4/raw/multi-node-hdfs-yarn
> - cluster build --blueprint multi-node-hdfs-yarn
> - cluster autoAssign
> - cluster create
> 2.) for 2.1.1 testing: https://github.com/sequenceiq/docker-ambari/tree/2.1.1 (baseurl for repo: http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.1.1)
> - same steps as above
> 3.) 2.1.1 with patch: same as 2.), only difference is add a volume parameter here:
> https://github.com/sequenceiq/docker-ambari/blob/2.1.1/ambari-functions#L128 ("-v" param, path to your ambari-server.jar),
> then type 'docker exec -it amb0 bash' to enter the container, there: ambari-server stop, replace your ambari-server.jar then restart ambari-server (in case of exitting the container after stop, type 'docker exec -it amb0 bash' again)
>
>
> Thanks,
>
> Oliver Szabo
>
>
Re: Review Request 38433: Config changes w/ blueprint provisioning
are not picked up
Posted by Oliver Szabo <os...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38433/
-----------------------------------------------------------
(Updated Sept. 16, 2015, 3:40 p.m.)
Review request for Ambari, Robert Nettleton and Sumit Mohanty.
Changes
-------
describe functional testing
Bugs: AMBARI-13099
https://issues.apache.org/jira/browse/AMBARI-13099
Repository: ambari
Description
-------
Problem:
Installing a cluster with a blueprint that contains host group level configurations are not picked up during install time. Ambari UI complains to restart few services after installation which doesn't make sense.
In the provided BP there are host group level config like:
{
"hdfs-site" :
{ "dfs.datanode.data.dir" : "/hadoopfs/fs1/hdfs/datanode,/hadoopfs/fs2/hdfs/datanode" }
},
{
"yarn-site" :
{ "yarn.nodemanager.local-dirs" : "/hadoopfs/fs1/yarn/nodemanager,/hadoopfs/fs2/yarn/nodemanager", "yarn.nodemanager.log-dirs" : "/hadoopfs/fs1/yarn/nodemanager/log,/hadoopfs/fs2/yarn/nodemanager/log" }
}
and those directories are not in use, but the default ones. Compared to Ambari 2.1.0 there is nothing wrong with the bp.
Summary:
Restart can solve this issue, because *-site.xml are not updated, but correct configuration in the blueprint.
Cause:
In case of "forceRefreshConfigTagsBeforeExecution" parameter exists for createHostAction, the additional config parameters are skipped.
Solution:
Remove the code which add "forceRefreshConfigTagsBeforeExecution" parameter to the execution command.
Diffs
-----
ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java 20a6932
ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java a7f206a
Diff: https://reviews.apache.org/r/38433/diff/
Testing (updated)
-------
Unit tests passed. (not contains any updated test)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:13 h
[INFO] Finished at: 2015-09-16T02:31:05+00:00
[INFO] Final Memory: 47M/520M
[INFO] ------------------------------------------------------------------------
functional testing:
I used this gist for blueprint: https://gist.githubusercontent.com/oleewere/3dd7b4b605e8569fe0c4/raw/multi-node-hdfs-yarn
Then I deploy this with 2.1.0 codebase. Service restart was not needed.
I also try this on 2.1.1 codebase. Service restart was needed there. After recompile the project (with my change) I have overrided the ambari-server.jar in /usr/lib/ambari-server/. After the blueprint install I did not have to restart hdfs/yarn services.
Reproduce test:
- on boot2docker & docker environment
1.) for 2.1.0 testing: https://github.com/sequenceiq/docker-ambari/tree/2.1.0 (baseurl for repo: http://s3.amazonaws.com/dev.hortonworks.com/ambari/centos6/2.x/BUILDS/2.1.0-1470/)
- source ambari-functions (. ambari-functions)
- then type: amb-start-cluster
- then start ambari shell: amb-shell
- in the shell the following commands starts the deployment :
- blueprint add --url https://gist.githubusercontent.com/oleewere/3dd7b4b605e8569fe0c4/raw/multi-node-hdfs-yarn
- cluster build --blueprint multi-node-hdfs-yarn
- cluster autoAssign
- cluster create
2.) for 2.1.1 testing: https://github.com/sequenceiq/docker-ambari/tree/2.1.1 (baseurl for repo: http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.1.1)
- same steps as above
3.) 2.1.1 with patch: same as 2.), only difference is add a volume parameter here:
https://github.com/sequenceiq/docker-ambari/blob/2.1.1/ambari-functions#L128 ("-v" param, path to your ambari-server.jar),
then type 'docker exec -it amb0 bash' to enter the container, there: ambari-server stop, replace your ambari-server.jar then restart ambari-server (in case of exitting the container after stop, type 'docker exec -it amb0 bash' again)
Thanks,
Oliver Szabo
Re: Review Request 38433: Config changes w/ blueprint provisioning
are not picked up
Posted by Robert Nettleton <rn...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38433/#review99226
-----------------------------------------------------------
Ship it!
Patch looks fine to me.
Could you please update the review with a summary of the functional testing done to verify this change? I'd request that at a minimum a multi-node cluster needs to be deployed successfully with this patch applied.
Thanks.
- Robert Nettleton
On Sept. 16, 2015, 1:40 p.m., Oliver Szabo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38433/
> -----------------------------------------------------------
>
> (Updated Sept. 16, 2015, 1:40 p.m.)
>
>
> Review request for Ambari, Robert Nettleton and Sumit Mohanty.
>
>
> Bugs: AMBARI-13099
> https://issues.apache.org/jira/browse/AMBARI-13099
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Problem:
>
> Installing a cluster with a blueprint that contains host group level configurations are not picked up during install time. Ambari UI complains to restart few services after installation which doesn't make sense.
> In the provided BP there are host group level config like:
> {
> "hdfs-site" :
> { "dfs.datanode.data.dir" : "/hadoopfs/fs1/hdfs/datanode,/hadoopfs/fs2/hdfs/datanode" }
> },
> {
> "yarn-site" :
> { "yarn.nodemanager.local-dirs" : "/hadoopfs/fs1/yarn/nodemanager,/hadoopfs/fs2/yarn/nodemanager", "yarn.nodemanager.log-dirs" : "/hadoopfs/fs1/yarn/nodemanager/log,/hadoopfs/fs2/yarn/nodemanager/log" }
> }
> and those directories are not in use, but the default ones. Compared to Ambari 2.1.0 there is nothing wrong with the bp.
>
> Summary:
>
> Restart can solve this issue, because *-site.xml are not updated, but correct configuration in the blueprint.
>
> Cause:
>
> In case of "forceRefreshConfigTagsBeforeExecution" parameter exists for createHostAction, the additional config parameters are skipped.
>
> Solution:
>
> Remove the code which add "forceRefreshConfigTagsBeforeExecution" parameter to the execution command.
>
>
> Diffs
> -----
>
> ambari-server/src/main/java/org/apache/ambari/server/actionmanager/ExecutionCommandWrapper.java 20a6932
> ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java a7f206a
>
> Diff: https://reviews.apache.org/r/38433/diff/
>
>
> Testing
> -------
>
> Unit tests passed. (not contains any updated test)
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 01:13 h
> [INFO] Finished at: 2015-09-16T02:31:05+00:00
> [INFO] Final Memory: 47M/520M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Oliver Szabo
>
>