You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Alejandro Fernandez <af...@hortonworks.com> on 2016/01/28 21:59:10 UTC
Review Request 42920: RU/EU from HDP 2.3 to 2.4 fails on HDFS Balancer
command if NN HA since config dfs.namenode.rpc-address needs to be deleted
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42920/
-----------------------------------------------------------
Review request for Ambari, Jonathan Hurley, Jayush Luniya, Mahadev Konar, Nate Cole, and Swapan Shridhar.
Bugs: AMBARI-14836
https://issues.apache.org/jira/browse/AMBARI-14836
Repository: ambari
Description
-------
STR:
Ambari 2.2.1
HDP 2.3 with HDFS, YARN, MR, ZK
Add NameNode HA
Install bits for HDP 2.4
Perform either EU/RU to HDP 2.4
Run HDFS Balancer command, which fails because property dfs.namenode.rpc-address was not deleted if HA is present.
```
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 401, in <module>
NameNode().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 337, in rebalancehdfs
logoutput = False,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"' KRB5CCNAME=/tmp/hdfs_rebalance_cc_9893e9bc94dfcbb97aa6012cf3df7ee3 ; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10'' returned 252. 16/01/28 19:09:51 INFO balancer.Balancer: Using a threshold of 10.0
16/01/28 19:09:51 INFO balancer.Balancer: namenodes = [hdfs://winchester, hdfs://localhost:8020]
16/01/28 19:09:51 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
16/01/28 19:09:51 INFO balancer.Balancer: included nodes = []
16/01/28 19:09:51 INFO balancer.Balancer: excluded nodes = []
16/01/28 19:09:51 INFO balancer.Balancer: source nodes = []
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
16/01/28 19:09:53 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
16/01/28 19:09:53 INFO block.BlockTokenSecretManager: Setting block keys
16/01/28 19:09:53 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
16/01/28 19:09:53 INFO block.BlockTokenSecretManager: Setting block keys
16/01/28 19:09:53 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
16/01/28 19:09:53 INFO block.BlockTokenSecretManager: Setting block keys
16/01/28 19:09:53 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
java.io.IOException: Another Balancer is running.. Exiting ...
Jan 28, 2016 7:09:53 PM Balancing took 2.307 seconds
```
Diffs
-----
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml ee83348
ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f248293
ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 0145f3b
Diff: https://reviews.apache.org/r/42920/diff/
Testing
-------
Need to verify during RU/EU
Thanks,
Alejandro Fernandez
Re: Review Request 42920: RU/EU from HDP 2.3 to 2.4 fails on HDFS
Balancer
command if NN HA since config dfs.namenode.rpc-address needs to be deleted
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42920/#review116816
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Jan. 28, 2016, 3:59 p.m., Alejandro Fernandez wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42920/
> -----------------------------------------------------------
>
> (Updated Jan. 28, 2016, 3:59 p.m.)
>
>
> Review request for Ambari, Jonathan Hurley, Jayush Luniya, Mahadev Konar, Nate Cole, and Swapan Shridhar.
>
>
> Bugs: AMBARI-14836
> https://issues.apache.org/jira/browse/AMBARI-14836
>
>
> Repository: ambari
>
>
> Description
> -------
>
> STR:
> Ambari 2.2.1
> HDP 2.3 with HDFS, YARN, MR, ZK
> Add NameNode HA
> Install bits for HDP 2.4
> Perform either EU/RU to HDP 2.4
> Run HDFS Balancer command, which fails because property dfs.namenode.rpc-address was not deleted if HA is present.
>
>
> ```
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 401, in <module>
> NameNode().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 337, in rebalancehdfs
> logoutput = False,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/current/hadoop-client/bin'"'"' KRB5CCNAME=/tmp/hdfs_rebalance_cc_9893e9bc94dfcbb97aa6012cf3df7ee3 ; hdfs --config /usr/hdp/current/hadoop-client/conf balancer -threshold 10'' returned 252. 16/01/28 19:09:51 INFO balancer.Balancer: Using a threshold of 10.0
> 16/01/28 19:09:51 INFO balancer.Balancer: namenodes = [hdfs://winchester, hdfs://localhost:8020]
> 16/01/28 19:09:51 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 10.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
> 16/01/28 19:09:51 INFO balancer.Balancer: included nodes = []
> 16/01/28 19:09:51 INFO balancer.Balancer: excluded nodes = []
> 16/01/28 19:09:51 INFO balancer.Balancer: source nodes = []
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
> 16/01/28 19:09:53 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 16/01/28 19:09:53 INFO block.BlockTokenSecretManager: Setting block keys
> 16/01/28 19:09:53 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
> 16/01/28 19:09:53 INFO block.BlockTokenSecretManager: Setting block keys
> 16/01/28 19:09:53 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
> 16/01/28 19:09:53 INFO block.BlockTokenSecretManager: Setting block keys
> 16/01/28 19:09:53 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
> java.io.IOException: Another Balancer is running.. Exiting ...
> Jan 28, 2016 7:09:53 PM Balancing took 2.307 seconds
> ```
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/nonrolling-upgrade-2.4.xml ee83348
> ambari-server/src/main/resources/stacks/HDP/2.3/upgrades/upgrade-2.4.xml f248293
> ambari-server/src/main/resources/stacks/HDP/2.4/upgrades/config-upgrade.xml 0145f3b
>
> Diff: https://reviews.apache.org/r/42920/diff/
>
>
> Testing
> -------
>
> Need to verify during RU/EU
>
>
> Thanks,
>
> Alejandro Fernandez
>
>