You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Dmitro Lisnichenko <dl...@hortonworks.com> on 2017/11/30 11:46:28 UTC
Review Request 64205: Snapshot HBase task failed during IOP migration
with TypeError
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64205/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
Bugs: AMBARI-22558
https://issues.apache.org/jira/browse/AMBARI-22558
Repository: ambari
Description
-------
*STR*
# Deployed cluster with Ambari version: 2.2.0 and IOP version: 4.2.0.0
# Upgrade Ambari to Target Version: 2.5.2.0-298 | Hash: 2453e16418fd964042452b649153dbe45f3c6009
# Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
# Register HDP-2.6.4.0-51 and call remove iop-select
# Install the new HDP version bits and start Express Upgrade
*Result*
Snapshot HBase task failed with below error:
{code}
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 157, in <module>
ExecuteUpgradeTasks().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
method(env)
File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 153, in actionexecute
shell.checked_call(task.command, logoutput=True, quiet=True)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'source /var/lib/ambari-agent/ambari-env.sh ; /usr/bin/ambari-python-wrap /var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py take_snapshot /var/lib/ambari-agent/data/command-404.json /var/lib/ambari-agent/cache/custom_actions /var/lib/ambari-agent/data/structured-out-404.json INFO /var/lib/ambari-agent/tmp' returned 1. 2017-11-29 07:01:50,049 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
2017-11-29 07:01:50,096 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 37, in <module>
HbaseMasterUpgrade().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 28, in take_snapshot
import params
File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py", line 107, in <module>
regionserver_xmn_percent = expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
TypeError: 'module' object is not callable
{code}
Upon checking the code [here|https://github.com/hortonworks/ambari/blob/AMBARI-2.6.1.0/ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py#L30], the issue seems to be with import of 'expect' module
I tried the following changes in params.py:
{code}
L30: from resource_management.libraries.functions import expect
L107: regionserver_xmn_percent = expect.expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
{code}
Now the snapshot command ran fine:
{code}
2017-11-29 11:14:15,472 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
2017-11-29 11:14:15,474 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-11-29 11:14:15,476 - checked_call['hostid'] {}
2017-11-29 11:14:15,494 - checked_call returned (0, '16ace057')
2017-11-29 11:14:15,495 - Execute[' echo 'snapshot_all' | /usr/iop/current/hbase-client/bin/hbase shell'] {'user': 'hbase'}
{code}
Diffs
-----
ambari-common/src/main/python/resource_management/libraries/functions/lzo_utils.py 68ee607b9b
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py f7fd5459c1
ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params_linux.py 0ca1f0bd21
ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py 0f6ee01c70
ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py b8e709a6db
ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 46f715ecde
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 5e1f4ac5cb
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 18e297865e
ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/params.py 4eb5b02c3f
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params.py f39d6326de
ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py 8f38843364
ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py eaf1ee4940
ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py d9fae763ad
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py 34c6b9ab96
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HBASE/package/scripts/params.py 97657ad7f4
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-ANY/scripts/params.py 1ed6d4d748
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-INSTALL/scripts/params.py 6193c11c22
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HBASE/package/scripts/params_linux.py bff022d1f7
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HDFS/package/scripts/params_linux.py a77ec82cdc
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/package/scripts/params.py f39d6326de
ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/package/scripts/params_linux.py 5868f5e604
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py c0c314dcdf
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/KAFKA/package/scripts/params.py 07e7ad19d1
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 8ad0d51376
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/params.py da75a8ee89
ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-ANY/scripts/params.py 2c2c90113a
ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/params.py 4eb5b02c3f
Diff: https://reviews.apache.org/r/64205/diff/1/
Testing
-------
mvn clean test
Thanks,
Dmitro Lisnichenko
Re: Review Request 64205: Snapshot HBase task failed during IOP
migration with TypeError
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64205/#review192320
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Nov. 30, 2017, 10:04 a.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64205/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2017, 10:04 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-22558
> https://issues.apache.org/jira/browse/AMBARI-22558
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *STR*
> # Deployed cluster with Ambari version: 2.2.0 and IOP version: 4.2.0.0
> # Upgrade Ambari to Target Version: 2.5.2.0-298 | Hash: 2453e16418fd964042452b649153dbe45f3c6009
> # Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
> # Register HDP-2.6.4.0-51 and call remove iop-select
> # Install the new HDP version bits and start Express Upgrade
>
> *Result*
> Snapshot HBase task failed with below error:
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 157, in <module>
> ExecuteUpgradeTasks().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 153, in actionexecute
> shell.checked_call(task.command, logoutput=True, quiet=True)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
> tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 'source /var/lib/ambari-agent/ambari-env.sh ; /usr/bin/ambari-python-wrap /var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py take_snapshot /var/lib/ambari-agent/data/command-404.json /var/lib/ambari-agent/cache/custom_actions /var/lib/ambari-agent/data/structured-out-404.json INFO /var/lib/ambari-agent/tmp' returned 1. 2017-11-29 07:01:50,049 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
> 2017-11-29 07:01:50,096 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 37, in <module>
> HbaseMasterUpgrade().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 28, in take_snapshot
> import params
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py", line 107, in <module>
> regionserver_xmn_percent = expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
> TypeError: 'module' object is not callable
> {code}
>
>
> Upon checking the code [here|https://github.com/hortonworks/ambari/blob/AMBARI-2.6.1.0/ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py#L30], the issue seems to be with import of 'expect' module
>
> I tried the following changes in params.py:
> {code}
> L30: from resource_management.libraries.functions import expect
> L107: regionserver_xmn_percent = expect.expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
> {code}
>
> Now the snapshot command ran fine:
> {code}
> 2017-11-29 11:14:15,472 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
> 2017-11-29 11:14:15,474 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
> 2017-11-29 11:14:15,476 - checked_call['hostid'] {}
> 2017-11-29 11:14:15,494 - checked_call returned (0, '16ace057')
> 2017-11-29 11:14:15,495 - Execute[' echo 'snapshot_all' | /usr/iop/current/hbase-client/bin/hbase shell'] {'user': 'hbase'}
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py c0c314dcdf
>
>
> Diff: https://reviews.apache.org/r/64205/diff/2/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 64205: Snapshot HBase task failed during IOP
migration with TypeError
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64205/#review192370
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Nov. 30, 2017, 4:20 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64205/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2017, 4:20 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-22558
> https://issues.apache.org/jira/browse/AMBARI-22558
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *STR*
> # Deployed cluster with Ambari version: 2.2.0 and IOP version: 4.2.0.0
> # Upgrade Ambari to Target Version: 2.5.2.0-298 | Hash: 2453e16418fd964042452b649153dbe45f3c6009
> # Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
> # Register HDP-2.6.4.0-51 and call remove iop-select
> # Install the new HDP version bits and start Express Upgrade
>
> *Result*
> Snapshot HBase task failed with below error:
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 157, in <module>
> ExecuteUpgradeTasks().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 153, in actionexecute
> shell.checked_call(task.command, logoutput=True, quiet=True)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
> tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 'source /var/lib/ambari-agent/ambari-env.sh ; /usr/bin/ambari-python-wrap /var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py take_snapshot /var/lib/ambari-agent/data/command-404.json /var/lib/ambari-agent/cache/custom_actions /var/lib/ambari-agent/data/structured-out-404.json INFO /var/lib/ambari-agent/tmp' returned 1. 2017-11-29 07:01:50,049 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
> 2017-11-29 07:01:50,096 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 37, in <module>
> HbaseMasterUpgrade().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 28, in take_snapshot
> import params
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py", line 107, in <module>
> regionserver_xmn_percent = expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
> TypeError: 'module' object is not callable
> {code}
>
>
> Upon checking the code [here|https://github.com/hortonworks/ambari/blob/AMBARI-2.6.1.0/ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py#L30], the issue seems to be with import of 'expect' module
>
> I tried the following changes in params.py:
> {code}
> L30: from resource_management.libraries.functions import expect
> L107: regionserver_xmn_percent = expect.expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
> {code}
>
> Now the snapshot command ran fine:
> {code}
> 2017-11-29 11:14:15,472 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
> 2017-11-29 11:14:15,474 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
> 2017-11-29 11:14:15,476 - checked_call['hostid'] {}
> 2017-11-29 11:14:15,494 - checked_call returned (0, '16ace057')
> 2017-11-29 11:14:15,495 - Execute[' echo 'snapshot_all' | /usr/iop/current/hbase-client/bin/hbase shell'] {'user': 'hbase'}
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py c0c314d
>
>
> Diff: https://reviews.apache.org/r/64205/diff/3/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 64205: Snapshot HBase task failed during IOP
migration with TypeError
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64205/
-----------------------------------------------------------
(Updated Nov. 30, 2017, 11:20 p.m.)
Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
Changes
-------
removed * import, checked on live IOP cluster
Bugs: AMBARI-22558
https://issues.apache.org/jira/browse/AMBARI-22558
Repository: ambari
Description
-------
*STR*
# Deployed cluster with Ambari version: 2.2.0 and IOP version: 4.2.0.0
# Upgrade Ambari to Target Version: 2.5.2.0-298 | Hash: 2453e16418fd964042452b649153dbe45f3c6009
# Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
# Register HDP-2.6.4.0-51 and call remove iop-select
# Install the new HDP version bits and start Express Upgrade
*Result*
Snapshot HBase task failed with below error:
{code}
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 157, in <module>
ExecuteUpgradeTasks().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
method(env)
File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 153, in actionexecute
shell.checked_call(task.command, logoutput=True, quiet=True)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'source /var/lib/ambari-agent/ambari-env.sh ; /usr/bin/ambari-python-wrap /var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py take_snapshot /var/lib/ambari-agent/data/command-404.json /var/lib/ambari-agent/cache/custom_actions /var/lib/ambari-agent/data/structured-out-404.json INFO /var/lib/ambari-agent/tmp' returned 1. 2017-11-29 07:01:50,049 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
2017-11-29 07:01:50,096 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 37, in <module>
HbaseMasterUpgrade().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 28, in take_snapshot
import params
File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py", line 107, in <module>
regionserver_xmn_percent = expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
TypeError: 'module' object is not callable
{code}
Upon checking the code [here|https://github.com/hortonworks/ambari/blob/AMBARI-2.6.1.0/ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py#L30], the issue seems to be with import of 'expect' module
I tried the following changes in params.py:
{code}
L30: from resource_management.libraries.functions import expect
L107: regionserver_xmn_percent = expect.expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
{code}
Now the snapshot command ran fine:
{code}
2017-11-29 11:14:15,472 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
2017-11-29 11:14:15,474 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-11-29 11:14:15,476 - checked_call['hostid'] {}
2017-11-29 11:14:15,494 - checked_call returned (0, '16ace057')
2017-11-29 11:14:15,495 - Execute[' echo 'snapshot_all' | /usr/iop/current/hbase-client/bin/hbase shell'] {'user': 'hbase'}
{code}
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py c0c314d
Diff: https://reviews.apache.org/r/64205/diff/3/
Changes: https://reviews.apache.org/r/64205/diff/2-3/
Testing
-------
mvn clean test
Thanks,
Dmitro Lisnichenko
Re: Review Request 64205: Snapshot HBase task failed during IOP
migration with TypeError
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64205/#review192300
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py
Lines 24 (patched)
<https://reviews.apache.org/r/64205/#comment270364>
import * has gotten us into a lot of trouble. It's best to just remove it and fill in the imports for this file explicitely...
- Jonathan Hurley
On Nov. 30, 2017, 10:04 a.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64205/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2017, 10:04 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-22558
> https://issues.apache.org/jira/browse/AMBARI-22558
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *STR*
> # Deployed cluster with Ambari version: 2.2.0 and IOP version: 4.2.0.0
> # Upgrade Ambari to Target Version: 2.5.2.0-298 | Hash: 2453e16418fd964042452b649153dbe45f3c6009
> # Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
> # Register HDP-2.6.4.0-51 and call remove iop-select
> # Install the new HDP version bits and start Express Upgrade
>
> *Result*
> Snapshot HBase task failed with below error:
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 157, in <module>
> ExecuteUpgradeTasks().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 153, in actionexecute
> shell.checked_call(task.command, logoutput=True, quiet=True)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
> tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 'source /var/lib/ambari-agent/ambari-env.sh ; /usr/bin/ambari-python-wrap /var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py take_snapshot /var/lib/ambari-agent/data/command-404.json /var/lib/ambari-agent/cache/custom_actions /var/lib/ambari-agent/data/structured-out-404.json INFO /var/lib/ambari-agent/tmp' returned 1. 2017-11-29 07:01:50,049 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
> 2017-11-29 07:01:50,096 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 37, in <module>
> HbaseMasterUpgrade().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 28, in take_snapshot
> import params
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py", line 107, in <module>
> regionserver_xmn_percent = expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
> TypeError: 'module' object is not callable
> {code}
>
>
> Upon checking the code [here|https://github.com/hortonworks/ambari/blob/AMBARI-2.6.1.0/ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py#L30], the issue seems to be with import of 'expect' module
>
> I tried the following changes in params.py:
> {code}
> L30: from resource_management.libraries.functions import expect
> L107: regionserver_xmn_percent = expect.expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
> {code}
>
> Now the snapshot command ran fine:
> {code}
> 2017-11-29 11:14:15,472 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
> 2017-11-29 11:14:15,474 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
> 2017-11-29 11:14:15,476 - checked_call['hostid'] {}
> 2017-11-29 11:14:15,494 - checked_call returned (0, '16ace057')
> 2017-11-29 11:14:15,495 - Execute[' echo 'snapshot_all' | /usr/iop/current/hbase-client/bin/hbase shell'] {'user': 'hbase'}
> {code}
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py c0c314dcdf
>
>
> Diff: https://reviews.apache.org/r/64205/diff/2/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 64205: Snapshot HBase task failed during IOP
migration with TypeError
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64205/
-----------------------------------------------------------
(Updated Nov. 30, 2017, 5:04 p.m.)
Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
Changes
-------
another approach
Bugs: AMBARI-22558
https://issues.apache.org/jira/browse/AMBARI-22558
Repository: ambari
Description
-------
*STR*
# Deployed cluster with Ambari version: 2.2.0 and IOP version: 4.2.0.0
# Upgrade Ambari to Target Version: 2.5.2.0-298 | Hash: 2453e16418fd964042452b649153dbe45f3c6009
# Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
# Register HDP-2.6.4.0-51 and call remove iop-select
# Install the new HDP version bits and start Express Upgrade
*Result*
Snapshot HBase task failed with below error:
{code}
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 157, in <module>
ExecuteUpgradeTasks().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
method(env)
File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 153, in actionexecute
shell.checked_call(task.command, logoutput=True, quiet=True)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'source /var/lib/ambari-agent/ambari-env.sh ; /usr/bin/ambari-python-wrap /var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py take_snapshot /var/lib/ambari-agent/data/command-404.json /var/lib/ambari-agent/cache/custom_actions /var/lib/ambari-agent/data/structured-out-404.json INFO /var/lib/ambari-agent/tmp' returned 1. 2017-11-29 07:01:50,049 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
2017-11-29 07:01:50,096 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 37, in <module>
HbaseMasterUpgrade().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 28, in take_snapshot
import params
File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py", line 107, in <module>
regionserver_xmn_percent = expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
TypeError: 'module' object is not callable
{code}
Upon checking the code [here|https://github.com/hortonworks/ambari/blob/AMBARI-2.6.1.0/ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py#L30], the issue seems to be with import of 'expect' module
I tried the following changes in params.py:
{code}
L30: from resource_management.libraries.functions import expect
L107: regionserver_xmn_percent = expect.expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
{code}
Now the snapshot command ran fine:
{code}
2017-11-29 11:14:15,472 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
2017-11-29 11:14:15,474 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-11-29 11:14:15,476 - checked_call['hostid'] {}
2017-11-29 11:14:15,494 - checked_call returned (0, '16ace057')
2017-11-29 11:14:15,495 - Execute[' echo 'snapshot_all' | /usr/iop/current/hbase-client/bin/hbase shell'] {'user': 'hbase'}
{code}
Diffs (updated)
-----
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py c0c314dcdf
Diff: https://reviews.apache.org/r/64205/diff/2/
Changes: https://reviews.apache.org/r/64205/diff/1-2/
Testing
-------
mvn clean test
Thanks,
Dmitro Lisnichenko
Re: Review Request 64205: Snapshot HBase task failed during IOP
migration with TypeError
Posted by Dmytro Grinenko <ha...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/64205/#review192269
-----------------------------------------------------------
Ship it!
ambari-common/src/main/python/resource_management/libraries/functions/lzo_utils.py
Line 86 (original), 86 (patched)
<https://reviews.apache.org/r/64205/#comment270336>
this is kinda oddly "expect.except"
- Dmytro Grinenko
On Nov. 30, 2017, 11:46 a.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/64205/
> -----------------------------------------------------------
>
> (Updated Nov. 30, 2017, 11:46 a.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Jonathan Hurley, and Nate Cole.
>
>
> Bugs: AMBARI-22558
> https://issues.apache.org/jira/browse/AMBARI-22558
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *STR*
> # Deployed cluster with Ambari version: 2.2.0 and IOP version: 4.2.0.0
> # Upgrade Ambari to Target Version: 2.5.2.0-298 | Hash: 2453e16418fd964042452b649153dbe45f3c6009
> # Upgrade Ambari to Target Version: 2.6.1.0-64 | Hash: cd4db8e9ac0ea7ce14fc1253959a121688f34952
> # Register HDP-2.6.4.0-51 and call remove iop-select
> # Install the new HDP version bits and start Express Upgrade
>
> *Result*
> Snapshot HBase task failed with below error:
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 157, in <module>
> ExecuteUpgradeTasks().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/custom_actions/scripts/ru_execute_tasks.py", line 153, in actionexecute
> shell.checked_call(task.command, logoutput=True, quiet=True)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
> tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 'source /var/lib/ambari-agent/ambari-env.sh ; /usr/bin/ambari-python-wrap /var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py take_snapshot /var/lib/ambari-agent/data/command-404.json /var/lib/ambari-agent/cache/custom_actions /var/lib/ambari-agent/data/structured-out-404.json INFO /var/lib/ambari-agent/tmp' returned 1. 2017-11-29 07:01:50,049 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
> 2017-11-29 07:01:50,096 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 37, in <module>
> HbaseMasterUpgrade().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/hbase_upgrade.py", line 28, in take_snapshot
> import params
> File "/var/lib/ambari-agent/cache/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py", line 107, in <module>
> regionserver_xmn_percent = expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
> TypeError: 'module' object is not callable
> {code}
>
>
> Upon checking the code [here|https://github.com/hortonworks/ambari/blob/AMBARI-2.6.1.0/ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py#L30], the issue seems to be with import of 'expect' module
>
> I tried the following changes in params.py:
> {code}
> L30: from resource_management.libraries.functions import expect
> L107: regionserver_xmn_percent = expect.expect("/configurations/hbase-env/hbase_regionserver_xmn_ratio", float) #AMBARI-15614
> {code}
>
> Now the snapshot command ran fine:
> {code}
> 2017-11-29 11:14:15,472 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=4.2.0.0, Upgrade Direction=upgrade -> 4.2.0.0
> 2017-11-29 11:14:15,474 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
> 2017-11-29 11:14:15,476 - checked_call['hostid'] {}
> 2017-11-29 11:14:15,494 - checked_call returned (0, '16ace057')
> 2017-11-29 11:14:15,495 - Execute[' echo 'snapshot_all' | /usr/iop/current/hbase-client/bin/hbase shell'] {'user': 'hbase'}
> {code}
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/libraries/functions/lzo_utils.py 68ee607b9b
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py f7fd5459c1
> ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params_linux.py 0ca1f0bd21
> ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py 0f6ee01c70
> ambari-server/src/main/resources/common-services/FALCON/0.5.0.2.1/package/scripts/params_linux.py b8e709a6db
> ambari-server/src/main/resources/common-services/HBASE/0.96.0.2.0/package/scripts/params_linux.py 46f715ecde
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 5e1f4ac5cb
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/params_linux.py 18e297865e
> ambari-server/src/main/resources/common-services/KERBEROS/1.10.3-10/package/scripts/params.py 4eb5b02c3f
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params.py f39d6326de
> ambari-server/src/main/resources/common-services/OOZIE/4.0.0.2.0/package/scripts/params_linux.py 8f38843364
> ambari-server/src/main/resources/common-services/SQOOP/1.4.4.2.0/package/scripts/params_linux.py eaf1ee4940
> ambari-server/src/main/resources/common-services/STORM/0.9.1/package/scripts/params_linux.py d9fae763ad
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5/package/scripts/params_linux.py 34c6b9ab96
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/HBASE/package/scripts/params.py 97657ad7f4
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-ANY/scripts/params.py 1ed6d4d748
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/hooks/before-INSTALL/scripts/params.py 6193c11c22
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HBASE/package/scripts/params_linux.py bff022d1f7
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/HDFS/package/scripts/params_linux.py a77ec82cdc
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/package/scripts/params.py f39d6326de
> ambari-server/src/main/resources/stacks/BigInsights/4.2.5/services/OOZIE/package/scripts/params_linux.py 5868f5e604
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/HBASE/package/scripts/params.py c0c314dcdf
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/KAFKA/package/scripts/params.py 07e7ad19d1
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 8ad0d51376
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/params.py da75a8ee89
> ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-ANY/scripts/params.py 2c2c90113a
> ambari-server/src/main/resources/stacks/PERF/1.0/services/KERBEROS/package/scripts/params.py 4eb5b02c3f
>
>
> Diff: https://reviews.apache.org/r/64205/diff/1/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>