You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Dmitro Lisnichenko <dl...@hortonworks.com> on 2015/12/13 22:11:59 UTC
Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Bugs: AMBARI-14362
https://issues.apache.org/jira/browse/AMBARI-14362
Repository: ambari
Description
-------
*Steps:*
Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
Did EU from HDP 2.2.9 to 2.3.2.0
Result:
Observed during EU that Kakfa broker restart failed on all hosts with error:
{code}
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
KafkaBroker().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
self.stop(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
user=params.kafka_user,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
failed.
{code}
The problem is likely due to custom service user being used for Kafka
Diffs
-----
ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
Diff: https://reviews.apache.org/r/41328/diff/
Testing
-------
mvn clean test + check on live cluster
Thanks,
Dmitro Lisnichenko
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Nate Cole <nc...@hortonworks.com>.
> On Dec. 14, 2015, 11:47 a.m., Nate Cole wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py, lines 233-235
> > <https://reviews.apache.org/r/41328/diff/3/?file=1162545#file1162545line233>
> >
> > The Directory resource allows for owner and group. Is that broken?
>
> Dmitro Lisnichenko wrote:
> No, but it does not assume setting ownership/permissions recursively on files in directory
Ah, I see now. Sheesh :)
- Nate
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110251
-----------------------------------------------------------
On Dec. 14, 2015, 10:07 a.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 10:07 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Dec. 14, 2015, 6:47 p.m., Nate Cole wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py, lines 233-235
> > <https://reviews.apache.org/r/41328/diff/3/?file=1162545#file1162545line233>
> >
> > The Directory resource allows for owner and group. Is that broken?
No, but it does not assume setting ownership/permissions recursively on files in directory
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110251
-----------------------------------------------------------
On Dec. 14, 2015, 5:07 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 5:07 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Dec. 14, 2015, 6:47 p.m., Nate Cole wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py, lines 243-245
> > <https://reviews.apache.org/r/41328/diff/3/?file=1162545#file1162545line243>
> >
> > The closing paren is out in no man's land. Can you put it after sudo=True please. Could probably all fit on one one.
fixed. Not on 1 line due to 80-character limit
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110251
-----------------------------------------------------------
On Dec. 14, 2015, 8:02 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 8:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110251
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py (lines 225 - 227)
<https://reviews.apache.org/r/41328/#comment170056>
The Directory resource allows for owner and group. Is that broken?
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py (lines 235 - 237)
<https://reviews.apache.org/r/41328/#comment170057>
The closing paren is out in no man's land. Can you put it after sudo=True please. Could probably all fit on one one.
- Nate Cole
On Dec. 14, 2015, 10:07 a.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 10:07 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Dec. 14, 2015, 6:29 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py, lines 235-239
> > <https://reviews.apache.org/r/41328/diff/3/?file=1162545#file1162545line235>
> >
> > Can we use some self-documenting names here? "d" and "l" are not very intuitive.
fixed
> On Dec. 14, 2015, 6:29 p.m., Jonathan Hurley wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py, line 88
> > <https://reviews.apache.org/r/41328/diff/3/?file=1162545#file1162545line88>
> >
> > If we're going to use some Python voodoo, we should probably explain why we're doing it so that it's not in advertently removed.
> >
> > My guess is this slice does some sort of copy operation on the original collection - why are we doing this here? I didn't think the Directory resource was destructive to lists.
Your guess is right, resources have side effect on list. Opened a 2.3.0 jira AMBARI-14373
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110247
-----------------------------------------------------------
On Dec. 14, 2015, 8:02 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 8:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110247
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py (line 80)
<https://reviews.apache.org/r/41328/#comment170048>
If we're going to use some Python voodoo, we should probably explain why we're doing it so that it's not in advertently removed.
My guess is this slice does some sort of copy operation on the original collection - why are we doing this here? I didn't think the Directory resource was destructive to lists.
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py (line 215)
<https://reviews.apache.org/r/41328/#comment170051>
Same with above comment - why slice nothing out of the list?
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py (lines 227 - 231)
<https://reviews.apache.org/r/41328/#comment170052>
Can we use some self-documenting names here? "d" and "l" are not very intuitive.
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py (lines 87 - 90)
<https://reviews.apache.org/r/41328/#comment170050>
Is this a Kafka only problem!? If it is, shouldn't it be fixed in Kafka? I'm always against special-casing bugs in the stack in Ambari and I want to understand more about why we need to do this workaround.
- Jonathan Hurley
On Dec. 14, 2015, 10:07 a.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 10:07 a.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Dec. 14, 2015, 8:17 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py, line 90
> > <https://reviews.apache.org/r/41328/diff/3/?file=1162546#file1162546line90>
> >
> > make_directories on stop() doesnt make sense. Why create directories when stopping. At best should be termed reset permissions and not make directories.
> >
> > Also, can you elaborate on how and when the permissions are changed. I am not quite clear on this.
Regarding naming - how about ensure_base_directories() ? The same code creates directories on Kafka start
Regarding how/when permissions are changed: they are changed when we install new repo version bits (probably by package scripts). I'm filing a jira for RE team as of now on that.
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110273
-----------------------------------------------------------
On Dec. 14, 2015, 8:02 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 8:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Dec. 14, 2015, 8:17 p.m., Jayush Luniya wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py, line 90
> > <https://reviews.apache.org/r/41328/diff/3/?file=1162546#file1162546line90>
> >
> > make_directories on stop() doesnt make sense. Why create directories when stopping. At best should be termed reset permissions and not make directories.
> >
> > Also, can you elaborate on how and when the permissions are changed. I am not quite clear on this.
>
> Dmitro Lisnichenko wrote:
> Regarding naming - how about ensure_base_directories() ? The same code creates directories on Kafka start
>
> Regarding how/when permissions are changed: they are changed when we install new repo version bits (probably by package scripts). I'm filing a jira for RE team as of now on that.
created RE jira with details, cc'ed you in it
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110273
-----------------------------------------------------------
On Dec. 14, 2015, 8:52 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 8:52 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Jayush Luniya <jl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110273
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py (line 90)
<https://reviews.apache.org/r/41328/#comment170090>
make_directories on stop() doesnt make sense. Why create directories when stopping. At best should be termed reset permissions and not make directories.
Also, can you elaborate on how and when the permissions are changed. I am not quite clear on this.
- Jayush Luniya
On Dec. 14, 2015, 6:02 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 6:02 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Nate Cole <nc...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110309
-----------------------------------------------------------
Ship it!
Ship It!
- Nate Cole
On Dec. 14, 2015, 1:52 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 1:52 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Jonathan Hurley <jh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110307
-----------------------------------------------------------
Ship it!
Ship It!
- Jonathan Hurley
On Dec. 14, 2015, 1:52 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 1:52 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Dec. 15, 2015, 1:47 a.m., Alejandro Fernandez wrote:
> > Pushed to trunk,
> > commit 55be31abcb3f8f1419df3387eb93bf1f51f18e54
> >
> > Pushed to branch-2.2,
> > commit 2e6b878729ae8276b486870a39e36699052d0a88
thanks
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110342
-----------------------------------------------------------
On Dec. 14, 2015, 8:52 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 8:52 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110342
-----------------------------------------------------------
Pushed to trunk,
commit 55be31abcb3f8f1419df3387eb93bf1f51f18e54
Pushed to branch-2.2,
commit 2e6b878729ae8276b486870a39e36699052d0a88
- Alejandro Fernandez
On Dec. 14, 2015, 6:52 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 14, 2015, 6:52 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 8:52 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Changes
-------
Renamed method
Bugs: AMBARI-14362
https://issues.apache.org/jira/browse/AMBARI-14362
Repository: ambari
Description
-------
*Steps:*
Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
Did EU from HDP 2.2.9 to 2.3.2.0
Result:
Observed during EU that Kakfa broker restart failed on all hosts with error:
{code}
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
KafkaBroker().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
self.stop(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
user=params.kafka_user,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
failed.
{code}
The problem is likely due to custom service user being used for Kafka
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
Diff: https://reviews.apache.org/r/41328/diff/
Testing
-------
mvn clean test + check on live cluster
Thanks,
Dmitro Lisnichenko
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 8:02 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Changes
-------
Uploaded new patch
Bugs: AMBARI-14362
https://issues.apache.org/jira/browse/AMBARI-14362
Repository: ambari
Description
-------
*Steps:*
Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
Did EU from HDP 2.2.9 to 2.3.2.0
Result:
Observed during EU that Kakfa broker restart failed on all hosts with error:
{code}
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
KafkaBroker().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
self.stop(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
user=params.kafka_user,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
failed.
{code}
The problem is likely due to custom service user being used for Kafka
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
Diff: https://reviews.apache.org/r/41328/diff/
Testing
-------
mvn clean test + check on live cluster
Thanks,
Dmitro Lisnichenko
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/
-----------------------------------------------------------
(Updated Dec. 14, 2015, 5:07 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Changes
-------
uploaded new patch
Bugs: AMBARI-14362
https://issues.apache.org/jira/browse/AMBARI-14362
Repository: ambari
Description
-------
*Steps:*
Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
Did EU from HDP 2.2.9 to 2.3.2.0
Result:
Observed during EU that Kakfa broker restart failed on all hosts with error:
{code}
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
KafkaBroker().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
self.stop(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
user=params.kafka_user,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
failed.
{code}
The problem is likely due to custom service user being used for Kafka
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
Diff: https://reviews.apache.org/r/41328/diff/
Testing
-------
mvn clean test + check on live cluster
Thanks,
Dmitro Lisnichenko
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Dec. 13, 2015, 11:21 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py, line 89
> > <https://reviews.apache.org/r/41328/diff/2/?file=1161946#file1161946line89>
> >
> > I don't know if we can call configure on a stop during RU.
> > The reason is that RU sends a restart command in which we stop (using the existing configs from say stack 2.2), and then we start using the new configs (say 2.3).
> > We should test this out.
you are right. Reworked patch, so now stop() only adjusts the required minimum of permissions/ownership
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110140
-----------------------------------------------------------
On Dec. 13, 2015, 11:14 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 11:14 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
> On Dec. 13, 2015, 11:21 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py, line 38
> > <https://reviews.apache.org/r/41328/diff/2/?file=1161945#file1161945line38>
> >
> > No need to use [:]
we need this since resources have side effects on passed lists (the list becomes empty)
- Dmitro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110140
-----------------------------------------------------------
On Dec. 13, 2015, 11:14 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 11:14 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/#review110140
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py (line 38)
<https://reviews.apache.org/r/41328/#comment169908>
No need to use [:]
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py (line 89)
<https://reviews.apache.org/r/41328/#comment169907>
I don't know if we can call configure on a stop during RU.
The reason is that RU sends a restart command in which we stop (using the existing configs from say stack 2.2), and then we start using the new configs (say 2.3).
We should test this out.
- Alejandro Fernandez
On Dec. 13, 2015, 9:14 p.m., Dmitro Lisnichenko wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41328/
> -----------------------------------------------------------
>
> (Updated Dec. 13, 2015, 9:14 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
>
>
> Bugs: AMBARI-14362
> https://issues.apache.org/jira/browse/AMBARI-14362
>
>
> Repository: ambari
>
>
> Description
> -------
>
> *Steps:*
> Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
> Did EU from HDP 2.2.9 to 2.3.2.0
>
> Result:
> Observed during EU that Kakfa broker restart failed on all hosts with error:
>
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
> KafkaBroker().execute()
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
> method(env)
> File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
> self.stop(env, upgrade_type=upgrade_type)
> File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
> user=params.kafka_user,
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
> self.env.run()
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
> self.run_action(resource, action)
> File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
> provider_action()
> File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
> raise Fail(err_msg)
> resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
> failed.
> {code}
>
> The problem is likely due to custom service user being used for Kafka
>
>
> Diffs
> -----
>
> ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
> ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
>
> Diff: https://reviews.apache.org/r/41328/diff/
>
>
> Testing
> -------
>
> mvn clean test + check on live cluster
>
>
> Thanks,
>
> Dmitro Lisnichenko
>
>
Re: Review Request 41328: Express Upgrade: Kafka broker restart failed
during EU from 2.2.9 to 2.3.2 with customized service user accounts
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41328/
-----------------------------------------------------------
(Updated Dec. 13, 2015, 11:14 p.m.)
Review request for Ambari, Alejandro Fernandez, Jonathan Hurley, Jayush Luniya, and Nate Cole.
Bugs: AMBARI-14362
https://issues.apache.org/jira/browse/AMBARI-14362
Repository: ambari
Description
-------
*Steps:*
Setup HDP 2.2.9 cluster with Ambari build #2.2.0.0-1281; the cluster has HA enabled and setup with customized service users for all services
Did EU from HDP 2.2.9 to 2.3.2.0
Result:
Observed during EU that Kakfa broker restart failed on all hosts with error:
{code}
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 101, in <module>
KafkaBroker().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 496, in restart
self.stop(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py", line 88, in stop
user=params.kafka_user,
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'source /usr/hdp/current/kafka-broker/config/kafka-env.sh; /usr/hdp/current/kafka-broker/bin/kafka stop' returned 1. Stopping Kafka [23359] /usr/hdp/current/kafka-broker/bin/kafka: line 110: /var/log/kafka/kafka.out: Permission denied
failed.
{code}
The problem is likely due to custom service user being used for Kafka
Diffs (updated)
-----
ambari-common/src/main/python/resource_management/core/resources/system.py 83abc64
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka.py a8bd9d5
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 6fcf08a
ambari-server/src/test/python/stacks/2.2/KAFKA/test_kafka_broker.py 72cb050
Diff: https://reviews.apache.org/r/41328/diff/
Testing
-------
mvn clean test + check on live cluster
Thanks,
Dmitro Lisnichenko