You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Oliver Szabo <os...@hortonworks.com> on 2015/12/09 14:31:32 UTC
Review Request 41133: Hive Server 2 silently dies on Blueprint cluster
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41133/
-----------------------------------------------------------
Review request for Ambari, Alejandro Fernandez and Sebastian Toader.
Bugs: AMBARI-14270
https://issues.apache.org/jira/browse/AMBARI-14270
Repository: ambari
Description
-------
HiveServer2 and SparkThriftserver uses 10000 port by default. Ambari redefines this port (10015) in case of commandParams/version exists (based on spark-hive-site-override.xml).
During blueprint provisioning commandParams/version not accesible (when all of the install/start task sent with predefined configurations), because Ambari needs at least 1 installed service package to determine the version.
On start, service packages already installed, so on that phase /usr/bin/hdp-select is usable. -> I used that to gather the hdp stack version, when version is not available. (did the same with KNOX too)
Notice that: With Blueprints, role command order applied only on the hosts, not on the cluster, so SparkThriftServer wait for HiveMetastore starts only on the same hosts. That means its possible: other SparkThriftServer cannot be started (but can be restarted after the installation)
Also I added 3 new properties (from hive-site) to SparkThriftServer:
'hive.metastore.connect.retries', 'hive.metastore.client.connect.retry.delay', 'hive.metastore.client.socket.timeout'
- default delay is too low, now it set as the same as in hive
Diffs
-----
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py ec972f6
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 9f0147a
Diff: https://reviews.apache.org/r/41133/diff/
Testing
-------
Total run:814
Total errors:0
Total failures:0
OK
INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
INFO: Return code from stack upgrade command, retcode = 0
StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
Returning DefaultStackAdvisor implementation
StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
Returning XYZ101StackAdvisor implementation
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:02:05.981s
[INFO] Finished at: Wed Dec 09 14:11:16 CET 2015
[INFO] Final Memory: 38M/938M
[INFO] ------------------------------------------------------------------------
Thanks,
Oliver Szabo
Re: Review Request 41133: Hive Server 2 silently dies on Blueprint
cluster
Posted by Sebastian Toader <st...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41133/#review109506
-----------------------------------------------------------
Ship it!
Fix and ship it !
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py (line 158)
<https://reviews.apache.org/r/41133/#comment169079>
Add some comment explaining why is this needed
- Sebastian Toader
On Dec. 9, 2015, 2:31 p.m., Oliver Szabo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41133/
> -----------------------------------------------------------
>
> (Updated Dec. 9, 2015, 2:31 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sebastian Toader.
>
>
> Bugs: AMBARI-14270
> https://issues.apache.org/jira/browse/AMBARI-14270
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HiveServer2 and SparkThriftserver uses 10000 port by default. Ambari redefines this port (10015) in case of commandParams/version exists (based on spark-hive-site-override.xml).
> During blueprint provisioning commandParams/version not accesible (when all of the install/start task sent with predefined configurations), because Ambari needs at least 1 installed service package to determine the version.
>
> On start, service packages already installed, so on that phase /usr/bin/hdp-select is usable. -> I used that to gather the hdp stack version, when version is not available. (did the same with KNOX too)
>
> Notice that: With Blueprints, role command order applied only on the hosts, not on the cluster, so SparkThriftServer wait for HiveMetastore starts only on the same hosts. That means its possible: other SparkThriftServer cannot be started (but can be restarted after the installation)
>
> Also I added 3 new properties (from hive-site) to SparkThriftServer:
> 'hive.metastore.connect.retries', 'hive.metastore.client.connect.retry.delay', 'hive.metastore.client.socket.timeout'
> - default delay is too low, now it set as the same as in hive
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py ec972f6
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 9f0147a
>
> Diff: https://reviews.apache.org/r/41133/diff/
>
>
> Testing
> -------
>
> Total run:814
> Total errors:0
> Total failures:0
> OK
> INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
> INFO: Return code from stack upgrade command, retcode = 0
> StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
> Returning DefaultStackAdvisor implementation
> StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
> StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
> Returning XYZ101StackAdvisor implementation
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 1:02:05.981s
> [INFO] Finished at: Wed Dec 09 14:11:16 CET 2015
> [INFO] Final Memory: 38M/938M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Oliver Szabo
>
>
Re: Review Request 41133: Hive Server 2 silently dies on Blueprint
cluster
Posted by Oliver Szabo <os...@hortonworks.com>.
> On Dec. 9, 2015, 7:30 p.m., Alejandro Fernandez wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py, line 159
> > <https://reviews.apache.org/r/41133/diff/2/?file=1157113#file1157113line159>
> >
> > Same comments above regarding version.
> >
> > stack_version_unformatted = str(config['hostLevelParams']['stack_version'])
> > hdp_stack_version = format_hdp_stack_version(stack_version_unformatted)
> >
> > # New Cluster Stack Version that is defined during the RESTART of a Stack Upgrade
> > version = default("/commandParams/version", None)
It will be fixed in AMBARI-14330
- Oliver
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41133/#review109588
-----------------------------------------------------------
On Dec. 9, 2015, 2:19 p.m., Oliver Szabo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41133/
> -----------------------------------------------------------
>
> (Updated Dec. 9, 2015, 2:19 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sebastian Toader.
>
>
> Bugs: AMBARI-14270
> https://issues.apache.org/jira/browse/AMBARI-14270
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HiveServer2 and SparkThriftserver uses 10000 port by default. Ambari redefines this port (10015) in case of commandParams/version exists (based on spark-hive-site-override.xml).
> During blueprint provisioning commandParams/version not accesible (when all of the install/start task sent with predefined configurations), because Ambari needs at least 1 installed service package to determine the version.
>
> On start, service packages already installed, so on that phase /usr/bin/hdp-select is usable. -> I used that to gather the hdp stack version, when version is not available. (did the same with KNOX too)
>
> Notice that: With Blueprints, role command order applied only on the hosts, not on the cluster, so SparkThriftServer wait for HiveMetastore starts only on the same hosts. That means its possible: other SparkThriftServer cannot be started (but can be restarted after the installation)
>
> Also I added 3 new properties (from hive-site) to SparkThriftServer:
> 'hive.metastore.connect.retries', 'hive.metastore.client.connect.retry.delay', 'hive.metastore.client.socket.timeout'
> - default delay is too low, now it set as the same as in hive
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py f26874f
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 18a0052
>
> Diff: https://reviews.apache.org/r/41133/diff/
>
>
> Testing
> -------
>
> Total run:814
> Total errors:0
> Total failures:0
> OK
> INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
> INFO: Return code from stack upgrade command, retcode = 0
> StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
> Returning DefaultStackAdvisor implementation
> StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
> StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
> Returning XYZ101StackAdvisor implementation
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 1:02:05.981s
> [INFO] Finished at: Wed Dec 09 14:11:16 CET 2015
> [INFO] Final Memory: 38M/938M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Oliver Szabo
>
>
Re: Review Request 41133: Hive Server 2 silently dies on Blueprint
cluster
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41133/#review109588
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py (line 43)
<https://reviews.apache.org/r/41133/#comment169163>
This is the version that the comment should be on during an RU/EU, why are we changing the variable if it's None to the current value pointed by /usr/hdp/current/knox-server?
There's another parameter for the stack, such as 2.2 or 2.3, that is used during normal operation.
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py (line 152)
<https://reviews.apache.org/r/41133/#comment169164>
Same comments above regarding version.
stack_version_unformatted = str(config['hostLevelParams']['stack_version'])
hdp_stack_version = format_hdp_stack_version(stack_version_unformatted)
# New Cluster Stack Version that is defined during the RESTART of a Stack Upgrade
version = default("/commandParams/version", None)
- Alejandro Fernandez
On Dec. 9, 2015, 2:19 p.m., Oliver Szabo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41133/
> -----------------------------------------------------------
>
> (Updated Dec. 9, 2015, 2:19 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sebastian Toader.
>
>
> Bugs: AMBARI-14270
> https://issues.apache.org/jira/browse/AMBARI-14270
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HiveServer2 and SparkThriftserver uses 10000 port by default. Ambari redefines this port (10015) in case of commandParams/version exists (based on spark-hive-site-override.xml).
> During blueprint provisioning commandParams/version not accesible (when all of the install/start task sent with predefined configurations), because Ambari needs at least 1 installed service package to determine the version.
>
> On start, service packages already installed, so on that phase /usr/bin/hdp-select is usable. -> I used that to gather the hdp stack version, when version is not available. (did the same with KNOX too)
>
> Notice that: With Blueprints, role command order applied only on the hosts, not on the cluster, so SparkThriftServer wait for HiveMetastore starts only on the same hosts. That means its possible: other SparkThriftServer cannot be started (but can be restarted after the installation)
>
> Also I added 3 new properties (from hive-site) to SparkThriftServer:
> 'hive.metastore.connect.retries', 'hive.metastore.client.connect.retry.delay', 'hive.metastore.client.socket.timeout'
> - default delay is too low, now it set as the same as in hive
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py f26874f
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 18a0052
>
> Diff: https://reviews.apache.org/r/41133/diff/
>
>
> Testing
> -------
>
> Total run:814
> Total errors:0
> Total failures:0
> OK
> INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
> INFO: Return code from stack upgrade command, retcode = 0
> StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
> Returning DefaultStackAdvisor implementation
> StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
> StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
> Returning XYZ101StackAdvisor implementation
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 1:02:05.981s
> [INFO] Finished at: Wed Dec 09 14:11:16 CET 2015
> [INFO] Final Memory: 38M/938M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Oliver Szabo
>
>
Re: Review Request 41133: Hive Server 2 silently dies on Blueprint
cluster
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41133/#review109514
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Dec. 9, 2015, 2:19 p.m., Oliver Szabo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41133/
> -----------------------------------------------------------
>
> (Updated Dec. 9, 2015, 2:19 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sebastian Toader.
>
>
> Bugs: AMBARI-14270
> https://issues.apache.org/jira/browse/AMBARI-14270
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HiveServer2 and SparkThriftserver uses 10000 port by default. Ambari redefines this port (10015) in case of commandParams/version exists (based on spark-hive-site-override.xml).
> During blueprint provisioning commandParams/version not accesible (when all of the install/start task sent with predefined configurations), because Ambari needs at least 1 installed service package to determine the version.
>
> On start, service packages already installed, so on that phase /usr/bin/hdp-select is usable. -> I used that to gather the hdp stack version, when version is not available. (did the same with KNOX too)
>
> Notice that: With Blueprints, role command order applied only on the hosts, not on the cluster, so SparkThriftServer wait for HiveMetastore starts only on the same hosts. That means its possible: other SparkThriftServer cannot be started (but can be restarted after the installation)
>
> Also I added 3 new properties (from hive-site) to SparkThriftServer:
> 'hive.metastore.connect.retries', 'hive.metastore.client.connect.retry.delay', 'hive.metastore.client.socket.timeout'
> - default delay is too low, now it set as the same as in hive
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py f26874f
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 18a0052
>
> Diff: https://reviews.apache.org/r/41133/diff/
>
>
> Testing
> -------
>
> Total run:814
> Total errors:0
> Total failures:0
> OK
> INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
> INFO: Return code from stack upgrade command, retcode = 0
> StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
> Returning DefaultStackAdvisor implementation
> StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
> StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
> Returning XYZ101StackAdvisor implementation
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 1:02:05.981s
> [INFO] Finished at: Wed Dec 09 14:11:16 CET 2015
> [INFO] Final Memory: 38M/938M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Oliver Szabo
>
>
Re: Review Request 41133: Hive Server 2 silently dies on Blueprint
cluster
Posted by Sebastian Toader <st...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41133/#review109508
-----------------------------------------------------------
Ship it!
Ship It!
- Sebastian Toader
On Dec. 9, 2015, 3:19 p.m., Oliver Szabo wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41133/
> -----------------------------------------------------------
>
> (Updated Dec. 9, 2015, 3:19 p.m.)
>
>
> Review request for Ambari, Alejandro Fernandez and Sebastian Toader.
>
>
> Bugs: AMBARI-14270
> https://issues.apache.org/jira/browse/AMBARI-14270
>
>
> Repository: ambari
>
>
> Description
> -------
>
> HiveServer2 and SparkThriftserver uses 10000 port by default. Ambari redefines this port (10015) in case of commandParams/version exists (based on spark-hive-site-override.xml).
> During blueprint provisioning commandParams/version not accesible (when all of the install/start task sent with predefined configurations), because Ambari needs at least 1 installed service package to determine the version.
>
> On start, service packages already installed, so on that phase /usr/bin/hdp-select is usable. -> I used that to gather the hdp stack version, when version is not available. (did the same with KNOX too)
>
> Notice that: With Blueprints, role command order applied only on the hosts, not on the cluster, so SparkThriftServer wait for HiveMetastore starts only on the same hosts. That means its possible: other SparkThriftServer cannot be started (but can be restarted after the installation)
>
> Also I added 3 new properties (from hive-site) to SparkThriftServer:
> 'hive.metastore.connect.retries', 'hive.metastore.client.connect.retry.delay', 'hive.metastore.client.socket.timeout'
> - default delay is too low, now it set as the same as in hive
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py f26874f
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 18a0052
>
> Diff: https://reviews.apache.org/r/41133/diff/
>
>
> Testing
> -------
>
> Total run:814
> Total errors:0
> Total failures:0
> OK
> INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
> INFO: Return code from stack upgrade command, retcode = 0
> StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
> Returning DefaultStackAdvisor implementation
> StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
> StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
> Returning XYZ101StackAdvisor implementation
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 1:02:05.981s
> [INFO] Finished at: Wed Dec 09 14:11:16 CET 2015
> [INFO] Final Memory: 38M/938M
> [INFO] ------------------------------------------------------------------------
>
>
> Thanks,
>
> Oliver Szabo
>
>
Re: Review Request 41133: Hive Server 2 silently dies on Blueprint
cluster
Posted by Oliver Szabo <os...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41133/
-----------------------------------------------------------
(Updated Dec. 9, 2015, 2:19 p.m.)
Review request for Ambari, Alejandro Fernandez and Sebastian Toader.
Changes
-------
added comment about hive metastore client properties
Bugs: AMBARI-14270
https://issues.apache.org/jira/browse/AMBARI-14270
Repository: ambari
Description
-------
HiveServer2 and SparkThriftserver uses 10000 port by default. Ambari redefines this port (10015) in case of commandParams/version exists (based on spark-hive-site-override.xml).
During blueprint provisioning commandParams/version not accesible (when all of the install/start task sent with predefined configurations), because Ambari needs at least 1 installed service package to determine the version.
On start, service packages already installed, so on that phase /usr/bin/hdp-select is usable. -> I used that to gather the hdp stack version, when version is not available. (did the same with KNOX too)
Notice that: With Blueprints, role command order applied only on the hosts, not on the cluster, so SparkThriftServer wait for HiveMetastore starts only on the same hosts. That means its possible: other SparkThriftServer cannot be started (but can be restarted after the installation)
Also I added 3 new properties (from hive-site) to SparkThriftServer:
'hive.metastore.connect.retries', 'hive.metastore.client.connect.retry.delay', 'hive.metastore.client.socket.timeout'
- default delay is too low, now it set as the same as in hive
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/params_linux.py f26874f
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 18a0052
Diff: https://reviews.apache.org/r/41133/diff/
Testing
-------
Total run:814
Total errors:0
Total failures:0
OK
INFO: AMBARI_SERVER_LIB is not set, using default /usr/lib/ambari-server
INFO: Return code from stack upgrade command, retcode = 0
StackAdvisor implementation for stack HDP1, version 2.0.6 was not found
Returning DefaultStackAdvisor implementation
StackAdvisor implementation for stack XYZ, version 1.0.0 was loaded
StackAdvisor implementation for stack XYZ, version 1.0.1 was loaded
Returning XYZ101StackAdvisor implementation
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:02:05.981s
[INFO] Finished at: Wed Dec 09 14:11:16 CET 2015
[INFO] Final Memory: 38M/938M
[INFO] ------------------------------------------------------------------------
Thanks,
Oliver Szabo