You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Andrew Onischuk <ao...@hortonworks.com> on 2015/05/20 18:53:31 UTC
Review Request 34478: Move hiveserver2 beeline check to Hive SC and do
mupliple other service-level optimizations
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34478/
-----------------------------------------------------------
Review request for Ambari and Vitalyi Brodetskyi.
Bugs: AMBARI-11271
https://issues.apache.org/jira/browse/AMBARI-11271
Repository: ambari
Description
-------
Currently multiple customers are complaining on high services (re-)start time.
>From what we understood probably more than a minute is bad.
So after discussion with Mahadev it was decided to move out beeline check
which takes usually ~2 minutes (resulting in hive restarting in ~4minutes)
from start to service check, which seems a good place for that.
There is a problem due to Hive process asynchroniously starts and can drop
start without Ambari knowing it.
Also this was implemented some time ago because of:
AMBARI-5800
Race condition when starting all services causing Hive service check to fail
But, since now service check does exactly the beeline check with mutiple
retries this won't happen
Test scenarios included:
1\. Restart Hive and rt after it start service check
2\. Restart Hive and rt after it restart Webhcat
3\. Start service check with Hive Server HA
Diffs
-----
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs.py 418571a
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py d02bb39
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 453d824
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_nfsgateway.py ac0e24d
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_snamenode.py 78ef977
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode.py ada05e1
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 3f1fb0d
ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 217f2f0
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 32bc0ba
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py 86312d2
ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/service_check.py 04bd3b2
ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 31d949a
ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 956d09b
ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service.py 3d6b8d1
ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d685d1d
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py b58959a
ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/shared_initialization.py dd93818
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py 4eb8178
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_journalnode.py 3fe2acb
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py c753c39
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_nfsgateway.py 7255ea5
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_snamenode.py fd665c3
ambari-server/src/test/python/stacks/2.0.6/HDFS/test_zkfc.py c2fdffa
ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 90f3e06
ambari-server/src/test/python/stacks/2.0.6/YARN/test_historyserver.py bc527e7
ambari-server/src/test/python/stacks/2.0.6/YARN/test_nodemanager.py 005b533
ambari-server/src/test/python/stacks/2.0.6/YARN/test_resourcemanager.py a2ee27f
ambari-server/src/test/python/stacks/2.0.6/ZOOKEEPER/test_zookeeper_server.py ca8cf16
ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py e752427
ambari-server/src/test/python/stacks/2.0.6/hooks/before-START/test_before_start.py fb2699a
ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py 9e8b405
Diff: https://reviews.apache.org/r/34478/diff/
Testing
-------
mvn clean test
Thanks,
Andrew Onischuk
Re: Review Request 34478: Move hiveserver2 beeline check to Hive SC
and do mupliple other service-level optimizations
Posted by Dmitro Lisnichenko <dl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34478/#review84538
-----------------------------------------------------------
Ship it!
Ship It!
- Dmitro Lisnichenko
On May 20, 2015, 4:53 p.m., Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34478/
> -----------------------------------------------------------
>
> (Updated May 20, 2015, 4:53 p.m.)
>
>
> Review request for Ambari and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-11271
> https://issues.apache.org/jira/browse/AMBARI-11271
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently multiple customers are complaining on high services (re-)start time.
> From what we understood probably more than a minute is bad.
>
> So after discussion with Mahadev it was decided to move out beeline check
> which takes usually ~2 minutes (resulting in hive restarting in ~4minutes)
> from start to service check, which seems a good place for that.
>
> There is a problem due to Hive process asynchroniously starts and can drop
> start without Ambari knowing it.
> Also this was implemented some time ago because of:
>
>
>
> AMBARI-5800
> Race condition when starting all services causing Hive service check to fail
>
>
> But, since now service check does exactly the beeline check with mutiple
> retries this won't happen
>
> Test scenarios included:
> 1\. Restart Hive and rt after it start service check
> 2\. Restart Hive and rt after it restart Webhcat
> 3\. Start service check with Hive Server HA
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs.py 418571a
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py d02bb39
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 453d824
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_nfsgateway.py ac0e24d
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_snamenode.py 78ef977
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode.py ada05e1
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 3f1fb0d
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 217f2f0
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 32bc0ba
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py 86312d2
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/service_check.py 04bd3b2
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 31d949a
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 956d09b
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service.py 3d6b8d1
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d685d1d
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py b58959a
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/shared_initialization.py dd93818
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py 4eb8178
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_journalnode.py 3fe2acb
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py c753c39
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_nfsgateway.py 7255ea5
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_snamenode.py fd665c3
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_zkfc.py c2fdffa
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 90f3e06
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_historyserver.py bc527e7
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_nodemanager.py 005b533
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_resourcemanager.py a2ee27f
> ambari-server/src/test/python/stacks/2.0.6/ZOOKEEPER/test_zookeeper_server.py ca8cf16
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py e752427
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-START/test_before_start.py fb2699a
> ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py 9e8b405
>
> Diff: https://reviews.apache.org/r/34478/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Andrew Onischuk
>
>
Re: Review Request 34478: Move hiveserver2 beeline check to Hive SC
and do mupliple other service-level optimizations
Posted by Vitalyi Brodetskyi <vb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34478/#review84537
-----------------------------------------------------------
Ship it!
Ship It!
- Vitalyi Brodetskyi
On Травень 20, 2015, 4:53 після полудня, Andrew Onischuk wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34478/
> -----------------------------------------------------------
>
> (Updated Травень 20, 2015, 4:53 після полудня)
>
>
> Review request for Ambari and Vitalyi Brodetskyi.
>
>
> Bugs: AMBARI-11271
> https://issues.apache.org/jira/browse/AMBARI-11271
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Currently multiple customers are complaining on high services (re-)start time.
> From what we understood probably more than a minute is bad.
>
> So after discussion with Mahadev it was decided to move out beeline check
> which takes usually ~2 minutes (resulting in hive restarting in ~4minutes)
> from start to service check, which seems a good place for that.
>
> There is a problem due to Hive process asynchroniously starts and can drop
> start without Ambari knowing it.
> Also this was implemented some time ago because of:
>
>
>
> AMBARI-5800
> Race condition when starting all services causing Hive service check to fail
>
>
> But, since now service check does exactly the beeline check with mutiple
> retries this won't happen
>
> Test scenarios included:
> 1\. Restart Hive and rt after it start service check
> 2\. Restart Hive and rt after it restart Webhcat
> 3\. Start service check with Hive Server HA
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs.py 418571a
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_datanode.py d02bb39
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py 453d824
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_nfsgateway.py ac0e24d
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_snamenode.py 78ef977
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/journalnode.py ada05e1
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/params_linux.py 3f1fb0d
> ambari-server/src/main/resources/common-services/HDFS/2.1.0.2.0/package/scripts/utils.py 217f2f0
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/metainfo.xml 32bc0ba
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py 86312d2
> ambari-server/src/main/resources/common-services/HIVE/0.12.0.2.0/package/scripts/service_check.py 04bd3b2
> ambari-server/src/main/resources/common-services/KAFKA/0.8.1.2.2/package/scripts/kafka_broker.py 31d949a
> ambari-server/src/main/resources/common-services/KNOX/0.5.0.2.2/package/scripts/knox_gateway.py 956d09b
> ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/service.py 3d6b8d1
> ambari-server/src/main/resources/common-services/ZOOKEEPER/3.4.5.2.0/package/scripts/zookeeper_server.py d685d1d
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py b58959a
> ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/shared_initialization.py dd93818
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_datanode.py 4eb8178
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_journalnode.py 3fe2acb
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_namenode.py c753c39
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_nfsgateway.py 7255ea5
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_snamenode.py fd665c3
> ambari-server/src/test/python/stacks/2.0.6/HDFS/test_zkfc.py c2fdffa
> ambari-server/src/test/python/stacks/2.0.6/HIVE/test_hive_server.py 90f3e06
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_historyserver.py bc527e7
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_nodemanager.py 005b533
> ambari-server/src/test/python/stacks/2.0.6/YARN/test_resourcemanager.py a2ee27f
> ambari-server/src/test/python/stacks/2.0.6/ZOOKEEPER/test_zookeeper_server.py ca8cf16
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-INSTALL/test_before_install.py e752427
> ambari-server/src/test/python/stacks/2.0.6/hooks/before-START/test_before_start.py fb2699a
> ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py 9e8b405
>
> Diff: https://reviews.apache.org/r/34478/diff/
>
>
> Testing
> -------
>
> mvn clean test
>
>
> Thanks,
>
> Andrew Onischuk
>
>