You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ya...@apache.org on 2021/12/15 02:02:11 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #1557] [DOC] The TTL Of Kyuubi Engines
This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new d805f49 [KYUUBI #1557] [DOC] The TTL Of Kyuubi Engines
d805f49 is described below
commit d805f49fe7d79744ebda39c385f16a29dd45e2e6
Author: Kent Yao <ya...@apache.org>
AuthorDate: Wed Dec 15 10:02:02 2021 +0800
[KYUUBI #1557] [DOC] The TTL Of Kyuubi Engines
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1557 from yaooqinn/ttk.
Closes #1557
8446180a [Kent Yao] [DOC] The TTL Of Kyuubi Engines
Authored-by: Kent Yao <ya...@apache.org>
Signed-off-by: Kent Yao <ya...@apache.org>
---
docs/community/badges.md | 25 --------------
docs/community/index.rst | 2 +-
docs/conf.py | 2 +-
docs/deployment/engine_lifecycle.md | 65 +++++++++++++++++++++++++++++++++++++
docs/deployment/index.rst | 10 +++---
docs/deployment/spark/index.rst | 4 +--
docs/develop_tools/index.rst | 3 +-
docs/imgs/engine_lifecycle.drawio | 1 +
docs/requirements.txt | 2 +-
9 files changed, 76 insertions(+), 38 deletions(-)
diff --git a/docs/community/badges.md b/docs/community/badges.md
deleted file mode 100644
index 15b6954..0000000
--- a/docs/community/badges.md
+++ /dev/null
@@ -1,25 +0,0 @@
-<!--
- - Licensed to the Apache Software Foundation (ASF) under one or more
- - contributor license agreements. See the NOTICE file distributed with
- - this work for additional information regarding copyright ownership.
- - The ASF licenses this file to You under the Apache License, Version 2.0
- - (the "License"); you may not use this file except in compliance with
- - the License. You may obtain a copy of the License at
- -
- - http://www.apache.org/licenses/LICENSE-2.0
- -
- - Unless required by applicable law or agreed to in writing, software
- - distributed under the License is distributed on an "AS IS" BASIS,
- - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- - See the License for the specific language governing permissions and
- - limitations under the License.
- -->
-
-<script async defer src="https://buttons.github.io/buttons.js"></script>
-
-# Community [](https://gitter.im/kyuubi-on-spark/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
-<a class="github-button" href="https://github.com/apache/incubator-kyuubi/subscription" data-color-scheme="no-preference: light; light: dark; dark: light;" data-icon="octicon-eye" data-show-count="true" aria-label="Watch Kyuubi on GitHub">Watch</a>
-<a class="github-button" href="https://github.com/apache/incubator-kyuubi" data-color-scheme="no-preference: light; light: dark; dark: light;" data-icon="octicon-star" data-show-count="true" aria-label="Star Kyuubi on GitHub">Star</a>
-<a class="github-button" href="https://github.com/apache/incubator-kyuubi/fork" data-color-scheme="no-preference: light; light: dark; dark: light;" data-icon="octicon-repo-forked" data-show-count="true" aria-label="Fork Kyuubi on GitHub">Fork</a>
-<a class="github-button" href="https://github.com/apache/incubator-kyuubi/issues" data-color-scheme="no-preference: light; light: dark; dark: light;" data-icon="octicon-issue-opened" data-show-count="true" aria-label="Issue Kyuubi on GitHub">Issue</a>
-<a class="github-button" href="https://github.com/apache/incubator-kyuubi/releases" data-color-scheme="no-preference: light; light: dark; dark: light;" data-icon="octicon-download" aria-label="Download Kyuubi on GitHub">Download</a>
diff --git a/docs/community/index.rst b/docs/community/index.rst
index e1dd2ef..af7af7a 100644
--- a/docs/community/index.rst
+++ b/docs/community/index.rst
@@ -28,4 +28,4 @@ Community
contributions
improvement-proposals
release
- badges
+
diff --git a/docs/conf.py b/docs/conf.py
index f73d5be..db9f032 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -118,7 +118,7 @@ pygments_style = 'sphinx'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['_static']
+html_static_path = []
htmlhelp_basename = 'Recommonmarkdoc'
github_doc_root = 'https://github.com/apache/incubator-kyuubi/tree/master/docs/'
diff --git a/docs/deployment/engine_lifecycle.md b/docs/deployment/engine_lifecycle.md
new file mode 100644
index 0000000..8259755
--- /dev/null
+++ b/docs/deployment/engine_lifecycle.md
@@ -0,0 +1,65 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements. See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License. You may obtain a copy of the License at
+ -
+ - http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+<div align=center>
+
+
+
+</div>
+
+# The TTL Of Kyuubi Engines
+
+For a multi-tenant cluster, its overall resource utilization is a KPI that measures how effectively its resource is utilized against its availability or capacity.
+To better improve the overall resource utilization of the cluster,
+- At cluster layer, we leverage the capabilities, such as [Capacity Scheduler](https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html), of resource scheduling management services, such as YARN and K8s.
+- At application layer, we'd be better to acquire and release resources according to the real workloads.
+
+## The Big Contributors Of Resource Waste
+
+- The time to wait for the resource to be allocated, such as the scheduling delay, the start/stop cost.
+ - A longer time-to-live(TTL) for allocated resources can significantly reduce such time costs within an application.
+
+- The time being idle of the resource.
+ - A shorter time to live for allocated resources can make all resources in rapid turnarounds across applications.
+
+## TTL Types In Kyuubi Engines
+
+<body><div class="mxgraph" style="" data-mxgraph="{"lightbox":false,"nav":true,"edit":"_blank","xml":"<mxfile host=\"Electron\" modified=\"2021-12-10T05:54:16.011Z\" agent=\"5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/15.8.7 Chrome/91.0.4472.164 Electron/13.6.2 Safari/537.36\" etag=\"OsSJijKpSO7wcXva956r\" version=\"15.8.7\" type=\"de [...]
+<script type="text/javascript" src="https://viewer.diagrams.net/js/viewer-static.min.js"></script>
+</body>
+
+- Engine TTL
+ - The TTL of engines describes how long an engine will be cached after all sessions are disconnected.
+- Executor TTL
+ - The TTL of the executor describes how long an executor will be cached when no tasks come.
+
+## Configurations
+
+### Engine TTL
+
+| Key | Default | Meaning | Type | Since |
+|----------------------------------------------|--------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|--------------------------------------|
+| kyuubi\.session\.engine<br>\.check\.interval | <div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT5M</div> | <div style='width: 170pt;word-wrap: break-word;white-space: normal'>The check interval for engine timeout</div> | <div style='width: 30pt'>duration</div> | <div style='width: 20pt'>1.0.0</div> |
+| kyuubi\.session\.engine<br>\.idle\.timeout | <div style='width: 65pt;word-wrap: break-word;white-space: normal'>PT30M</div> | <div style='width: 170pt;word-wrap: break-word;white-space: normal'>engine timeout, the engine will self-terminate when it's not accessed for this duration. 0 or negative means not to self-terminate.</div> | <div style='width: 30pt'>duration</div> | <div style='width: 20pt'>1.0.0</div> |
+
+The above two configurations can be used together to set the TTL of engines.
+These configurations are user-facing and able to use in JDBC urls.
+Note that, for [connection](engine_share_level.html#connection) share level engines that will be terminated at once when the connection is disconnected, these configurations not necessarily work in this case.
+
+### Executor TTL
+
+Executor TTL is part of functionality of Apache Spark's [Dynamic Resource Allocation](./spark/dynamic_allocation.md).
diff --git a/docs/deployment/index.rst b/docs/deployment/index.rst
index 9ff9ee0..e75e9ad 100644
--- a/docs/deployment/index.rst
+++ b/docs/deployment/index.rst
@@ -29,14 +29,9 @@ Basics
:numbered: 3
:glob:
- engine_on_yarn
- engine_on_kubernetes
kyuubi_on_kubernetes
hive_metastore
high_availability_guide
- metrics
- logging
- trouble_shooting
Configurations
--------------
@@ -47,7 +42,6 @@ Configurations
:glob:
settings
- spark/index
Engines
-------
@@ -57,4 +51,8 @@ Engines
:numbered: 3
:glob:
+ engine_on_yarn
+ engine_on_kubernetes
engine_share_level
+ engine_lifecycle
+ spark/index
\ No newline at end of file
diff --git a/docs/deployment/spark/index.rst b/docs/deployment/spark/index.rst
index 772d8b9..e1ae679 100644
--- a/docs/deployment/spark/index.rst
+++ b/docs/deployment/spark/index.rst
@@ -16,8 +16,8 @@
.. image:: ../../imgs/kyuubi_logo.png
:align: center
-The Engine Configuration Guide
-==============================
+The Spark SQL Engine Configuration Guide
+========================================
Kyuubi aims to bring Spark to end-users who need not qualify with Spark or something else related to the big data area.
End-users can write SQL queries through JDBC against Kyuubi and nothing more.
diff --git a/docs/develop_tools/index.rst b/docs/develop_tools/index.rst
index e266068..aa7815c 100644
--- a/docs/develop_tools/index.rst
+++ b/docs/develop_tools/index.rst
@@ -17,7 +17,7 @@
:align: center
Develop Tools
-===========
+=============
.. toctree::
:maxdepth: 2
@@ -28,5 +28,4 @@ Develop Tools
build_document
testing
debugging
- community
developer
diff --git a/docs/imgs/engine_lifecycle.drawio b/docs/imgs/engine_lifecycle.drawio
new file mode 100644
index 0000000..9c1e779
--- /dev/null
+++ b/docs/imgs/engine_lifecycle.drawio
@@ -0,0 +1 @@
+<mxfile host="Electron" modified="2021-12-10T05:53:28.559Z" agent="5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/15.8.7 Chrome/91.0.4472.164 Electron/13.6.2 Safari/537.36" etag="X8U9Z0qSG0z15LNUBbzm" version="15.8.7" type="device"><diagram id="5KKAXer4_n0DyOSvi0H2" name="第 1 页">7X1Xt6TKkfWv0aN64c0j3hSeojAv38Kbwnv49R+c7r7qVrc0mjV3pDtS11qnTpGQSZIRuWNHZAB/gplmF8awL9QuSes/QUCy/wlm/wRBIEBC17+75PhcQkLY54J8LJMvB/2lwC7P9GvNL6VLmaTTdwfOXVfPZf99Ydy1bRrP35WF [...]
\ No newline at end of file
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 6d96074..e19b991 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -17,7 +17,7 @@
# under the License.
#
-sphinx==3.5.4
+sphinx==4.3.1
sphinx_rtd_theme==0.5.2
sphinx-markdown-tables==0.0.15
recommonmark==0.7.1