You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by su...@apache.org on 2019/08/15 23:08:10 UTC
[incubator-druid] branch master updated: Add a warning about
experimental segment locking (#8301)
This is an automated email from the ASF dual-hosted git repository.
surekha pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/master by this push:
new eeae5d9 Add a warning about experimental segment locking (#8301)
eeae5d9 is described below
commit eeae5d936501d5a9f5f7d4a555f363f97c002119
Author: Jihoon Son <ji...@apache.org>
AuthorDate: Thu Aug 15 16:07:59 2019 -0700
Add a warning about experimental segment locking (#8301)
* Add a warning about experimental segment locking
* fix typo
---
docs/content/configuration/index.md | 2 +-
docs/content/ingestion/locking-and-priority.md | 8 ++++++--
2 files changed, 7 insertions(+), 3 deletions(-)
diff --git a/docs/content/configuration/index.md b/docs/content/configuration/index.md
index 134ff18..4812d66 100644
--- a/docs/content/configuration/index.md
+++ b/docs/content/configuration/index.md
@@ -907,7 +907,7 @@ These Overlord static configurations can be defined in the `overlord/runtime.pro
|`druid.indexer.runner.type`|Choices "local" or "remote". Indicates whether tasks should be run locally or in a distributed environment. Experimental task runner "httpRemote" is also available which is same as "remote" but uses HTTP to interact with Middle Manaters instead of Zookeeper.|local|
|`druid.indexer.storage.type`|Choices are "local" or "metadata". Indicates whether incoming tasks should be stored locally (in heap) or in metadata storage. Storing incoming tasks in metadata storage allows for tasks to be resumed if the Overlord should fail.|local|
|`druid.indexer.storage.recentlyFinishedThreshold`|A duration of time to store task results.|PT24H|
-|`druid.indexer.tasklock.forceTimeChunkLock`|If set, all tasks are enforced to use time chunk lock. If not set, each task automatically chooses a lock type to use. This configuration can be overwritten by setting `forceTimeChunkLock` in the [task context](../ingestion/locking-and-priority.html#task-context). See [Task Locking & Priority](../ingestion/locking-and-priority.html) for more details about locking in tasks.|true|
+|`druid.indexer.tasklock.forceTimeChunkLock`|_**Setting this to false is still experimental**_<br/> If set, all tasks are enforced to use time chunk lock. If not set, each task automatically chooses a lock type to use. This configuration can be overwritten by setting `forceTimeChunkLock` in the [task context](../ingestion/locking-and-priority.html#task-context). See [Task Locking & Priority](../ingestion/locking-and-priority.html) for more details about locking in tasks.|true|
|`druid.indexer.queue.maxSize`|Maximum number of active tasks at one time.|Integer.MAX_VALUE|
|`druid.indexer.queue.startDelay`|Sleep this long before starting Overlord queue management. This can be useful to give a cluster time to re-orient itself after e.g. a widespread network issue.|PT1M|
|`druid.indexer.queue.restartDelay`|Sleep this long when Overlord queue management throws an exception before trying again.|PT30S|
diff --git a/docs/content/ingestion/locking-and-priority.md b/docs/content/ingestion/locking-and-priority.md
index f0ba3e1..eaeab99 100644
--- a/docs/content/ingestion/locking-and-priority.md
+++ b/docs/content/ingestion/locking-and-priority.md
@@ -40,7 +40,7 @@ are used to determine the overshadow relation between segments as seen below.
A segment `s1` overshadows another `s2` if
-- `s1` has a higher major version than `s2`.
+- `s1` has a higher major version than `s2`, or
- `s1` has the same major version and a higher minor version than `s2`.
Here are some examples.
@@ -71,6 +71,10 @@ For example, a Kafka indexing task and a compaction task can always write segmen
The reason for this is because a Kafka indexing task always appends new segments, while a compaction task always overwrites existing segments.
The segments created with the segment locking have the _same_ major version and a _higher_ minor version.
+<div class="note caution">
+The segment locking is still experimental. It could have unknown bugs which potentially lead to incorrect query results.
+</div>
+
To enable segment locking, you may need to set `forceTimeChunkLock` to `false` in the [task context](#task-context).
Once `forceTimeChunkLock` is unset, the task will choose a proper lock type to use automatically.
Please note that segment lock is not always available. The most common use case where time chunk lock is enforced is
@@ -126,7 +130,7 @@ The task context is used for various individual task configuration. The followin
|property|default|description|
|--------|-------|-----------|
|taskLockTimeout|300000|task lock timeout in millisecond. For more details, see [Locking](#locking).|
-|forceTimeChunkLock|true|Force to always use time chunk lock. If not set, each task automatically chooses a lock type to use. If this set, it will overwrite the `druid.indexer.tasklock.forceTimeChunkLock` [configuration for the overlord](../configuration/index.html#overlord-operations). See [Locking](#locking) for more details.|
+|forceTimeChunkLock|true|_**Setting this to false is still experimental**_<br/> Force to always use time chunk lock. If not set, each task automatically chooses a lock type to use. If this is set, it will overwrite the `druid.indexer.tasklock.forceTimeChunkLock` [configuration for the overlord](../configuration/index.html#overlord-operations). See [Locking](#locking) for more details.|
|priority|Different based on task types. See [Priority](#priority).|Task priority|
<div class="note caution">
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org