You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "petermarshallio (via GitHub)" <gi...@apache.org> on 2024/09/02 15:16:50 UTC

[PR] Supervisor spec links (druid)

petermarshallio opened a new pull request, #16988:
URL: https://github.com/apache/druid/pull/16988

   Updates to the ingestion spec, supervisor spec, and Kafka and Kinesis specification pages to improve formatting consistency and reinforce their common conceptual structure through heading changes and the addition of links between pages.
   
   ### Description
   
   Made table field entries consistently fixed width.
   Copied down wording from the intro for each section in ingestion-spec to supervisor spec, and then to Kafka and Kinesis.
   Made headings in Kafka / Kinesis docs consistent with ingestion-spec and supervisor docs.
   Changed the "supervisor spec" heading to the JSON object name (i.e. spec).
   Added links up and down the "tree".
   Changed some instances of Kinesis and Kafka to Amazon Kinesis and Apache Kafka.
   
   <hr>
   
   This PR has:
   
   - [x] been self-reviewed.
   - [ ] a release note entry in the PR description.
   - [x] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744332978


##########
docs/ingestion/ingestion-spec.md:
##########
@@ -515,13 +516,13 @@ For information on defining an `indexSpec` in a query context, see [SQL-based in
 
 |Field|Description|Default|
 |-----|-----------|-------|
-|bitmap|Compression format for bitmap indexes. Should be a JSON object with `type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
-|dimensionCompression|Compression format for dimension columns. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
-|stringDictionaryEncoding|Encoding format for string value dictionaries used by STRING and [COMPLEX&lt;json&gt;](../querying/nested-columns.md) columns. To enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`. Optionally, you can specify the `bucketSize` and `formatVersion` properties. See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
-|metricCompression|Compression format for primitive type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more efficient than `uncompressed`, but not supported by older versions of Druid).|`lz4`|
-|longEncoding|Encoding format for long-typed columns. Applies regardless of whether they are dimensions or metrics. Options are `auto` or `longs`. `auto` encodes the values using offset or lookup table depending on column cardinality, and store them with variable size. `longs` stores the value as-is with 8 bytes each.|`longs`|
-|complexMetricCompression|Compression format for complex type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`. Options other than `uncompressed` are not compatible with Druid versions older than 31, and only applies to complex metrics which do not have specialized column formats.|`uncompressed`|
-|jsonCompression|Compression format to use for nested column raw data. Options are `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|`bitmap`|Compression format for bitmap indexes. Should be a JSON object with `type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
+|`dimensionCompression`|Compression format for dimension columns. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|`stringDictionaryEncoding`|Encoding format for string value dictionaries used by STRING and [COMPLEX&lt;json&gt;](../querying/nested-columns.md) columns. To enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`. Optionally, you can specify the `bucketSize` and `formatVersion` properties. See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
+|`metricCompression`|Compression format for primitive type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more efficient than `uncompressed`, but not supported by older versions of Druid).|`lz4`|
+|`longEncoding`|Encoding format for long-typed columns. Applies regardless of whether they are dimensions or metrics. Options are `auto` or `longs`. `auto` encodes the values using offset or lookup table depending on column cardinality, and store them with variable size. `longs` stores the value as-is with 8 bytes each.|`longs`|
+|`complexMetricCompression`|Compression format for complex type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`. Options other than `uncompressed` are not compatible with Druid versions older than 31, and only applies to complex metrics which do not have specialized column formats.|`uncompressed`|
+|`jsonCompression`|Compression format to use for nested column raw data. Options are `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|

Review Comment:
   ```suggestion
   |`jsonCompression`|Compression format to use for nested column raw data. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744317821


##########
docs/ingestion/ingestion-spec.md:
##########
@@ -472,24 +474,23 @@ An example `ioConfig` to read JSON data is:
 }
 ```
 
-For details, see the documentation provided by each [ingestion method](./index.md#ingestion-methods).
-
 ## `tuningConfig`
 
-You specify tuning properties in a `tuningConfig` object, which goes at the top level of an ingestion spec.
-Some properties apply to all [ingestion methods](./index.md#ingestion-methods), but most are specific to each individual ingestion method.
+Specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.
+
+For details, see the documentation for `tuningConfig`  for each [ingestion method](./index.md#ingestion-methods).
 
-The following table lists the common tuning properties shared among ingestion methods:
+The following table lists the common tuning properties shared by all ingestion methods:
 
 |Field|Description|Default|
 |-----|-----------|-------|
-|type|Each ingestion method has its own tuning type code. You must specify the type code that matches your ingestion method. Common options are `index`, `hadoop`, `kafka`, and `kinesis`.||
-|maxRowsInMemory|The maximum number of records to store in memory before persisting to disk. Note that this is the number of rows post-rollup, and so it may not be equal to the number of input records. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first).|`1000000`|
-|maxBytesInMemory|The maximum aggregate size of records, in bytes, to store in the JVM heap before persisting. This is based on a rough estimate of memory usage. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first). `maxBytesInMemory` also includes heap usage of artifacts created from intermediary persists. This means that after every persist, the amount of `maxBytesInMemory` until the next persist will decrease. If the sum of bytes of all intermediary persisted artifacts exceeds `maxBytesInMemory` the task fails.<br /><br />Setting `maxBytesInMemory` to -1 disables this check, meaning Druid will rely entirely on `maxRowsInMemory` to control memory usage. Setting it to zero means the default value will be used (one-sixth of JVM heap size).<br /><br />Note that the estimate of memory usage is designed to be an overestimate, and can be especially high when using complex ingest-time aggregators, including s
 ketches. If this causes your indexing workloads to persist to disk too often, you can set `maxBytesInMemory` to -1 and rely on `maxRowsInMemory` instead.|One-sixth of max JVM heap size|
-|skipBytesInMemoryOverheadCheck|The calculation of maxBytesInMemory takes into account overhead objects created during ingestion and each intermediate persist. Setting this to true can exclude the bytes of these overhead objects from maxBytesInMemory check.|false|
-|indexSpec|Defines segment storage format options to use at indexing time.|See [`indexSpec`](#indexspec) for more information.|
-|indexSpecForIntermediatePersists|Defines segment storage format options to use at indexing time for intermediate persisted temporary segments.|See [`indexSpec`](#indexspec) for more information.|
-|Other properties|Each ingestion method has its own list of additional tuning properties. See the documentation for each method for a full list: [Kafka indexing service](../ingestion/kafka-ingestion.md#tuning-configuration), [Kinesis indexing service](../ingestion/kinesis-ingestion.md#tuning-configuration), [Native batch](native-batch.md#tuningconfig), and [Hadoop-based](hadoop.md#tuningconfig).||
+|`type`|Each ingestion method has its own tuning type code. You must specify the type code that matches your ingestion method. Common options are `index`, `hadoop`, `kafka`, and `kinesis`.||
+|`maxRowsInMemory`|The maximum number of records to store in memory before persisting to disk. Note that this is the number of rows post-rollup, and so it may not be equal to the number of input records. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first).|`1000000`|
+|`maxBytesInMemory`|The maximum aggregate size of records, in bytes, to store in the JVM heap before persisting. This is based on a rough estimate of memory usage. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first). `maxBytesInMemory` also includes heap usage of artifacts created from intermediary persists. This means that after every persist, the amount of `maxBytesInMemory` until the next persist will decrease. If the sum of bytes of all intermediary persisted artifacts exceeds `maxBytesInMemory` the task fails.<br /><br />Setting `maxBytesInMemory` to -1 disables this check, meaning Druid will rely entirely on `maxRowsInMemory` to control memory usage. Setting it to zero means the default value will be used (one-sixth of JVM heap size).<br /><br />Note that the estimate of memory usage is designed to be an overestimate, and can be especially high when using complex ingest-time aggregators, including
  sketches. If this causes your indexing workloads to persist to disk too often, you can set `maxBytesInMemory` to -1 and rely on `maxRowsInMemory` instead.|One-sixth of max JVM heap size|
+|`skipBytesInMemoryOverheadCheck`|The calculation of maxBytesInMemory takes into account overhead objects created during ingestion and each intermediate persist. Setting this to true can exclude the bytes of these overhead objects from maxBytesInMemory check.|false|
+|[`indexSpec`](#indexspec)|Defines segment storage format options to use at indexing time.|See [`indexSpec`](#indexspec) for more information.|
+|[`indexSpecForIntermediatePersists`](#indexspec)|Defines segment storage format options to use at indexing time for intermediate persisted temporary segments.|See [`indexSpec`](#indexspec) for more information.|

Review Comment:
   ```suggestion
   |`indexSpecForIntermediatePersists`|Defines segment storage format options to use at indexing time for intermediate persisted temporary segments.|See [`indexSpec`](#indexspec) for more information.|
   ```
   No need to link it twice. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744346487


##########
docs/ingestion/kinesis-ingestion.md:
##########
@@ -27,19 +27,25 @@ import TabItem from '@theme/TabItem';
   ~ under the License.
   -->
 
-When you enable the Kinesis indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Kinesis indexing tasks. Kinesis indexing tasks read events using the Kinesis shard and sequence number mechanism to guarantee exactly-once ingestion. The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+When you enable the Amazon Kinesis indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Kinesis indexing tasks.
 
-This topic contains configuration information for the Kinesis indexing service supervisor for Apache Druid.
+The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+
+Kinesis indexing tasks read events using the Kinesis shard and sequence number mechanism to guarantee exactly-once ingestion. 
+
+This topic contains configuration information for the Amazon Kinesis indexing service supervisor for Apache Druid.

Review Comment:
   ```suggestion
   This topic contains configuration information for the Kinesis indexing service supervisor for Apache Druid.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744349979


##########
docs/ingestion/kinesis-ingestion.md:
##########
@@ -119,10 +125,13 @@ The following example shows a supervisor spec for a stream with the name `Kinesi
 ```
 </details>
 
-### I/O configuration
+### `ioConfig`
+
+The `ioConfig` influences how data is read from a source system, such as Apache Kafka, Amazon S3, a mounted filesystem, or any other supported source system.
 
-The following table outlines the `ioConfig` configuration properties specific to Kinesis.
-For configuration properties shared across all streaming ingestion methods, refer to [Supervisor I/O configuration](supervisor.md#io-configuration).
+The following table outlines the `ioConfig` configuration properties specific to Amazon Kinesis.
+
+For configuration properties shared across all supervisor-led ingestion, refer to the [supervisor `ioConfig`](supervisor.md#ioconfig).

Review Comment:
   Same comment about "streaming ingestion methods" vs "supervisor-led ingestion".



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744314176


##########
docs/ingestion/ingestion-spec.md:
##########
@@ -472,24 +474,23 @@ An example `ioConfig` to read JSON data is:
 }
 ```
 
-For details, see the documentation provided by each [ingestion method](./index.md#ingestion-methods).
-
 ## `tuningConfig`
 
-You specify tuning properties in a `tuningConfig` object, which goes at the top level of an ingestion spec.
-Some properties apply to all [ingestion methods](./index.md#ingestion-methods), but most are specific to each individual ingestion method.
+Specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.
+
+For details, see the documentation for `tuningConfig`  for each [ingestion method](./index.md#ingestion-methods).

Review Comment:
   ```suggestion
   For details, see the documentation for `tuningConfig` for each [ingestion method](./index.md#ingestion-methods).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744329420


##########
docs/ingestion/ingestion-spec.md:
##########
@@ -515,13 +516,13 @@ For information on defining an `indexSpec` in a query context, see [SQL-based in
 
 |Field|Description|Default|
 |-----|-----------|-------|
-|bitmap|Compression format for bitmap indexes. Should be a JSON object with `type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
-|dimensionCompression|Compression format for dimension columns. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
-|stringDictionaryEncoding|Encoding format for string value dictionaries used by STRING and [COMPLEX&lt;json&gt;](../querying/nested-columns.md) columns. To enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`. Optionally, you can specify the `bucketSize` and `formatVersion` properties. See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
-|metricCompression|Compression format for primitive type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more efficient than `uncompressed`, but not supported by older versions of Druid).|`lz4`|
-|longEncoding|Encoding format for long-typed columns. Applies regardless of whether they are dimensions or metrics. Options are `auto` or `longs`. `auto` encodes the values using offset or lookup table depending on column cardinality, and store them with variable size. `longs` stores the value as-is with 8 bytes each.|`longs`|
-|complexMetricCompression|Compression format for complex type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`. Options other than `uncompressed` are not compatible with Druid versions older than 31, and only applies to complex metrics which do not have specialized column formats.|`uncompressed`|
-|jsonCompression|Compression format to use for nested column raw data. Options are `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|`bitmap`|Compression format for bitmap indexes. Should be a JSON object with `type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
+|`dimensionCompression`|Compression format for dimension columns. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|`stringDictionaryEncoding`|Encoding format for string value dictionaries used by STRING and [COMPLEX&lt;json&gt;](../querying/nested-columns.md) columns. To enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`. Optionally, you can specify the `bucketSize` and `formatVersion` properties. See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
+|`metricCompression`|Compression format for primitive type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more efficient than `uncompressed`, but not supported by older versions of Druid).|`lz4`|

Review Comment:
   ```suggestion
   |`metricCompression`|Compression format for primitive type metric columns. One of `lz4`, `lzf`, `zstd`, `uncompressed`, or `none`. `none` is more efficient than `uncompressed`, but it's not supported by older versions of Druid.|`lz4`|
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #16988: Docs - supervisor spec links
URL: https://github.com/apache/druid/pull/16988


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744347746


##########
docs/ingestion/kinesis-ingestion.md:
##########
@@ -27,19 +27,25 @@ import TabItem from '@theme/TabItem';
   ~ under the License.
   -->
 
-When you enable the Kinesis indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Kinesis indexing tasks. Kinesis indexing tasks read events using the Kinesis shard and sequence number mechanism to guarantee exactly-once ingestion. The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+When you enable the Amazon Kinesis indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Kinesis indexing tasks.
 
-This topic contains configuration information for the Kinesis indexing service supervisor for Apache Druid.
+The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+
+Kinesis indexing tasks read events using the Kinesis shard and sequence number mechanism to guarantee exactly-once ingestion. 
+
+This topic contains configuration information for the Amazon Kinesis indexing service supervisor for Apache Druid.
 
 ## Setup
 
 To use the Kinesis indexing service, you must first load the `druid-kinesis-indexing-service` core extension on both the Overlord and the MiddleManager. See [Loading extensions](../configuration/extensions.md#loading-extensions) for more information.
 
 Review [Known issues](#known-issues) before deploying the `druid-kinesis-indexing-service` extension to production.
 
-## Supervisor spec configuration
+## `spec`
 
-This section outlines the configuration properties that are specific to the Amazon Kinesis streaming ingestion method. For configuration properties shared across all streaming ingestion methods supported by Druid, see [Supervisor spec](supervisor.md#supervisor-spec).
+This section outlines the configuration properties that are specific to supervisor-led streaming ingestion for Amazon Kinesis.

Review Comment:
   I think the original version (without "supervisor-led streaming") was easier to read and comprehend. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744352133


##########
docs/ingestion/supervisor.md:
##########
@@ -24,30 +24,40 @@ sidebar_label: Supervisor
   -->
 
 A supervisor manages streaming ingestion from external streaming sources into Apache Druid.
-Supervisors oversee the state of indexing tasks to coordinate handoffs, manage failures, and ensure that the scalability and replication requirements are maintained.
 
-This topic uses the Apache Kafka term offset to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is sequence number.
+Supervisors oversee the state of indexing tasks to coordinate [handoffs](../design/storage#indexing-and-handoff), manage failures, and ensure that the scalability and replication requirements are maintained.
 
-## Supervisor spec
+:::info
+This topic uses the Apache Kafka term _offset_ to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is _sequence number_.
+:::
 
-Druid uses a JSON specification, often referred to as the supervisor spec, to define streaming ingestion tasks.
-The supervisor spec specifies how Druid should consume, process, and index streaming data.
+Druid uses a JSON document, often referred to as a supervisor spec, to specify how Druid should consume, process, and index streaming data. A supervisor spec is a type of [native ingestion spec](../ingestion/ingestion-spec.md).

Review Comment:
   Why did you choose to replace "JSON specification" with "JSON document"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744316879


##########
docs/ingestion/ingestion-spec.md:
##########
@@ -472,24 +474,23 @@ An example `ioConfig` to read JSON data is:
 }
 ```
 
-For details, see the documentation provided by each [ingestion method](./index.md#ingestion-methods).
-
 ## `tuningConfig`
 
-You specify tuning properties in a `tuningConfig` object, which goes at the top level of an ingestion spec.
-Some properties apply to all [ingestion methods](./index.md#ingestion-methods), but most are specific to each individual ingestion method.
+Specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.
+
+For details, see the documentation for `tuningConfig`  for each [ingestion method](./index.md#ingestion-methods).
 
-The following table lists the common tuning properties shared among ingestion methods:
+The following table lists the common tuning properties shared by all ingestion methods:
 
 |Field|Description|Default|
 |-----|-----------|-------|
-|type|Each ingestion method has its own tuning type code. You must specify the type code that matches your ingestion method. Common options are `index`, `hadoop`, `kafka`, and `kinesis`.||
-|maxRowsInMemory|The maximum number of records to store in memory before persisting to disk. Note that this is the number of rows post-rollup, and so it may not be equal to the number of input records. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first).|`1000000`|
-|maxBytesInMemory|The maximum aggregate size of records, in bytes, to store in the JVM heap before persisting. This is based on a rough estimate of memory usage. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first). `maxBytesInMemory` also includes heap usage of artifacts created from intermediary persists. This means that after every persist, the amount of `maxBytesInMemory` until the next persist will decrease. If the sum of bytes of all intermediary persisted artifacts exceeds `maxBytesInMemory` the task fails.<br /><br />Setting `maxBytesInMemory` to -1 disables this check, meaning Druid will rely entirely on `maxRowsInMemory` to control memory usage. Setting it to zero means the default value will be used (one-sixth of JVM heap size).<br /><br />Note that the estimate of memory usage is designed to be an overestimate, and can be especially high when using complex ingest-time aggregators, including s
 ketches. If this causes your indexing workloads to persist to disk too often, you can set `maxBytesInMemory` to -1 and rely on `maxRowsInMemory` instead.|One-sixth of max JVM heap size|
-|skipBytesInMemoryOverheadCheck|The calculation of maxBytesInMemory takes into account overhead objects created during ingestion and each intermediate persist. Setting this to true can exclude the bytes of these overhead objects from maxBytesInMemory check.|false|
-|indexSpec|Defines segment storage format options to use at indexing time.|See [`indexSpec`](#indexspec) for more information.|
-|indexSpecForIntermediatePersists|Defines segment storage format options to use at indexing time for intermediate persisted temporary segments.|See [`indexSpec`](#indexspec) for more information.|
-|Other properties|Each ingestion method has its own list of additional tuning properties. See the documentation for each method for a full list: [Kafka indexing service](../ingestion/kafka-ingestion.md#tuning-configuration), [Kinesis indexing service](../ingestion/kinesis-ingestion.md#tuning-configuration), [Native batch](native-batch.md#tuningconfig), and [Hadoop-based](hadoop.md#tuningconfig).||
+|`type`|Each ingestion method has its own tuning type code. You must specify the type code that matches your ingestion method. Common options are `index`, `hadoop`, `kafka`, and `kinesis`.||
+|`maxRowsInMemory`|The maximum number of records to store in memory before persisting to disk. Note that this is the number of rows post-rollup, and so it may not be equal to the number of input records. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first).|`1000000`|
+|`maxBytesInMemory`|The maximum aggregate size of records, in bytes, to store in the JVM heap before persisting. This is based on a rough estimate of memory usage. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first). `maxBytesInMemory` also includes heap usage of artifacts created from intermediary persists. This means that after every persist, the amount of `maxBytesInMemory` until the next persist will decrease. If the sum of bytes of all intermediary persisted artifacts exceeds `maxBytesInMemory` the task fails.<br /><br />Setting `maxBytesInMemory` to -1 disables this check, meaning Druid will rely entirely on `maxRowsInMemory` to control memory usage. Setting it to zero means the default value will be used (one-sixth of JVM heap size).<br /><br />Note that the estimate of memory usage is designed to be an overestimate, and can be especially high when using complex ingest-time aggregators, including
  sketches. If this causes your indexing workloads to persist to disk too often, you can set `maxBytesInMemory` to -1 and rely on `maxRowsInMemory` instead.|One-sixth of max JVM heap size|
+|`skipBytesInMemoryOverheadCheck`|The calculation of maxBytesInMemory takes into account overhead objects created during ingestion and each intermediate persist. Setting this to true can exclude the bytes of these overhead objects from maxBytesInMemory check.|false|

Review Comment:
   ```suggestion
   |`skipBytesInMemoryOverheadCheck`|The calculation of `maxBytesInMemory` takes into account overhead objects created during ingestion and each intermediate persist. Setting this to true can exclude the bytes of these overhead objects from `maxBytesInMemory` check.|false|
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744333736


##########
docs/ingestion/kafka-ingestion.md:
##########
@@ -29,18 +29,23 @@ To use the Kafka indexing service, you must be on Apache Kafka version 0.11.x or
 If you are using an older version, refer to the [Apache Kafka upgrade guide](https://kafka.apache.org/documentation/#upgrade).
 :::
 
-When you enable the Kafka indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Kafka indexing tasks.
-Kafka indexing tasks read events using Kafka partition and offset mechanism to guarantee exactly-once ingestion. The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+When you enable the Apache Kafka indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Apache Kafka indexing tasks.

Review Comment:
   ```suggestion
   When you enable the Apache Kafka indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Kafka indexing tasks.
   ```
   Don't need to mention Apache again. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744334809


##########
docs/ingestion/kafka-ingestion.md:
##########
@@ -29,18 +29,23 @@ To use the Kafka indexing service, you must be on Apache Kafka version 0.11.x or
 If you are using an older version, refer to the [Apache Kafka upgrade guide](https://kafka.apache.org/documentation/#upgrade).
 :::
 
-When you enable the Kafka indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Kafka indexing tasks.
-Kafka indexing tasks read events using Kafka partition and offset mechanism to guarantee exactly-once ingestion. The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+When you enable the Apache Kafka indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Apache Kafka indexing tasks.
 
-This topic contains configuration information for the Kafka indexing service supervisor for Apache Druid.
+The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+
+Kafka indexing tasks read events using Kafka partition and offset mechanism to guarantee exactly-once ingestion. 
+
+This topic contains configuration information for the Apache Kafka indexing service supervisor for Apache Druid.

Review Comment:
   ```suggestion
   This topic contains configuration information for the Kafka indexing service supervisor for Apache Druid.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744335167


##########
docs/ingestion/kafka-ingestion.md:
##########
@@ -29,18 +29,23 @@ To use the Kafka indexing service, you must be on Apache Kafka version 0.11.x or
 If you are using an older version, refer to the [Apache Kafka upgrade guide](https://kafka.apache.org/documentation/#upgrade).
 :::
 
-When you enable the Kafka indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Kafka indexing tasks.
-Kafka indexing tasks read events using Kafka partition and offset mechanism to guarantee exactly-once ingestion. The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+When you enable the Apache Kafka indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Apache Kafka indexing tasks.
 
-This topic contains configuration information for the Kafka indexing service supervisor for Apache Druid.
+The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+
+Kafka indexing tasks read events using Kafka partition and offset mechanism to guarantee exactly-once ingestion. 
+
+This topic contains configuration information for the Apache Kafka indexing service supervisor for Apache Druid.
 
 ## Setup
 
 To use the Kafka indexing service, you must first load the `druid-kafka-indexing-service` extension on both the Overlord and the MiddleManager. See [Loading extensions](../configuration/extensions.md) for more information.
 
-## Supervisor spec configuration
+## `spec`
 
-This section outlines the configuration properties that are specific to the Apache Kafka streaming ingestion method. For configuration properties shared across all streaming ingestion methods supported by Druid, see [Supervisor spec](supervisor.md#supervisor-spec).
+This section outlines the configuration properties that are specific to supervisor-led streaming ingestion for Apache Kafka.

Review Comment:
   ```suggestion
   This section outlines the configuration properties that are specific to supervisor-led streaming ingestion for Kafka.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744349231


##########
docs/ingestion/kinesis-ingestion.md:
##########
@@ -119,10 +125,13 @@ The following example shows a supervisor spec for a stream with the name `Kinesi
 ```
 </details>
 
-### I/O configuration
+### `ioConfig`
+
+The `ioConfig` influences how data is read from a source system, such as Apache Kafka, Amazon S3, a mounted filesystem, or any other supported source system.
 
-The following table outlines the `ioConfig` configuration properties specific to Kinesis.
-For configuration properties shared across all streaming ingestion methods, refer to [Supervisor I/O configuration](supervisor.md#io-configuration).
+The following table outlines the `ioConfig` configuration properties specific to Amazon Kinesis.

Review Comment:
   ```suggestion
   The following table outlines the `ioConfig` configuration properties specific to Kinesis.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744371331


##########
docs/ingestion/supervisor.md:
##########
@@ -24,30 +24,40 @@ sidebar_label: Supervisor
   -->
 
 A supervisor manages streaming ingestion from external streaming sources into Apache Druid.
-Supervisors oversee the state of indexing tasks to coordinate handoffs, manage failures, and ensure that the scalability and replication requirements are maintained.
 
-This topic uses the Apache Kafka term offset to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is sequence number.
+Supervisors oversee the state of indexing tasks to coordinate [handoffs](../design/storage#indexing-and-handoff), manage failures, and ensure that the scalability and replication requirements are maintained.
 
-## Supervisor spec
+:::info
+This topic uses the Apache Kafka term _offset_ to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is _sequence number_.
+:::
 
-Druid uses a JSON specification, often referred to as the supervisor spec, to define streaming ingestion tasks.
-The supervisor spec specifies how Druid should consume, process, and index streaming data.
+Druid uses a JSON document, often referred to as a supervisor spec, to specify how Druid should consume, process, and index streaming data. A supervisor spec is a type of [native ingestion spec](../ingestion/ingestion-spec.md).
 
 The following table outlines the high-level configuration options for a supervisor spec:
 
 |Property|Type|Description|Required|
 |--------|----|-----------|--------|
-|`type`|String|The supervisor type. One of `kafka`or `kinesis`.|Yes|
-|`spec`|Object|The container object for the supervisor configuration.|Yes|
-|`spec.dataSchema`|Object|The schema for the indexing task to use during ingestion. See [`dataSchema`](../ingestion/ingestion-spec.md#dataschema) for more information.|Yes|
-|`spec.ioConfig`|Object|The I/O configuration object to define the connection and I/O-related settings for the supervisor and indexing tasks.|Yes|
-|`spec.tuningConfig`|Object|The tuning configuration object to define performance-related settings for the supervisor and indexing tasks.|No|
+|`type`|String|The type of ingestion to supervise, one of [`kafka`](kafka-ingestion.md) or [`kinesis`](kinesis-ingestion.md).|Yes|
+|[`spec`](#spec)|Object|The container object for the supervisor configuration.|Yes|
 |`suspended`|Boolean|Puts the supervisor in a suspended state|No|
 
-### I/O configuration
+## `spec`

Review Comment:
   Hm, I am not sure if presenting the information this way makes it easer to comprehend. Personally, I think it's beneficial to include `dataSchema`, `ioConfig`, and `tuningConfig` in the high-level configurations table. Especially, since both `dataSchema` and `ioConfig` are required.  If you prefer to move them to a separate section, I think we should keep the existing descriptions and update the style of the bullet points. For example:
   
   ```suggestion
   - `dataSchema`: Schema for the indexing task to use during ingestion.
   - `ioConfig`: I/O configuration object to define the connection and I/O-related settings for the supervisor and indexing tasks.
   - `tuningConfig`: Tuning configuration object to define performance-related settings for the supervisor and indexing tasks.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16988:
URL: https://github.com/apache/druid/pull/16988#issuecomment-2635428770

   This pull request/issue has been closed due to lack of activity. If you think that
   is incorrect, or the pull request requires review, you can revive the PR at any time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744315542


##########
docs/ingestion/ingestion-spec.md:
##########
@@ -472,24 +474,23 @@ An example `ioConfig` to read JSON data is:
 }
 ```
 
-For details, see the documentation provided by each [ingestion method](./index.md#ingestion-methods).
-
 ## `tuningConfig`
 
-You specify tuning properties in a `tuningConfig` object, which goes at the top level of an ingestion spec.
-Some properties apply to all [ingestion methods](./index.md#ingestion-methods), but most are specific to each individual ingestion method.
+Specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.

Review Comment:
   ```suggestion
   You specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.
   ```
   I think it reads better with "you" since it's a conceptual topic and not a tutorial. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744348951


##########
docs/ingestion/kinesis-ingestion.md:
##########
@@ -119,10 +125,13 @@ The following example shows a supervisor spec for a stream with the name `Kinesi
 ```
 </details>
 
-### I/O configuration
+### `ioConfig`
+
+The `ioConfig` influences how data is read from a source system, such as Apache Kafka, Amazon S3, a mounted filesystem, or any other supported source system.

Review Comment:
   ```suggestion
   The `ioConfig` influences how data is read from a source system, such as Amazon Kinesis, Amazon S3, a mounted filesystem, or any other supported source system.
   ```
   This this topic is specific to Kinesis, we should use it in the examples list instead of Kafka.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744350328


##########
docs/ingestion/kinesis-ingestion.md:
##########
@@ -148,10 +157,13 @@ The Kinesis indexing service supports the following values for `inputFormat`:
 
 You can use `parser` to read [`thrift`](../development/extensions-contrib/thrift.md) formats.
 
-### Tuning configuration
+### `tuningConfig`
+
+Specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.

Review Comment:
   ```suggestion
   You specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.
   ```



##########
docs/ingestion/kinesis-ingestion.md:
##########
@@ -148,10 +157,13 @@ The Kinesis indexing service supports the following values for `inputFormat`:
 
 You can use `parser` to read [`thrift`](../development/extensions-contrib/thrift.md) formats.
 
-### Tuning configuration
+### `tuningConfig`
+
+Specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.
+
+The following table outlines the `tuningConfig` configuration properties specific to Amazon Kinesis.

Review Comment:
   ```suggestion
   The following table outlines the `tuningConfig` configuration properties specific to Kinesis.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744363874


##########
docs/ingestion/supervisor.md:
##########
@@ -24,30 +24,40 @@ sidebar_label: Supervisor
   -->
 
 A supervisor manages streaming ingestion from external streaming sources into Apache Druid.
-Supervisors oversee the state of indexing tasks to coordinate handoffs, manage failures, and ensure that the scalability and replication requirements are maintained.
 
-This topic uses the Apache Kafka term offset to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is sequence number.
+Supervisors oversee the state of indexing tasks to coordinate [handoffs](../design/storage#indexing-and-handoff), manage failures, and ensure that the scalability and replication requirements are maintained.
 
-## Supervisor spec
+:::info
+This topic uses the Apache Kafka term _offset_ to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is _sequence number_.
+:::
 
-Druid uses a JSON specification, often referred to as the supervisor spec, to define streaming ingestion tasks.
-The supervisor spec specifies how Druid should consume, process, and index streaming data.
+Druid uses a JSON document, often referred to as a supervisor spec, to specify how Druid should consume, process, and index streaming data. A supervisor spec is a type of [native ingestion spec](../ingestion/ingestion-spec.md).
 
 The following table outlines the high-level configuration options for a supervisor spec:
 
 |Property|Type|Description|Required|
 |--------|----|-----------|--------|
-|`type`|String|The supervisor type. One of `kafka`or `kinesis`.|Yes|
-|`spec`|Object|The container object for the supervisor configuration.|Yes|
-|`spec.dataSchema`|Object|The schema for the indexing task to use during ingestion. See [`dataSchema`](../ingestion/ingestion-spec.md#dataschema) for more information.|Yes|
-|`spec.ioConfig`|Object|The I/O configuration object to define the connection and I/O-related settings for the supervisor and indexing tasks.|Yes|
-|`spec.tuningConfig`|Object|The tuning configuration object to define performance-related settings for the supervisor and indexing tasks.|No|
+|`type`|String|The type of ingestion to supervise, one of [`kafka`](kafka-ingestion.md) or [`kinesis`](kinesis-ingestion.md).|Yes|
+|[`spec`](#spec)|Object|The container object for the supervisor configuration.|Yes|
 |`suspended`|Boolean|Puts the supervisor in a suspended state|No|
 
-### I/O configuration
+## `spec`
+
+The `spec` section consists of three components:
+
+- `dataSchema`. The schema for the indexing task to use during ingestion.
+- [`ioConfig`](#ioconfig), which tells Druid how to connect to the [source system](./index.md#ingestion-methods) and how to parse data.
+- [`tuningConfig`](#tuningconfig), which controls performance-related settings for the supervisor and indexing tasks.
+
+For configuration properties shared across all ingestion methods supported by Druid, including the `dataSchema`, see [native ingestion specs](../ingestion/ingestion-spec.md).
+
+### `ioConfig`
 
-The following table outlines the `ioConfig` configuration properties that apply to both Apache Kafka and Amazon Kinesis ingestion methods.
-For configuration properties specific to Kafka and Kinesis, see [Kafka I/O configuration](kafka-ingestion.md#io-configuration) and [Kinesis I/O configuration](kinesis-ingestion.md#io-configuration) respectively.
+The `ioConfig` influences how data is read from a source system. See native ingestion specification documentation on the [`ioConfig`](../ingestion/ingestion-spec.md#ioconfig) for more information.
+
+The following table outlines the configuration properties that apply to supervisor-led ingestion methods, including Apache Kafka and Amazon Kinesis.

Review Comment:
   I think the original version was easier to read and comprehend.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744353498


##########
docs/ingestion/supervisor.md:
##########
@@ -24,30 +24,40 @@ sidebar_label: Supervisor
   -->
 
 A supervisor manages streaming ingestion from external streaming sources into Apache Druid.
-Supervisors oversee the state of indexing tasks to coordinate handoffs, manage failures, and ensure that the scalability and replication requirements are maintained.
 
-This topic uses the Apache Kafka term offset to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is sequence number.
+Supervisors oversee the state of indexing tasks to coordinate [handoffs](../design/storage#indexing-and-handoff), manage failures, and ensure that the scalability and replication requirements are maintained.
 
-## Supervisor spec
+:::info
+This topic uses the Apache Kafka term _offset_ to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is _sequence number_.
+:::
 
-Druid uses a JSON specification, often referred to as the supervisor spec, to define streaming ingestion tasks.
-The supervisor spec specifies how Druid should consume, process, and index streaming data.
+Druid uses a JSON document, often referred to as a supervisor spec, to specify how Druid should consume, process, and index streaming data. A supervisor spec is a type of [native ingestion spec](../ingestion/ingestion-spec.md).
 
 The following table outlines the high-level configuration options for a supervisor spec:
 
 |Property|Type|Description|Required|
 |--------|----|-----------|--------|
-|`type`|String|The supervisor type. One of `kafka`or `kinesis`.|Yes|
-|`spec`|Object|The container object for the supervisor configuration.|Yes|
-|`spec.dataSchema`|Object|The schema for the indexing task to use during ingestion. See [`dataSchema`](../ingestion/ingestion-spec.md#dataschema) for more information.|Yes|
-|`spec.ioConfig`|Object|The I/O configuration object to define the connection and I/O-related settings for the supervisor and indexing tasks.|Yes|
-|`spec.tuningConfig`|Object|The tuning configuration object to define performance-related settings for the supervisor and indexing tasks.|No|
+|`type`|String|The type of ingestion to supervise, one of [`kafka`](kafka-ingestion.md) or [`kinesis`](kinesis-ingestion.md).|Yes|

Review Comment:
   ```suggestion
   |`type`|String|The type of ingestion to supervise. One of [`kafka`](kafka-ingestion.md) or [`kinesis`](kinesis-ingestion.md).|Yes|
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744363307


##########
docs/ingestion/supervisor.md:
##########
@@ -182,12 +192,16 @@ The following example shows a supervisor spec with `lagBased` autoscaler:
 ```
 </details>
 
-### Tuning configuration
+### `tuningConfig`
+
+The `tuningConfig` controls various tuning parameters specific to each ingestion method. See native ingestion specification documentation on the [`tuningConfig`](../ingestion/ingestion-spec.md#tuningconfig) for more information.
 
 The `tuningConfig` object is optional. If you don't specify the `tuningConfig` object, Druid uses the default configuration settings.
 
-The following table outlines the `tuningConfig` configuration properties that apply to both Kafka and Kinesis ingestion methods.
-For configuration properties specific to Kafka and Kinesis, see [Kafka tuning configuration](kafka-ingestion.md#tuning-configuration) and [Kinesis tuning configuration](kinesis-ingestion.md#tuning-configuration) respectively.
+For configuration properties specific to Kafka and Kinesis, see [Kafka  `tuningConfig`](kafka-ingestion.md#tuningconfig) and [Kinesis `tuningConfig`](kinesis-ingestion.md#tuningconfig) respectively.
+
+The following table outlines the tuning properties that apply to supervisor-led ingestion methods, including Apache Kafka and Amazon Kinesis.

Review Comment:
   I think the original version was easier to read and comprehend. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744341891


##########
docs/ingestion/kafka-ingestion.md:
##########
@@ -113,10 +118,13 @@ The following example shows a supervisor spec for the Kafka indexing service:
 
 </details>
 
-### I/O configuration
+### `ioConfig`
+
+The `ioConfig` influences how data is read from a source system, such as Apache Kafka, Amazon S3, a mounted filesystem, or any other supported source system.
 
-The following table outlines the `ioConfig` configuration properties specific to Kafka.
-For configuration properties shared across all streaming ingestion methods, refer to [Supervisor I/O configuration](supervisor.md#io-configuration).
+The following table outlines the `ioConfig` configuration properties specific to Apache Kafka.
+
+For configuration properties shared across all supervisor-led ingestion, refer to [supervisor `ioConfig`](supervisor.md#ioconfig).

Review Comment:
   I wonder if "streaming ingestion methods" is more suitable (easier to comprehend) than "supervisor-led ingestion".



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16988:
URL: https://github.com/apache/druid/pull/16988#issuecomment-2574159364

   This pull request has been marked as stale due to 60 days of inactivity.
   It will be closed in 4 weeks if no further activity occurs. If you think
   that's incorrect or this pull request should instead be reviewed, please simply
   write any comment. Even if closed, you can still revive the PR at any time or
   discuss it on the dev@druid.apache.org list.
   Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744332472


##########
docs/ingestion/ingestion-spec.md:
##########
@@ -515,13 +516,13 @@ For information on defining an `indexSpec` in a query context, see [SQL-based in
 
 |Field|Description|Default|
 |-----|-----------|-------|
-|bitmap|Compression format for bitmap indexes. Should be a JSON object with `type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
-|dimensionCompression|Compression format for dimension columns. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
-|stringDictionaryEncoding|Encoding format for string value dictionaries used by STRING and [COMPLEX&lt;json&gt;](../querying/nested-columns.md) columns. To enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`. Optionally, you can specify the `bucketSize` and `formatVersion` properties. See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
-|metricCompression|Compression format for primitive type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more efficient than `uncompressed`, but not supported by older versions of Druid).|`lz4`|
-|longEncoding|Encoding format for long-typed columns. Applies regardless of whether they are dimensions or metrics. Options are `auto` or `longs`. `auto` encodes the values using offset or lookup table depending on column cardinality, and store them with variable size. `longs` stores the value as-is with 8 bytes each.|`longs`|
-|complexMetricCompression|Compression format for complex type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`. Options other than `uncompressed` are not compatible with Druid versions older than 31, and only applies to complex metrics which do not have specialized column formats.|`uncompressed`|
-|jsonCompression|Compression format to use for nested column raw data. Options are `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|`bitmap`|Compression format for bitmap indexes. Should be a JSON object with `type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
+|`dimensionCompression`|Compression format for dimension columns. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|`stringDictionaryEncoding`|Encoding format for string value dictionaries used by STRING and [COMPLEX&lt;json&gt;](../querying/nested-columns.md) columns. To enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`. Optionally, you can specify the `bucketSize` and `formatVersion` properties. See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
+|`metricCompression`|Compression format for primitive type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more efficient than `uncompressed`, but not supported by older versions of Druid).|`lz4`|
+|`longEncoding`|Encoding format for long-typed columns. Applies regardless of whether they are dimensions or metrics. Options are `auto` or `longs`. `auto` encodes the values using offset or lookup table depending on column cardinality, and store them with variable size. `longs` stores the value as-is with 8 bytes each.|`longs`|
+|`complexMetricCompression`|Compression format for complex type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`. Options other than `uncompressed` are not compatible with Druid versions older than 31, and only applies to complex metrics which do not have specialized column formats.|`uncompressed`|

Review Comment:
   ```suggestion
   |`complexMetricCompression`|Compression format for complex type metric columns. One of `lz4`, `lzf`, `zstd`, `uncompressed`. Options other than `uncompressed` are not compatible with Druid versions older than 31 and only apply to complex metrics which do not have specialized column formats.|`uncompressed`|
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744331648


##########
docs/ingestion/ingestion-spec.md:
##########
@@ -515,13 +516,13 @@ For information on defining an `indexSpec` in a query context, see [SQL-based in
 
 |Field|Description|Default|
 |-----|-----------|-------|
-|bitmap|Compression format for bitmap indexes. Should be a JSON object with `type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
-|dimensionCompression|Compression format for dimension columns. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
-|stringDictionaryEncoding|Encoding format for string value dictionaries used by STRING and [COMPLEX&lt;json&gt;](../querying/nested-columns.md) columns. To enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`. Optionally, you can specify the `bucketSize` and `formatVersion` properties. See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
-|metricCompression|Compression format for primitive type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more efficient than `uncompressed`, but not supported by older versions of Druid).|`lz4`|
-|longEncoding|Encoding format for long-typed columns. Applies regardless of whether they are dimensions or metrics. Options are `auto` or `longs`. `auto` encodes the values using offset or lookup table depending on column cardinality, and store them with variable size. `longs` stores the value as-is with 8 bytes each.|`longs`|
-|complexMetricCompression|Compression format for complex type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`. Options other than `uncompressed` are not compatible with Druid versions older than 31, and only applies to complex metrics which do not have specialized column formats.|`uncompressed`|
-|jsonCompression|Compression format to use for nested column raw data. Options are `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|`bitmap`|Compression format for bitmap indexes. Should be a JSON object with `type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
+|`dimensionCompression`|Compression format for dimension columns. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|`stringDictionaryEncoding`|Encoding format for string value dictionaries used by STRING and [COMPLEX&lt;json&gt;](../querying/nested-columns.md) columns. To enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`. Optionally, you can specify the `bucketSize` and `formatVersion` properties. See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
+|`metricCompression`|Compression format for primitive type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more efficient than `uncompressed`, but not supported by older versions of Druid).|`lz4`|
+|`longEncoding`|Encoding format for long-typed columns. Applies regardless of whether they are dimensions or metrics. Options are `auto` or `longs`. `auto` encodes the values using offset or lookup table depending on column cardinality, and store them with variable size. `longs` stores the value as-is with 8 bytes each.|`longs`|

Review Comment:
   ```suggestion
   |`longEncoding`|Encoding format for long-typed columns. Applies regardless of whether they are dimensions or metrics. Options are `auto` or `longs`. `auto` encodes the values using offset or lookup table depending on column cardinality and stores them with variable size. `longs` stores the value as-is with 8 bytes each.|`longs`|
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744338765


##########
docs/ingestion/kafka-ingestion.md:
##########
@@ -113,10 +118,13 @@ The following example shows a supervisor spec for the Kafka indexing service:
 
 </details>
 
-### I/O configuration
+### `ioConfig`
+
+The `ioConfig` influences how data is read from a source system, such as Apache Kafka, Amazon S3, a mounted filesystem, or any other supported source system.
 
-The following table outlines the `ioConfig` configuration properties specific to Kafka.
-For configuration properties shared across all streaming ingestion methods, refer to [Supervisor I/O configuration](supervisor.md#io-configuration).
+The following table outlines the `ioConfig` configuration properties specific to Apache Kafka.

Review Comment:
   ```suggestion
   The following table outlines the `ioConfig` configuration properties specific to Kafka.
   ```
   We don't have to mention Apache every time we refer to Apache Kafka. Once at the beginning of the topic is sufficient. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744342885


##########
docs/ingestion/kafka-ingestion.md:
##########
@@ -413,10 +421,13 @@ This query returns:
 |------------------|-----------|-----------------|-------------|
 |`development`|`wiki-edit`|`1680795276351`|`wiki-edits`|
 
-### Tuning configuration
+### `tuningConfig`
+
+Specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.
+
+The following table outlines the `tuningConfig` configuration properties specific to Apache Kafka.
 
-The following table outlines the `tuningConfig` configuration properties specific to Kafka.
-For configuration properties shared across all streaming ingestion methods, refer to [Supervisor tuning configuration](supervisor.md#tuning-configuration).
+For configuration properties shared across all supervisor-led ingestion, refer to [Supervisor `tuningConfig`](supervisor.md#tuningconfig).

Review Comment:
   Why did you choose to replace "steaming ingestion methods" with "supervisor-led ingestion"?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744344103


##########
docs/ingestion/kafka-ingestion.md:
##########
@@ -413,10 +421,13 @@ This query returns:
 |------------------|-----------|-----------------|-------------|
 |`development`|`wiki-edit`|`1680795276351`|`wiki-edits`|
 
-### Tuning configuration
+### `tuningConfig`
+
+Specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.

Review Comment:
   Since this is not a tutorial, I think starting this sentence with "you" improves readability. 
   ```suggestion
   You specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "techdocsmith (via GitHub)" <gi...@apache.org>.
techdocsmith commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744411099


##########
docs/ingestion/supervisor.md:
##########
@@ -24,30 +24,40 @@ sidebar_label: Supervisor
   -->
 
 A supervisor manages streaming ingestion from external streaming sources into Apache Druid.
-Supervisors oversee the state of indexing tasks to coordinate handoffs, manage failures, and ensure that the scalability and replication requirements are maintained.
 
-This topic uses the Apache Kafka term offset to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is sequence number.
+Supervisors oversee the state of indexing tasks to coordinate [handoffs](../design/storage#indexing-and-handoff), manage failures, and ensure that the scalability and replication requirements are maintained.
 
-## Supervisor spec
+:::info
+This topic uses the Apache Kafka term _offset_ to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is _sequence number_.
+:::
 
-Druid uses a JSON specification, often referred to as the supervisor spec, to define streaming ingestion tasks.
-The supervisor spec specifies how Druid should consume, process, and index streaming data.
+Druid uses a JSON document, often referred to as a supervisor spec, to specify how Druid should consume, process, and index streaming data. A supervisor spec is a type of [native ingestion spec](../ingestion/ingestion-spec.md).
 
 The following table outlines the high-level configuration options for a supervisor spec:
 
 |Property|Type|Description|Required|
 |--------|----|-----------|--------|
-|`type`|String|The supervisor type. One of `kafka`or `kinesis`.|Yes|
-|`spec`|Object|The container object for the supervisor configuration.|Yes|
-|`spec.dataSchema`|Object|The schema for the indexing task to use during ingestion. See [`dataSchema`](../ingestion/ingestion-spec.md#dataschema) for more information.|Yes|
-|`spec.ioConfig`|Object|The I/O configuration object to define the connection and I/O-related settings for the supervisor and indexing tasks.|Yes|
-|`spec.tuningConfig`|Object|The tuning configuration object to define performance-related settings for the supervisor and indexing tasks.|No|
+|`type`|String|The type of ingestion to supervise, one of [`kafka`](kafka-ingestion.md) or [`kinesis`](kinesis-ingestion.md).|Yes|
+|[`spec`](#spec)|Object|The container object for the supervisor configuration.|Yes|
 |`suspended`|Boolean|Puts the supervisor in a suspended state|No|
 
-### I/O configuration
+## `spec`
+
+The `spec` section consists of three components:
+
+- `dataSchema`. The schema for the indexing task to use during ingestion.
+- [`ioConfig`](#ioconfig), which tells Druid how to connect to the [source system](./index.md#ingestion-methods) and how to parse data.
+- [`tuningConfig`](#tuningconfig), which controls performance-related settings for the supervisor and indexing tasks.
+
+For configuration properties shared across all ingestion methods supported by Druid, including the `dataSchema`, see [native ingestion specs](../ingestion/ingestion-spec.md).
+
+### `ioConfig`
 
-The following table outlines the `ioConfig` configuration properties that apply to both Apache Kafka and Amazon Kinesis ingestion methods.
-For configuration properties specific to Kafka and Kinesis, see [Kafka I/O configuration](kafka-ingestion.md#io-configuration) and [Kinesis I/O configuration](kinesis-ingestion.md#io-configuration) respectively.
+The `ioConfig` influences how data is read from a source system. See native ingestion specification documentation on the [`ioConfig`](../ingestion/ingestion-spec.md#ioconfig) for more information.
+
+The following table outlines the configuration properties that apply to supervisor-led ingestion methods, including Apache Kafka and Amazon Kinesis.

Review Comment:
   ```suggestion
   The `ioConfig` defines how Druid should read data from a source system. See native ingestion specification documentation on the [`ioConfig`](../ingestion/ingestion-spec.md#ioconfig) for more information.
   
   The following table outlines the configuration properties that apply to streaming ingestion supervisors, including Apache Kafka and Amazon Kinesis.
   ```



##########
docs/ingestion/kafka-ingestion.md:
##########
@@ -113,10 +118,13 @@ The following example shows a supervisor spec for the Kafka indexing service:
 
 </details>
 
-### I/O configuration
+### `ioConfig`
+
+The `ioConfig` influences how data is read from a source system, such as Apache Kafka, Amazon S3, a mounted filesystem, or any other supported source system.
 
-The following table outlines the `ioConfig` configuration properties specific to Kafka.
-For configuration properties shared across all streaming ingestion methods, refer to [Supervisor I/O configuration](supervisor.md#io-configuration).
+The following table outlines the `ioConfig` configuration properties specific to Apache Kafka.
+
+For configuration properties shared across all supervisor-led ingestion, refer to [supervisor `ioConfig`](supervisor.md#ioconfig).

Review Comment:
   Maybe:
   > common to all streaming ingestion supervisor configurations.



##########
docs/ingestion/supervisor.md:
##########
@@ -24,30 +24,40 @@ sidebar_label: Supervisor
   -->
 
 A supervisor manages streaming ingestion from external streaming sources into Apache Druid.
-Supervisors oversee the state of indexing tasks to coordinate handoffs, manage failures, and ensure that the scalability and replication requirements are maintained.
 
-This topic uses the Apache Kafka term offset to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is sequence number.
+Supervisors oversee the state of indexing tasks to coordinate [handoffs](../design/storage#indexing-and-handoff), manage failures, and ensure that the scalability and replication requirements are maintained.
 
-## Supervisor spec
+:::info
+This topic uses the Apache Kafka term _offset_ to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is _sequence number_.
+:::
 
-Druid uses a JSON specification, often referred to as the supervisor spec, to define streaming ingestion tasks.
-The supervisor spec specifies how Druid should consume, process, and index streaming data.
+Druid uses a JSON document, often referred to as a supervisor spec, to specify how Druid should consume, process, and index streaming data. A supervisor spec is a type of [native ingestion spec](../ingestion/ingestion-spec.md).

Review Comment:
   ```suggestion
   Druid uses a JSON formatted supervisor spec, to specify how Druid should consume, process, and index streaming data. A supervisor spec is a type of [native ingestion spec](../ingestion/ingestion-spec.md).
   ```
   It is not "often referred to" but always a "supervisor spec". JSON document reads vaguely here.



##########
docs/ingestion/kinesis-ingestion.md:
##########
@@ -27,19 +27,25 @@ import TabItem from '@theme/TabItem';
   ~ under the License.
   -->
 
-When you enable the Kinesis indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Kinesis indexing tasks. Kinesis indexing tasks read events using the Kinesis shard and sequence number mechanism to guarantee exactly-once ingestion. The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+When you enable the Amazon Kinesis indexing service, you can configure supervisors on the Overlord to manage the creation and lifetime of Kinesis indexing tasks.
 
-This topic contains configuration information for the Kinesis indexing service supervisor for Apache Druid.
+The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures, and ensure that scalability and replication requirements are maintained.
+
+Kinesis indexing tasks read events using the Kinesis shard and sequence number mechanism to guarantee exactly-once ingestion. 
+
+This topic contains configuration information for the Amazon Kinesis indexing service supervisor for Apache Druid.
 
 ## Setup
 
 To use the Kinesis indexing service, you must first load the `druid-kinesis-indexing-service` core extension on both the Overlord and the MiddleManager. See [Loading extensions](../configuration/extensions.md#loading-extensions) for more information.
 
 Review [Known issues](#known-issues) before deploying the `druid-kinesis-indexing-service` extension to production.
 
-## Supervisor spec configuration
+## `spec`
 
-This section outlines the configuration properties that are specific to the Amazon Kinesis streaming ingestion method. For configuration properties shared across all streaming ingestion methods supported by Druid, see [Supervisor spec](supervisor.md#supervisor-spec).
+This section outlines the configuration properties that are specific to supervisor-led streaming ingestion for Amazon Kinesis.

Review Comment:
   if you must mention supervisor: "Streaming ingestion supervisors" is easier to read than  "supervisor-led streaming"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744359430


##########
docs/ingestion/supervisor.md:
##########
@@ -24,30 +24,40 @@ sidebar_label: Supervisor
   -->
 
 A supervisor manages streaming ingestion from external streaming sources into Apache Druid.
-Supervisors oversee the state of indexing tasks to coordinate handoffs, manage failures, and ensure that the scalability and replication requirements are maintained.
 
-This topic uses the Apache Kafka term offset to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is sequence number.
+Supervisors oversee the state of indexing tasks to coordinate [handoffs](../design/storage#indexing-and-handoff), manage failures, and ensure that the scalability and replication requirements are maintained.
 
-## Supervisor spec
+:::info
+This topic uses the Apache Kafka term _offset_ to refer to the identifier for records in a partition. If you are using Amazon Kinesis, the equivalent is _sequence number_.
+:::
 
-Druid uses a JSON specification, often referred to as the supervisor spec, to define streaming ingestion tasks.
-The supervisor spec specifies how Druid should consume, process, and index streaming data.
+Druid uses a JSON document, often referred to as a supervisor spec, to specify how Druid should consume, process, and index streaming data. A supervisor spec is a type of [native ingestion spec](../ingestion/ingestion-spec.md).
 
 The following table outlines the high-level configuration options for a supervisor spec:
 
 |Property|Type|Description|Required|
 |--------|----|-----------|--------|
-|`type`|String|The supervisor type. One of `kafka`or `kinesis`.|Yes|
-|`spec`|Object|The container object for the supervisor configuration.|Yes|
-|`spec.dataSchema`|Object|The schema for the indexing task to use during ingestion. See [`dataSchema`](../ingestion/ingestion-spec.md#dataschema) for more information.|Yes|
-|`spec.ioConfig`|Object|The I/O configuration object to define the connection and I/O-related settings for the supervisor and indexing tasks.|Yes|
-|`spec.tuningConfig`|Object|The tuning configuration object to define performance-related settings for the supervisor and indexing tasks.|No|
+|`type`|String|The type of ingestion to supervise, one of [`kafka`](kafka-ingestion.md) or [`kinesis`](kinesis-ingestion.md).|Yes|
+|[`spec`](#spec)|Object|The container object for the supervisor configuration.|Yes|
 |`suspended`|Boolean|Puts the supervisor in a suspended state|No|

Review Comment:
   Is `suspended` on the same level as `type` and `spec`? I thought it was part of the status report.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744362160


##########
docs/ingestion/supervisor.md:
##########
@@ -182,12 +192,16 @@ The following example shows a supervisor spec with `lagBased` autoscaler:
 ```
 </details>
 
-### Tuning configuration
+### `tuningConfig`
+
+The `tuningConfig` controls various tuning parameters specific to each ingestion method. See native ingestion specification documentation on the [`tuningConfig`](../ingestion/ingestion-spec.md#tuningconfig) for more information.

Review Comment:
   ```suggestion
   The `tuningConfig` controls various tuning parameters specific to each ingestion method. See native ingestion specification documentation on the [`tuningConfig`](../ingestion/ingestion-spec.md#tuningconfig) for more information.
   ```
   Is this link correct (../ingestion/ingestion-spec.md#tuningconfig)? Shouldn't it be (ingestion/ingestion-spec.md#tuningconfig)?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "ektravel (via GitHub)" <gi...@apache.org>.
ektravel commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744317421


##########
docs/ingestion/ingestion-spec.md:
##########
@@ -472,24 +474,23 @@ An example `ioConfig` to read JSON data is:
 }
 ```
 
-For details, see the documentation provided by each [ingestion method](./index.md#ingestion-methods).
-
 ## `tuningConfig`
 
-You specify tuning properties in a `tuningConfig` object, which goes at the top level of an ingestion spec.
-Some properties apply to all [ingestion methods](./index.md#ingestion-methods), but most are specific to each individual ingestion method.
+Specify tuning properties in a `tuningConfig` object at the top level of an ingestion spec.
+
+For details, see the documentation for `tuningConfig`  for each [ingestion method](./index.md#ingestion-methods).
 
-The following table lists the common tuning properties shared among ingestion methods:
+The following table lists the common tuning properties shared by all ingestion methods:
 
 |Field|Description|Default|
 |-----|-----------|-------|
-|type|Each ingestion method has its own tuning type code. You must specify the type code that matches your ingestion method. Common options are `index`, `hadoop`, `kafka`, and `kinesis`.||
-|maxRowsInMemory|The maximum number of records to store in memory before persisting to disk. Note that this is the number of rows post-rollup, and so it may not be equal to the number of input records. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first).|`1000000`|
-|maxBytesInMemory|The maximum aggregate size of records, in bytes, to store in the JVM heap before persisting. This is based on a rough estimate of memory usage. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first). `maxBytesInMemory` also includes heap usage of artifacts created from intermediary persists. This means that after every persist, the amount of `maxBytesInMemory` until the next persist will decrease. If the sum of bytes of all intermediary persisted artifacts exceeds `maxBytesInMemory` the task fails.<br /><br />Setting `maxBytesInMemory` to -1 disables this check, meaning Druid will rely entirely on `maxRowsInMemory` to control memory usage. Setting it to zero means the default value will be used (one-sixth of JVM heap size).<br /><br />Note that the estimate of memory usage is designed to be an overestimate, and can be especially high when using complex ingest-time aggregators, including s
 ketches. If this causes your indexing workloads to persist to disk too often, you can set `maxBytesInMemory` to -1 and rely on `maxRowsInMemory` instead.|One-sixth of max JVM heap size|
-|skipBytesInMemoryOverheadCheck|The calculation of maxBytesInMemory takes into account overhead objects created during ingestion and each intermediate persist. Setting this to true can exclude the bytes of these overhead objects from maxBytesInMemory check.|false|
-|indexSpec|Defines segment storage format options to use at indexing time.|See [`indexSpec`](#indexspec) for more information.|
-|indexSpecForIntermediatePersists|Defines segment storage format options to use at indexing time for intermediate persisted temporary segments.|See [`indexSpec`](#indexspec) for more information.|
-|Other properties|Each ingestion method has its own list of additional tuning properties. See the documentation for each method for a full list: [Kafka indexing service](../ingestion/kafka-ingestion.md#tuning-configuration), [Kinesis indexing service](../ingestion/kinesis-ingestion.md#tuning-configuration), [Native batch](native-batch.md#tuningconfig), and [Hadoop-based](hadoop.md#tuningconfig).||
+|`type`|Each ingestion method has its own tuning type code. You must specify the type code that matches your ingestion method. Common options are `index`, `hadoop`, `kafka`, and `kinesis`.||
+|`maxRowsInMemory`|The maximum number of records to store in memory before persisting to disk. Note that this is the number of rows post-rollup, and so it may not be equal to the number of input records. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first).|`1000000`|
+|`maxBytesInMemory`|The maximum aggregate size of records, in bytes, to store in the JVM heap before persisting. This is based on a rough estimate of memory usage. Ingested records will be persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are reached (whichever happens first). `maxBytesInMemory` also includes heap usage of artifacts created from intermediary persists. This means that after every persist, the amount of `maxBytesInMemory` until the next persist will decrease. If the sum of bytes of all intermediary persisted artifacts exceeds `maxBytesInMemory` the task fails.<br /><br />Setting `maxBytesInMemory` to -1 disables this check, meaning Druid will rely entirely on `maxRowsInMemory` to control memory usage. Setting it to zero means the default value will be used (one-sixth of JVM heap size).<br /><br />Note that the estimate of memory usage is designed to be an overestimate, and can be especially high when using complex ingest-time aggregators, including
  sketches. If this causes your indexing workloads to persist to disk too often, you can set `maxBytesInMemory` to -1 and rely on `maxRowsInMemory` instead.|One-sixth of max JVM heap size|
+|`skipBytesInMemoryOverheadCheck`|The calculation of maxBytesInMemory takes into account overhead objects created during ingestion and each intermediate persist. Setting this to true can exclude the bytes of these overhead objects from maxBytesInMemory check.|false|
+|[`indexSpec`](#indexspec)|Defines segment storage format options to use at indexing time.|See [`indexSpec`](#indexspec) for more information.|

Review Comment:
   ```suggestion
   |`indexSpec`|Defines segment storage format options to use at indexing time.|See [`indexSpec`](#indexspec) for more information.|
   ```
   No need to link it twice.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Docs - supervisor spec links (druid)

Posted by "techdocsmith (via GitHub)" <gi...@apache.org>.
techdocsmith commented on code in PR #16988:
URL: https://github.com/apache/druid/pull/16988#discussion_r1744382924


##########
docs/ingestion/ingestion-spec.md:
##########
@@ -515,13 +516,13 @@ For information on defining an `indexSpec` in a query context, see [SQL-based in
 
 |Field|Description|Default|
 |-----|-----------|-------|
-|bitmap|Compression format for bitmap indexes. Should be a JSON object with `type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
-|dimensionCompression|Compression format for dimension columns. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
-|stringDictionaryEncoding|Encoding format for string value dictionaries used by STRING and [COMPLEX&lt;json&gt;](../querying/nested-columns.md) columns. To enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`. Optionally, you can specify the `bucketSize` and `formatVersion` properties. See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
-|metricCompression|Compression format for primitive type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more efficient than `uncompressed`, but not supported by older versions of Druid).|`lz4`|
-|longEncoding|Encoding format for long-typed columns. Applies regardless of whether they are dimensions or metrics. Options are `auto` or `longs`. `auto` encodes the values using offset or lookup table depending on column cardinality, and store them with variable size. `longs` stores the value as-is with 8 bytes each.|`longs`|
-|complexMetricCompression|Compression format for complex type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`. Options other than `uncompressed` are not compatible with Druid versions older than 31, and only applies to complex metrics which do not have specialized column formats.|`uncompressed`|
-|jsonCompression|Compression format to use for nested column raw data. Options are `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|`bitmap`|Compression format for bitmap indexes. Should be a JSON object with `type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
+|`dimensionCompression`|Compression format for dimension columns. One of `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|`stringDictionaryEncoding`|Encoding format for string value dictionaries used by STRING and [COMPLEX&lt;json&gt;](../querying/nested-columns.md) columns. To enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`. Optionally, you can specify the `bucketSize` and `formatVersion` properties. See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
+|`metricCompression`|Compression format for primitive type metric columns. Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more efficient than `uncompressed`, but not supported by older versions of Druid).|`lz4`|

Review Comment:
   Do we know which version began support for `none`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org