You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/08/10 13:00:30 UTC

[GitHub] [druid] petermarshallio opened a new pull request #11569: Docs - ioConfig config index table

petermarshallio opened a new pull request #11569:
URL: https://github.com/apache/druid/pull/11569


   Adds a new table to the "ioConfig" section of the ingestion doc, listing the configuration lines that appear inside ioConfig for each ingestion type.
   
   Frankly, not sure how useful this is, but thought I would offer it up :)
   
   This PR has:
   - [X] been self-reviewed.
   - [ ] been tested in a test Druid cluster.
   
   @techdocsmith @suneet-s welcome your thoughts on the value of this table...?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] petermarshallio edited a comment on pull request #11569: Docs - ioConfig config index table

Posted by GitBox <gi...@apache.org>.
petermarshallio edited a comment on pull request #11569:
URL: https://github.com/apache/druid/pull/11569#issuecomment-972872062


   Like #11936 / #11980 on reflection I think this PR is a partial solution for `ioConfig` and may just make the page itself massive and something else that would have to be referenced when `ioConfig` is enhanced over time.  @techdocsmith I'm thinking that I'll just close this PR and raise one specifically for the other stylistic updates.  Concur?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] petermarshallio closed pull request #11569: Docs - ioConfig config index table

Posted by GitBox <gi...@apache.org>.
petermarshallio closed pull request #11569:
URL: https://github.com/apache/druid/pull/11569


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] techdocsmith commented on a change in pull request #11569: Docs - ioConfig config index table

Posted by GitBox <gi...@apache.org>.
techdocsmith commented on a change in pull request #11569:
URL: https://github.com/apache/druid/pull/11569#discussion_r832742438



##########
File path: docs/ingestion/index.md
##########
@@ -680,10 +680,45 @@ See [Flatten spec](./data-formats.md#flattenspec) for more details.
 ## `ioConfig`
 
 The `ioConfig` influences how data is read from a source system, such as Apache Kafka, Amazon S3, a mounted
-filesystem, or any other supported source system. The `inputFormat` property applies to all
-[ingestion method](#ingestion-methods) except for Hadoop ingestion. The Hadoop ingestion still
-uses the [`parser`](#parser-deprecated) in the legacy `dataSchema`.
-The rest of `ioConfig` is specific to each individual ingestion method.
+filesystem, or any other supported source system.
+
+> The `inputFormat` property applies to all [ingestion methods](#ingestion-methods) except for Hadoop-based ingestion,
+> which uses the deprecated [`parser`](#parser-deprecated) technique inside the legacy `dataSchema` to determine how to
+> read and process source data rows.
+
+The `ioConfig` section contains configuration options that are specific to each individual ingestion method.  For more details, see the documentation provided by each [ingestion method](#ingestion-methods).
+
+| configuration | [`index_parallel`](https://druid.apache.org/docs/latest/ingestion/native-batch.html#ioconfig) | [`kafka`](https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html#kafkasupervisorioconfig) | [`kinesis`](https://druid.apache.org/docs/latest/development/extensions-core/kinesis-ingestion.html#kinesissupervisorioconfig) | [`hadoop`](https://druid.apache.org/docs/latest/ingestion/hadoop.html#ioconfig) | [`compact`](https://druid.apache.org/docs/latest/ingestion/data-management.html#compaction-ioconfig) |

Review comment:
       This table isn't doing a lot of work b/c there isn't much overlap could we list the properties according to type?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] petermarshallio edited a comment on pull request #11569: Docs - ioConfig config index table

Posted by GitBox <gi...@apache.org>.
petermarshallio edited a comment on pull request #11569:
URL: https://github.com/apache/druid/pull/11569#issuecomment-972872062


   Like #11936 on reflection I think this PR is a partial solution for `ioConfig` and may just make the page itself massive and something else that would have to be referenced when `ioConfig` is enhanced over time.  @techdocsmith I'm thinking that I'll just close this PR and raise one specifically for the other stylistic updates.  Concur?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] petermarshallio commented on pull request #11569: Docs - ioConfig config index table

Posted by GitBox <gi...@apache.org>.
petermarshallio commented on pull request #11569:
URL: https://github.com/apache/druid/pull/11569#issuecomment-972872062


   Relates to #11936 but I think this is just a partial solution to the issue and in fact might just make the page itself massive and something else that would have to be referenced when `ioConfig` is enhanced over time.  @techdocsmith I'm thinking that I'll just close this PR and raise one specifically for the other stylistic updates.  Concur?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org