You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/11/23 11:45:51 UTC
[GitHub] [druid] petermarshallio opened a new issue #11980: Docs - ioConfig options can be simplified
petermarshallio opened a new issue #11980:
URL: https://github.com/apache/druid/issues/11980
**In documentation, ioConfig configuration options are split into different ingestion task types, introducing potential hops between pages, and potentially giving conflicting information when a reader searches for a config option that is common across ingestion types and described differently.**
### Affected Version
0.22.0
### Description
- `ioConfig` documentation is split across five pages:
https://druid.apache.org/docs/latest/ingestion/native-batch.html#ioconfig
https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html#kafkasupervisorioconfig
https://druid.apache.org/docs/latest/development/extensions-core/kinesis-ingestion.html#kinesissupervisorioconfig
https://druid.apache.org/docs/latest/ingestion/hadoop.html#ioconfig
https://druid.apache.org/docs/latest/ingestion/data-management.html#compaction-ioconfig
- There are 8 options common to both kinesis and kafka ingestion that are duplicated
- Some addition options are unique to kafka (6) and some unique to kinesis (8) but they're difficult to find
- inputFormat is common to Kinesis, Kafka, and Batch but contains deprecated Parser information
- Some options are only available in index_parallel, compact, and hadoop-type jobs
### Proposal
Based on the presumption that learners are interested in tuning options specific to their own ingestion source systems:
- New "Ingestion connections" section after "Ingestion spec"
- New intro page begins with definition of ioConfig and has links out to hadoop and compaction ioConfig sections
- New "Stream connections" page begins with the 8 common options, with sections for the Kafka and Kinesis-specific options
- New "Batch connections" page focuses on index_parallel options only
- Existing [Data Formats](https://druid.apache.org/docs/latest/ingestion/data-formats.html) page revisited to focus on `inputFormat` only, e.g. parser section could be separated and links added to the Stream connections page as "deprecated"
- Refine the config options according to their functional effect with common headings across the batch and streaming pages.
- Refine definitions for all config options, explaining differences by exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] petermarshallio commented on issue #11980: Docs - ioConfig options can be simplified
Posted by GitBox <gi...@apache.org>.
petermarshallio commented on issue #11980:
URL: https://github.com/apache/druid/issues/11980#issuecomment-976434753
@techdocsmith I've tried to summarise what we could do around ioConfig in this issue. All commentary welcome!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org