[GitHub] [druid] a2l007 commented on a change in pull request #10387: Support combining inputsource for parallel ingestion

[GitBox <gi...@apache.org>]

a2l007 commented on a change in pull request #10387:
URL: https://github.com/apache/druid/pull/10387#discussion_r487429439



##########
File path: docs/ingestion/native-batch.md
##########
@@ -1368,6 +1368,48 @@ Compared to the other native batch InputSources, SQL InputSource behaves differe
 * Similar to file-based input formats, any updates to existing data will replace the data in segments specific to the intervals specified in the `granularitySpec`.
 
 
+### Combining Input Source
+
+The Combining input source is used to read data from multiple InputSources. This input source should be only used if all the delegate input sources are
+ _splittable_ and can be used by the [Parallel task](#parallel-task). This input source will identify the splits from its delegates and each split will be processed by a worker task. Similar to other input sources, this input source supports a single `inputFormat`. Therefore, please note that delegate input sources requiring an `inputFormat` must have the same format for input data.

Review comment:
       Yeah did think about that and it would be definitely be more useful if we can handle multiple inputFormats as well. For doing this, we might have to push `inputFormat` to be a part of the `inputSource` config. Since it would impact other inputSources as well, I thought it would be better to investigate that separately and have an independent PR for that.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org