[GitHub] [druid] suneet-s commented on a change in pull request #10264: Fix CombiningFirehose compatibility

[GitBox <gi...@apache.org>]

suneet-s commented on a change in pull request #10264:
URL: https://github.com/apache/druid/pull/10264#discussion_r471539146



##########
File path: server/src/main/java/org/apache/druid/segment/realtime/firehose/CombiningFirehoseFactory.java
##########
@@ -33,11 +36,12 @@
 import java.io.IOException;
 import java.util.Iterator;
 import java.util.List;
+import java.util.stream.Stream;
 
 /**
  * Creates firehose that combines data from different Firehoses. Useful for ingesting data from multiple sources.
  */
-public class CombiningFirehoseFactory implements FirehoseFactory<InputRowParser>
+public class CombiningFirehoseFactory implements FiniteFirehoseFactory<InputRowParser, List<FirehoseFactory>>

Review comment:
       I looked for implementations of `FirehoseFactory` that do not also implement `FiniteFirehoseFactory` and I found a few... eg. `ClippedFirehoseFactory`, `FixedCountFirehoseFactory`, etc.
   
   I think the underlying problem is in `IndexIOConfig#getNonNullInputSource`
   
   `IndexIOConfig` accepts a FirehoseFactory, but [getNonNullInputSource](https://github.com/apache/druid/blob/f6594fff608d4b2e071c7bdd6d86d7f87398ce4f/indexing-service/src/main/java/org/apache/druid/indexing/common/task/IndexTask.java#L1135) expects a `FiniteFirehoseFactory`.
   
   I wonder if we could fix this by creating an adapter that translates any FirehoseFactory into a non-splittable FiniteFirehoseFactory (ie. the same functionality you've implemented in this class).
   
   I understand that the FirehoseFactories are deprecated, so I'm open to other ways of fixing the broken quickstart material as well - can we update the spec used so that the combining firehose factory works without a code change? 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org