[I] Kafka Streams runner: read a source's splits in parallel [beam]

["junaiddshaukat (via GitHub)" <gi...@apache.org>]

junaiddshaukat opened a new issue, #39626:
URL: https://github.com/apache/beam/issues/39626

   The Kafka Streams runner splits a source into exactly one part and reads it with a single reader, so a source that could be read in parallel is not.
   
   `ReadTranslator` calls `split(1, options)` and requires exactly one split back. A source returning more is rejected at translation rather than having the extra splits silently dropped, since that would be data loss, but the effect is that such a source cannot be read at all.
   
   Reading several splits means giving each one its own reader and its own watermark, and distributing them across instances, which belongs with the topic-based shuffle work. Bounded and unbounded sources are both affected.
   
   Part of #18479.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org