Re: [DISCUSS]FLIP-150: Introduce Hybrid Source

[Nicholas Jiang <pr...@163.com>]

Hi Thomas,
Thanks for your detailed reply for the design of Hybrid Source. I would
reply to the questions you mentioned as follows:

1.Has there been related activity since the FLIP was created?
Yes, the FLIP has the initial version of Hybrid Source implementation. You'd
better refer to the repository: git@github.com:wuchong/flink-hackathon.git .

2.The actual FLIP-27 source reader would need to signal to the
"HybridSourceReader" (HSR) that they are done and then the HSR would send
the switch event to the coordinator?
The communication in Hybrid Source should be between "HybridSourceReader"
and "HybridSplitEnumerator". After a "SourceReader" in "HybridSourceReader"
finishes reading the source split assigned from split enumerator and
switches to another "SourceReader",  "HybridSourceReader" send a signal to
"HybridSplitEnumerator" to notify that the "SourceReader" finishes reading.
Then "HybridSplitEnumerator" receives the "SourceReaderSwitchedEvent" signal
and switch to another "SplitEnumerator".

3.The actual split type that flows between enumerator and reader would be
"HybridSourceSplit" and it would wrap the specific split (in the example
either HDFS or Kafka)?
"HybridSourceSplit" wraps the source split types for the "SourceSplit"
created by the "SplitEnumerator" in "HybridSplitEnumerator".

4."HybridSplitEnumerator" still needs a way to extract them from the actual
enumerator. That could be done by the enumerator checkpoint state mapping
function looking at the current split assignments, which would not require
modification of existing enumerators?
IMO, the above way to extract them from the actual enumerator could be
defined in the state conversion function which is invoked in
"HybridSplitEnumerator" when switch one "SwitchableSplitEnumerator" to
another "SwitchableSplitEnumerator". Because the "SwitchableSplitEnumerator"
interface could return the end state of the actual enumerator.

I don't know if it has answered your confusion for Hybrid Source. If you
still have questions, keep discussion with me. Thanks for your attention.

Best,
Nicholas Jiang



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/