You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Xinbin Huang <bi...@apache.org> on 2022/03/21 22:35:14 UTC

Temporarily pause splits fetching when unavailable

Hi,

I'm building a custom source that reads from a distributed event queue,
similar to Pulsar or Kafka.

One interesting characteristic of this event queue is that some partitions
may temporarily become unavailable (i.e. a few hours). And it'll need to
resume from the previous progress when the partitions come online again, so
the state needs to be maintained.

The current solution is to suppress the error when reading from unavailable
partitions and keep recreating the client utilized by the split reader.
Apparently, this is not ideal because it is simply wasting resources on
doing nothing.

One potential alternative would be to use
`SplitEnumerator.handleSourceEvent` to signal the reader and temporarily
pause the fetcher somehow. Is this a feasible solution?

Best
Bin