[jira] [Created] (FLINK-24622) Unified sources do not close scheduled threads from SplitEnumeratorContext#callAsync()

["Mason Chen (Jira)" <ji...@apache.org>]

Mason Chen created FLINK-24622:
----------------------------------

             Summary: Unified sources do not close scheduled threads from SplitEnumeratorContext#callAsync()
                 Key: FLINK-24622
                 URL: https://issues.apache.org/jira/browse/FLINK-24622
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Common
    Affects Versions: 1.13.2
            Reporter: Mason Chen


From user mailing list:

I was wondering how to cancel a task that is enqueued by the callAsync() method, the one that takes in a time interval. For example, the KafkaSource uses this for topic partition discovery. It would be straightforward if the API returned the underlying future so that a process can cancel it. 

For Kafka, the enumerator shutdown seems to be unclean since it only closes the admin client and kafka consumer but not the topic partition discovery task. Furthermore, exceptions from that task will cause job failure and can potentially happen if the task is still running with the admin client closed. How can we address this?

This seems to be a bug with the current KafkaSource and also the unified Sources in general. Can you open a bug ticket in jira? I think the enumerator should take of first joining all the async threads before closing the enumerator.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)