[jira] [Created] (KAFKA-7004) Support configurable restore consumer poll timeout

["Shawn Nguyen (JIRA)" <ji...@apache.org>]

Shawn Nguyen created KAFKA-7004:
-----------------------------------

             Summary: Support configurable restore consumer poll timeout
                 Key: KAFKA-7004
                 URL: https://issues.apache.org/jira/browse/KAFKA-7004
             Project: Kafka
          Issue Type: Improvement
          Components: streams
    Affects Versions: 1.1.0
            Reporter: Shawn Nguyen


In the StateChangelogReader, the restore consumer is currently hard coded to poll for 10ms at most per call.
{noformat}
public Collection<TopicPartition> restore(final RestoringTasks active) { if (!needsInitializing.isEmpty()) { initialize(); } if (needsRestoring.isEmpty()) { restoreConsumer.unsubscribe(); return completed(); } final Set<TopicPartition> restoringPartitions = new HashSet<>(needsRestoring.keySet()); try { final ConsumerRecords<byte[], byte[]> allRecords = restoreConsumer.poll(10); for (final TopicPartition partition : restoringPartitions) { restorePartition(allRecords, partition, active.restoringTaskFor(partition)); }
{noformat}
It'd be nice to be able to configure the restore consumer to poll for a larger timeout (e.g. 500ms) to give it more time to accumulate records for the restoration task. In the main event loop for polling in StreamThread.java, the main consumer uses the POLL_MS_CONFIG set in StreamsConfig.java to configure the max poll timeout. We could construct a similar config in the StreamsConfig class, but prefixed with the consumer type (restore in this case).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)