[jira] [Commented] (FLINK-29398) Utilize Rack Awareness in Flink Consumer

["Chirag Dewan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-29398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17711739#comment-17711739 ] 

Chirag Dewan commented on FLINK-29398:
--------------------------------------

This is a good thread and a useful feature. Thanks for this.

When can I expect this to be delivered as a Flink release?

Also, any more insights on how you used it in your multi-AZ cluster? Are you deploying a stretched Flink cluster or a Flink cluster in every AZ in some active-active or active-passive mode? 

With a stretched cluster, I assume we will simply replace the broker-consumer cost with the TaskManager-JobManager networking cost. Is this assumption correct?

> Utilize Rack Awareness in Flink Consumer
> ----------------------------------------
>
>                 Key: FLINK-29398
>                 URL: https://issues.apache.org/jira/browse/FLINK-29398
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka
>            Reporter: Jeremy DeGroot
>            Assignee: Jeremy DeGroot
>            Priority: Major
>              Labels: pull-request-available
>
> [KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] was implemented some time ago in Kafka. This allows brokers and consumers to communicate about the rack (or AWS Availability Zone) they're located in. Reading from a local broker can save money in bandwidth and improve latency for your consumers.
> Flink Kafka consumers currently cannot easily use rack awareness if they're deployed across multiple racks or availability zones, because they have no control over which rack the Task Manager they'll be assigned to may be in. 
> This improvement proposes that a Kafka Consumer could be configured with a callback or Future that could be run when it's being configured on the task manager, that will set the appropriate value at runtime if a value is provided. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)