[jira] [Updated] (FLINK-16396) Support binding Kafka egresses with YAML-ized modules

["Tzu-Li (Gordon) Tai (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-16396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tzu-Li (Gordon) Tai updated FLINK-16396:
----------------------------------------
    Priority: Blocker  (was: Critical)

> Support binding Kafka egresses with YAML-ized modules
> -----------------------------------------------------
>
>                 Key: FLINK-16396
>                 URL: https://issues.apache.org/jira/browse/FLINK-16396
>             Project: Flink
>          Issue Type: New Feature
>          Components: Stateful Functions
>    Affects Versions: statefun-1.1
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Blocker
>
> For a full multi-language story in Stateful Functions, we're still lacking a YAML-ized way to bind egresses. We should start with a Kafka one, as that is the currently only supported ingress for multi-language applications.
> We propose to initially add a more "manual" approach for this, which gives the user full freedom on partitioning and record serialization.
> A more specialized egress with built-in routing / protobuf serialization can maybe be added in the future once we collect more knowledge on usage from users.
> The plan is as follows:
> * Add a {{KafkaProducerRecord}} protobuf message with schema:
> {code}
> message KafkaProducerRecord {
>     string topic
>     string key
>     bytes valueBytes
> }
> {code}
> How the value was serialized would be completely user-space; the bytes are written as is to Kafka. Partitioning depends on the partitioner used by the Kafka client, which may be configured with Kafka properties via YAML.
> * The Flink sink provider for this egress should expect only messages of {{KafkaProducerRecord}} (wrapped within a protobuf Any). It simply transforms the information in the received {{KafkaProducerRecord}} to Kafka's own {{ProducerRecord}}, and writes that to Kafka.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)