[jira] [Created] (FLINK-16396) Support binding Kafka egresses with YAML-ized modules

["Tzu-Li (Gordon) Tai (Jira)" <ji...@apache.org>]

Tzu-Li (Gordon) Tai created FLINK-16396:
-------------------------------------------

             Summary: Support binding Kafka egresses with YAML-ized modules
                 Key: FLINK-16396
                 URL: https://issues.apache.org/jira/browse/FLINK-16396
             Project: Flink
          Issue Type: New Feature
          Components: Stateful Functions
    Affects Versions: statefun-1.1
            Reporter: Tzu-Li (Gordon) Tai
            Assignee: Tzu-Li (Gordon) Tai


For a full multi-language story in Stateful Functions, we're still lacking a YAML-ized way to bind egresses. We should start with a Kafka one, as that is the currently only supported ingress for multi-language applications.

We propose to initially add a more "manual" approach for this, which gives the user full freedom on partitioning and record serialization.
A more specialized egress with built-in routing / protobuf serialization can maybe be added in the future once we collect more knowledge on usage from users.

The plan is as follows:

* Add a {{KafkaProducerRecord}} protobuf message with schema:
{code}
message KafkaProducerRecord {
    string topic
    string key
    bytes valueBytes
}
{code}
How the value was serialized would be completely user-space; the bytes are written as is to Kafka. Partitioning depends on the partitioner used by the Kafka client, which may be configured with Kafka properties via YAML.

* The Flink sink provider for this egress should expect only messages of {{KafkaProducerRecord}} (wrapped within a protobuf Any). It simply transforms the information in the received {{KafkaProducerRecord}} to Kafka's own {{ProducerRecord}}, and writes that to Kafka.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)