You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "AlexAxeman (via GitHub)" <gi...@apache.org> on 2023/03/20 19:11:14 UTC

[GitHub] [flink] AlexAxeman opened a new pull request, #22228: [FLINK-31049] [flink-connector-kafka]Add support for Kafka record headers to KafkaSink

AlexAxeman opened a new pull request, #22228:
URL: https://github.com/apache/flink/pull/22228

   ## What is the purpose of the change
   
   The default `org.apache.flink.connector.kafka.sink.KafkaSink` does not support adding Kafka record headers when using KafkaRecordSerializationSchemaBuilder, which is the most convenient way to create a Kafka sink. This PR adds support for Kafka headers to `KafkaRecordSerializationSchemaBuilder`.
   
   
   ## Brief change log
   
     - *Implemented a `HeaderProducer` that allows creating `Header`s from the input element*
     - *Added setters to `KafkaRecordSerializationSchemaBuilder` to allow setting a `HeaderProducer`*
     - *Added an optional `HeaderProducer` constructor argument to `KafkaRecordSerializationSchemaWrapper` that now uses a `ProducerRecord` constructor that includes the headers.* 
   
   
   ## Verifying this change
   
   Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing
   
   This change added tests and can be verified as follows:
   
     - *Added tests to `KafkaRecordSerializationSchemaBuilderTest`*
   
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? yes?
     - If yes, how is the feature documented? JavaDocs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] tzulitai commented on pull request #22228: [FLINK-31049] [flink-connector-kafka]Add support for Kafka record headers to KafkaSink

Posted by "tzulitai (via GitHub)" <gi...@apache.org>.
tzulitai commented on PR #22228:
URL: https://github.com/apache/flink/pull/22228#issuecomment-1482955341

   Thanks for opening the PR @AlexAxeman! After you reopen the PR against `apache/flink-connector-kafka:main`, please feel free to ping me for a review on the change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #22228: [FLINK-31049] [flink-connector-kafka]Add support for Kafka record headers to KafkaSink

Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #22228:
URL: https://github.com/apache/flink/pull/22228#issuecomment-1482016410

   @AlexAxeman Given that https://issues.apache.org/jira/browse/FLINK-30859 will be worked on shortly, could you open this PR towards `apache/flink-connector-kafka:main`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-31049] [flink-connector-kafka]Add support for Kafka record headers to KafkaSink [flink]

Posted by "AlexAxeman (via GitHub)" <gi...@apache.org>.
AlexAxeman closed pull request #22228: [FLINK-31049] [flink-connector-kafka]Add support for Kafka record headers to KafkaSink
URL: https://github.com/apache/flink/pull/22228


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-31049] [flink-connector-kafka]Add support for Kafka record headers to KafkaSink [flink]

Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on PR #22228:
URL: https://github.com/apache/flink/pull/22228#issuecomment-1746926177

   @tzulitai @MartijnVisser I see https://github.com/apache/flink/pull/22797 , this pr is no longer relevant to this repository, I suggest closing this pr and porting the change to the [new Flink connector kafka repo]( https://github.com/apache/flink-connector-kafka) if it is still relevant.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-31049] [flink-connector-kafka]Add support for Kafka record headers to KafkaSink [flink]

Posted by "AlexAxeman (via GitHub)" <gi...@apache.org>.
AlexAxeman commented on PR #22228:
URL: https://github.com/apache/flink/pull/22228#issuecomment-1746945765

   It was implemented in the new repo in [this PR](https://github.com/apache/flink-connector-kafka/pull/18).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #22228: [FLINK-31049] [flink-connector-kafka]Add support for Kafka record headers to KafkaSink

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #22228:
URL: https://github.com/apache/flink/pull/22228#issuecomment-1476799487

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "1df0da7d5379ee6bcd44869a42688af76f820eac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1df0da7d5379ee6bcd44869a42688af76f820eac",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1df0da7d5379ee6bcd44869a42688af76f820eac UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org