You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/10/31 21:56:59 UTC

[GitHub] [flink] syhily opened a new pull request, #21207: [FLINK-25686][Connector/Pulsar] Add schema evolution support for Pulsar connector

syhily opened a new pull request, #21207:
URL: https://github.com/apache/flink/pull/21207

   ## What is the purpose of the change
   
   Currently the source connector reads all messages as `Message<byte[]>`, and implements a deserialization abstraction to deserialize the byte array message. This deserialization abstraction supports both flink schema and pulsar schema. However because of this extra layer of abstraction, even when using pulsar schema, the schema info is not provided to the consumer (client), so it is not using any pulsar’s schema validation mechanism. (If pulsar client has a schema, it will send the schema to broker on connect and broker will validate if it is a valid schema according to a preset compatibility rule: [pulsar.apache.org/docs/en/schema-evolution-compatibility](https://pulsar.apache.org/docs/en/schema-evolution-compatibility/)).
   
   ## Brief change log
   
     - Add new `BytesSchema` for wrapping Pulsar original schema.
     - Added tests for different schemas.
     - Changed PulsarSchemaWrapper#decode() method.
     - Passing schema to split reader.
     - Refactor KeyValueSchemaFactory.
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
     - Added tests in PulsarDeserializationSchemaTest
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): (yes)
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
     - The serializers: (yes)
     - The runtime per-record code paths (performance sensitive): (no)
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
     - The S3 file system connector: (no)
   
   ## Documentation
   
     - Does this pull request introduce a new feature? (yes)
     - If yes, how is the feature documented? (JavaDocs)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] tisonkun commented on pull request #21207: [FLINK-25686][Connector/Pulsar] Add schema evolution support for Pulsar connector

Posted by GitBox <gi...@apache.org>.
tisonkun commented on PR #21207:
URL: https://github.com/apache/flink/pull/21207#issuecomment-1331557250

   Closing...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] syhily commented on pull request #21207: [FLINK-25686][Connector/Pulsar] Add schema evolution support for Pulsar connector

Posted by GitBox <gi...@apache.org>.
syhily commented on PR #21207:
URL: https://github.com/apache/flink/pull/21207#issuecomment-1306072125

   `KafkaShuffleExactlyOnceITCase` always failed on my PR. I hate it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #21207: [FLINK-25686][Connector/Pulsar] Add schema evolution support for Pulsar connector

Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #21207:
URL: https://github.com/apache/flink/pull/21207#issuecomment-1297744613

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "a5d473d293dac03edf7d4387da01d4ec3ac31ae6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a5d473d293dac03edf7d4387da01d4ec3ac31ae6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a5d473d293dac03edf7d4387da01d4ec3ac31ae6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] tisonkun closed pull request #21207: [FLINK-25686][Connector/Pulsar] Add schema evolution support for Pulsar connector

Posted by GitBox <gi...@apache.org>.
tisonkun closed pull request #21207: [FLINK-25686][Connector/Pulsar] Add schema evolution support for Pulsar connector
URL: https://github.com/apache/flink/pull/21207


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] MartijnVisser commented on pull request #21207: [FLINK-25686][Connector/Pulsar] Add schema evolution support for Pulsar connector

Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on PR #21207:
URL: https://github.com/apache/flink/pull/21207#issuecomment-1330955927

   With https://github.com/apache/flink-connector-pulsar/pull/1 getting close to get merged, let's not merge this PR but:
   
   1. Wait for the mentioned PR to be merged
   2. We'll release the Pulsar connector 3.0.0, which is the equivalent of the `release-1.16` version. 
   3. We'll update the Pulsar connector with whatever changes that exist in `master` but haven't been externalised yet. 
   4. Move this PR (and other Pulsar PRs) to the external connector repository so we can separately from the Flink release, release a new version of the Flink connector. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] tisonkun commented on pull request #21207: [FLINK-25686][Connector/Pulsar] Add schema evolution support for Pulsar connector

Posted by GitBox <gi...@apache.org>.
tisonkun commented on PR #21207:
URL: https://github.com/apache/flink/pull/21207#issuecomment-1305288701

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] tisonkun commented on pull request #21207: [FLINK-25686][Connector/Pulsar] Add schema evolution support for Pulsar connector

Posted by GitBox <gi...@apache.org>.
tisonkun commented on PR #21207:
URL: https://github.com/apache/flink/pull/21207#issuecomment-1331557183

   @MartijnVisser Thanks for your help! Sounds good :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] syhily commented on pull request #21207: [FLINK-25686][Connector/Pulsar] Add schema evolution support for Pulsar connector

Posted by GitBox <gi...@apache.org>.
syhily commented on PR #21207:
URL: https://github.com/apache/flink/pull/21207#issuecomment-1305792548

   @tisonkun I'll check it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org