You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "sunxiaojian (via GitHub)" <gi...@apache.org> on 2023/11/07 10:06:03 UTC
[PR] Debezium-json supports debezium records with included schema. [incubator-paimon]
sunxiaojian opened a new pull request, #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close https://github.com/apache/incubator-paimon/issues/2227
<!-- What is the purpose of the change -->
KafkaDebeziumWithSchemaSyncTableActionITCase
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature]Debezium-json supports debezium records with included schema. [incubator-paimon]
Posted by "MonsterChenzhuo (via GitHub)" <gi...@apache.org>.
MonsterChenzhuo commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1798329020
Format should cover the requirements of this part, because there are many users to kafak through flinkcdc dbz format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature]Debezium-json supports debezium records with included schema. [incubator-paimon]
Posted by "MonsterChenzhuo (via GitHub)" <gi...@apache.org>.
MonsterChenzhuo commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1798304657
debezium will have several different situations, and I have made some definitions
The native of him is like this:
debezium-native-avro
debezium-native-json
The so-called native means that the primary key information of the table exists in the key of kafka.
The flinkcdc debezium format is like this:
debezium-exclude-json
debezium-include-json
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature]Debezium-json supports debezium records with included schema. [incubator-paimon]
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1798323581
> debezium will have several different situations, and I have made some definitions The native of him is like this: debezium-native-avro debezium-native-json -> #2261 The so-called native means that the primary key information of the table exists in the key of kafka.
>
> The flinkcdc debezium format is like this: debezium-exclude-json debezium-include-json
>
> Example： <img alt="图片" width="1165" src="https://user-images.githubusercontent.com/60029759/281019824-8ddd1213-ee49-49fc-96b0-e1886913ee2a.png">
> debezium will have several different situations, and I have made some definitions The native of him is like this: debezium-native-avro debezium-native-json -> #2261 The so-called native means that the primary key information of the table exists in the key of kafka.
>
> The flinkcdc debezium format is like this: debezium-exclude-json debezium-include-json
>
> Example： <img alt="图片" width="1165" src="https://user-images.githubusercontent.com/60029759/281019824-8ddd1213-ee49-49fc-96b0-e1886913ee2a.png">
1. I have submitted a PR in native mode for reference, https://github.com/apache/incubator-paimon/pull/2261
2. Debezium JSON should be handled with only one parser as much as possible, and the configuration should be consistent when using debezium JSON, such as flink-kafka-connector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature]Debezium-json supports debezium records with included schema. [incubator-paimon]
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1798224996
@MonsterChenzhuo @yuzelin PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature]Debezium-json supports debezium records with included schema. [incubator-paimon]
Posted by "yuzelin (via GitHub)" <gi...@apache.org>.
yuzelin commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1813867940
Resolve conflicts and remove JsonPrimaryKeyDeserializationSchema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature]Debezium-json supports debezium records with included schema. [incubator-paimon]
Posted by "MonsterChenzhuo (via GitHub)" <gi...@apache.org>.
MonsterChenzhuo commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1798342429
> 1. [[Feature] Add debezium-json format #2261](https://github.com/apache/incubator-paimon/pull/2261)
https://github.com/apache/incubator-paimon/pull/2261There is no problem with this pr.
But DataFormat needs to be changed.
> > > > debezium will have several different situations, and I have made some definitions The native of him is like this: debezium-native-avro debezium-native-json -> #2261 The so-called native means that the primary key information of the table exists in the key of kafka.
> > > > The flinkcdc debezium format is like this: debezium-exclude-json debezium-include-json
> > > > Example： <img alt="图片" width="1165" src="https://user-images.githubusercontent.com/60029759/281019824-8ddd1213-ee49-49fc-96b0-e1886913ee2a.png">
> > >
> > >
> > > @MonsterChenzhuo
> > > ```
> > > 1. I have submitted a PR in native mode for reference, [[Feature] Add debezium-json format #2261](https://github.com/apache/incubator-paimon/pull/2261)
> > >
> > > 2. Debezium JSON should be handled with only one parser as much as possible, and the configuration should be consistent when using debezium JSON, such as flink-kafka-connector
> > > ```
> >
> >
> > Flink cdc often does not store the primary key information of kafka. Format should cover the requirements of this part, because there are many users to kafak through flinkcdc dbz format.
>
> I'm just saying to maintain consistent configuration, the deserialization method should still be defined by oneself
>
> And don't break it down into "debezium-exclude-json" and ”debezium-include-json", just keep one debezium-json
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature]Debezium-json supports debezium records with included schema. [incubator-paimon]
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1798337086
> > > debezium will have several different situations, and I have made some definitions The native of him is like this: debezium-native-avro debezium-native-json -> #2261 The so-called native means that the primary key information of the table exists in the key of kafka.
> > > The flinkcdc debezium format is like this: debezium-exclude-json debezium-include-json
> > > Example： <img alt="图片" width="1165" src="https://user-images.githubusercontent.com/60029759/281019824-8ddd1213-ee49-49fc-96b0-e1886913ee2a.png">
> >
> >
> > @MonsterChenzhuo
> > ```
> > 1. I have submitted a PR in native mode for reference, [[Feature] Add debezium-json format #2261](https://github.com/apache/incubator-paimon/pull/2261)
> >
> > 2. Debezium JSON should be handled with only one parser as much as possible, and the configuration should be consistent when using debezium JSON, such as flink-kafka-connector
> > ```
>
> Flink cdc often does not store the primary key information of kafka. Format should cover the requirements of this part, because there are many users to kafak through flinkcdc dbz format.
I'm just saying to maintain consistent configuration, the deserialization method should still be defined by oneself
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature]Debezium-json supports debezium records with included schema. [incubator-paimon]
Posted by "sunxiaojian (via GitHub)" <gi...@apache.org>.
sunxiaojian commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1798324739
> debezium will have several different situations, and I have made some definitions The native of him is like this: debezium-native-avro debezium-native-json -> #2261 The so-called native means that the primary key information of the table exists in the key of kafka.
>
> The flinkcdc debezium format is like this: debezium-exclude-json debezium-include-json
>
> Example： <img alt="图片" width="1165" src="https://user-images.githubusercontent.com/60029759/281019824-8ddd1213-ee49-49fc-96b0-e1886913ee2a.png">
1. I have submitted a PR in native mode for reference, https://github.com/apache/incubator-paimon/pull/2261
2. Debezium JSON should be handled with only one parser as much as possible, and the configuration should be consistent when using debezium JSON, such as flink-kafka-connector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] Debezium-json supports debezium records with included schema [incubator-paimon]
Posted by "luckyLJY (via GitHub)" <gi...@apache.org>.
luckyLJY commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1913429485
可以请教一下debezium json的配置项有哪些吗？目前我个人配置的json无法实现表结构的改变。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature]Debezium-json supports debezium records with included schema. [incubator-paimon]
Posted by "MonsterChenzhuo (via GitHub)" <gi...@apache.org>.
MonsterChenzhuo commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1798299875
> @MonsterChenzhuo @yuzelin PTAL
ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Feature]Debezium-json supports debezium records with included schema. [incubator-paimon]
Posted by "MonsterChenzhuo (via GitHub)" <gi...@apache.org>.
MonsterChenzhuo commented on PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280#issuecomment-1798327411
> > debezium will have several different situations, and I have made some definitions The native of him is like this: debezium-native-avro debezium-native-json -> #2261 The so-called native means that the primary key information of the table exists in the key of kafka.
> > The flinkcdc debezium format is like this: debezium-exclude-json debezium-include-json
> > Example： <img alt="图片" width="1165" src="https://user-images.githubusercontent.com/60029759/281019824-8ddd1213-ee49-49fc-96b0-e1886913ee2a.png">
>
> @MonsterChenzhuo
>
> 1. I have submitted a PR in native mode for reference, [[Feature] Add debezium-json format #2261](https://github.com/apache/incubator-paimon/pull/2261)
>
> 2. Debezium JSON should be handled with only one parser as much as possible, and the configuration should be consistent when using debezium JSON, such as flink-kafka-connector
Flink cdc often does not store the primary key information of kafka.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [cdc] Debezium-json supports debezium records with included schema [incubator-paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #2280:
URL: https://github.com/apache/incubator-paimon/pull/2280
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org