You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "qg-lin (via GitHub)" <gi...@apache.org> on 2024/07/19 10:26:38 UTC
[PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
qg-lin opened a new pull request, #3482:
URL: https://github.com/apache/flink-cdc/pull/3482
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#discussion_r1912690775
##########
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/schema/Schema.java:
##########
@@ -25,6 +25,8 @@
import org.apache.flink.cdc.common.types.RowType;
import org.apache.flink.cdc.common.utils.Preconditions;
+import org.apache.commons.lang3.StringUtils;
Review Comment:
We may use `org.apache.flink.cdc.common.utils.StringUtils#isNullOrWhitespaceOnly`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2401016021
This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2465920318
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3482: [FLINK-34865][pipeline-connector]support sync table and column comments
URL: https://github.com/apache/flink-cdc/pull/3482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "qg-lin (via GitHub)" <gi...@apache.org>.
qg-lin commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2238865439
resolve some problems from PR https://github.com/apache/flink-cdc/pull/3039, thanks for @lvyanquan and @dufeng1010 and @yuxiqian
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#discussion_r1684233027
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisDataSinkOptions.java:
##########
@@ -61,6 +61,12 @@ public class DorisDataSinkOptions {
.withDescription(
"Use automatic redirection of fe without explicitly obtaining the be list");
+ public static final ConfigOption<String> CHARSET_ENCODING =
+ ConfigOptions.key("charset-encoding")
+ .stringType()
+ .defaultValue("UTF-8")
+ .withDescription("Charset encoding for doris http client, default UTF-8.");
+
Review Comment:
Could you please mention this new config in docs?
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisMetadataApplier.java:
##########
Review Comment:
Seems this PR also enables Doris connector to apply comments from upstream. Could you please also add an comment sync E2e case in `MySql2DorisE2eITCase`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2262100368
@yuxiqian Would you like to help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "qg-lin (via GitHub)" <gi...@apache.org>.
qg-lin commented on code in PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#discussion_r1709102837
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisDataSinkOptions.java:
##########
@@ -61,6 +61,12 @@ public class DorisDataSinkOptions {
.withDescription(
"Use automatic redirection of fe without explicitly obtaining the be list");
+ public static final ConfigOption<String> CHARSET_ENCODING =
+ ConfigOptions.key("charset-encoding")
+ .stringType()
+ .defaultValue("UTF-8")
+ .withDescription("Charset encoding for doris http client, default UTF-8.");
+
Review Comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "MOBIN-F (via GitHub)" <gi...@apache.org>.
MOBIN-F commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2487262098
hi,@qg-lin thank you for PR, which is very useful for us. I did some tests after meger and found a problem.
If user defines transform in pipeline, the current TransformParser#generateProjectionColumns method will filter out column comment, such as:
```
source:
type: mysql
....
include-comments.enabled: true
sink:
type: paimon
....
transform:
- source-table: testdb.test_table1
projection: \*, date_format(create_sys_tm,'yyyyMMdd') as pt
primary-keys: id, pt
partition-keys: pt
description: project fields and filter
pipeline:
name: MySQL to Paimon Pipeline
parallelism: 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "qg-lin (via GitHub)" <gi...@apache.org>.
qg-lin commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2277193856
> fix paimon 
@melin thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2573438893
@qg-lin @MOBIN-F would you like to update this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2590182709
@qg-lin Would you like to rebase your PR to latest master branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2262101289
Sure, I'll take it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang merged PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "melin (via GitHub)" <gi...@apache.org>.
melin commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2277162861
fix 
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]
Posted by "qg-lin (via GitHub)" <gi...@apache.org>.
qg-lin commented on code in PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#discussion_r1709104617
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisMetadataApplier.java:
##########
Review Comment:
My pleasure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org