You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "qg-lin (via GitHub)" <gi...@apache.org> on 2024/07/19 10:26:38 UTC

[PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

qg-lin opened a new pull request, #3482:
URL: https://github.com/apache/flink-cdc/pull/3482

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#discussion_r1912690775


##########
flink-cdc-common/src/main/java/org/apache/flink/cdc/common/schema/Schema.java:
##########
@@ -25,6 +25,8 @@
 import org.apache.flink.cdc.common.types.RowType;
 import org.apache.flink.cdc.common.utils.Preconditions;
 
+import org.apache.commons.lang3.StringUtils;

Review Comment:
   We may use `org.apache.flink.cdc.common.utils.StringUtils#isNullOrWhitespaceOnly`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2401016021

   This pull request has been automatically marked as stale because it has not had recent activity for 60 days. It will be closed in 30 days if no further activity occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2465920318

   This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3482: [FLINK-34865][pipeline-connector]support sync table and column comments
URL: https://github.com/apache/flink-cdc/pull/3482


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "qg-lin (via GitHub)" <gi...@apache.org>.
qg-lin commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2238865439

   resolve some problems from PR https://github.com/apache/flink-cdc/pull/3039, thanks for @lvyanquan and @dufeng1010 and @yuxiqian 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#discussion_r1684233027


##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisDataSinkOptions.java:
##########
@@ -61,6 +61,12 @@ public class DorisDataSinkOptions {
                     .withDescription(
                             "Use automatic redirection of fe without explicitly obtaining the be list");
 
+    public static final ConfigOption<String> CHARSET_ENCODING =
+            ConfigOptions.key("charset-encoding")
+                    .stringType()
+                    .defaultValue("UTF-8")
+                    .withDescription("Charset encoding for doris http client, default UTF-8.");
+

Review Comment:
   Could you please mention this new config in docs?



##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisMetadataApplier.java:
##########


Review Comment:
   Seems this PR also enables Doris connector to apply comments from upstream. Could you please also add an comment sync E2e case in `MySql2DorisE2eITCase`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2262100368

   @yuxiqian Would you like to help review this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "qg-lin (via GitHub)" <gi...@apache.org>.
qg-lin commented on code in PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#discussion_r1709102837


##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisDataSinkOptions.java:
##########
@@ -61,6 +61,12 @@ public class DorisDataSinkOptions {
                     .withDescription(
                             "Use automatic redirection of fe without explicitly obtaining the be list");
 
+    public static final ConfigOption<String> CHARSET_ENCODING =
+            ConfigOptions.key("charset-encoding")
+                    .stringType()
+                    .defaultValue("UTF-8")
+                    .withDescription("Charset encoding for doris http client, default UTF-8.");
+

Review Comment:
   added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "MOBIN-F (via GitHub)" <gi...@apache.org>.
MOBIN-F commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2487262098

   hi,@qg-lin thank you for PR, which is very useful for us. I did some tests after meger and found a problem.
   If user defines transform in pipeline, the current TransformParser#generateProjectionColumns method will filter out column comment, such as:
   ```
   source:
     type: mysql
     ....
     include-comments.enabled: true
   
   sink:
     type: paimon
     ....
   
   transform:
     - source-table: testdb.test_table1
       projection: \*, date_format(create_sys_tm,'yyyyMMdd') as pt
       primary-keys: id, pt
       partition-keys: pt
       description: project fields and filter
   	
   pipeline:
     name: MySQL to Paimon Pipeline
     parallelism: 1
   ```
     


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "qg-lin (via GitHub)" <gi...@apache.org>.
qg-lin commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2277193856

   > fix paimon ![image](https://private-user-images.githubusercontent.com/1145830/356470073-c2e9ceef-80f3-433e-bddf-165d7390a44b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjMxODI1NjcsIm5iZiI6MTcyMzE4MjI2NywicGF0aCI6Ii8xMTQ1ODMwLzM1NjQ3MDA3My1jMmU5Y2VlZi04MGYzLTQzM2UtYmRkZi0xNjVkNzM5MGE0NGIucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgwOSUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MDlUMDU0NDI3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9MTdmMzc1ZjgyYjdhOGZiOTY2MTRmMDAyMjhiNzRmMWYyOGFlZjlkY2I3N2VmY2JiMTIxMzdlMDYyMGJhMTU5NiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.NqsKRTql94WvQt6tNdc682WUZhWOT8QyJIfINrKWM-c)
   
   @melin thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2573438893

   @qg-lin @MOBIN-F would you like to update this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2590182709

   @qg-lin Would you like to rebase your PR to latest master branch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2262101289

   Sure, I'll take it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang merged PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "melin (via GitHub)" <gi...@apache.org>.
melin commented on PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#issuecomment-2277162861

   fix ![image](https://github.com/user-attachments/assets/9287a0f6-8563-4690-830d-56e75f1d7a7a)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-34865][pipeline-connector]support sync table and column comments [flink-cdc]

Posted by "qg-lin (via GitHub)" <gi...@apache.org>.
qg-lin commented on code in PR #3482:
URL: https://github.com/apache/flink-cdc/pull/3482#discussion_r1709104617


##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-doris/src/main/java/org/apache/flink/cdc/connectors/doris/sink/DorisMetadataApplier.java:
##########


Review Comment:
   My pleasure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org