You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "legendtkl (via GitHub)" <gi...@apache.org> on 2023/07/01 13:48:48 UTC

[GitHub] [incubator-paimon] legendtkl opened a new pull request, #1476: [WIP][flink][cdc] add option to decide convert mysql tinyint(1) to bool or not

legendtkl opened a new pull request, #1476:
URL: https://github.com/apache/incubator-paimon/pull/1476

   
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   Linked issue: close https://github.com/)](https://github.com/apache/incubator-paimon/issues/1395
   
   <!-- What is the purpose of the change -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] legendtkl closed pull request #1476: [WIP][flink][cdc] add option to decide convert mysql tinyint(1) to bool or not

Posted by "legendtkl (via GitHub)" <gi...@apache.org>.
legendtkl closed pull request #1476: [WIP][flink][cdc] add option to decide convert mysql tinyint(1) to bool or not
URL: https://github.com/apache/incubator-paimon/pull/1476


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] yuzelin commented on a diff in pull request #1476: [flink][cdc] add option to decide convert mysql tinyint(1) to bool or not

Posted by "yuzelin (via GitHub)" <gi...@apache.org>.
yuzelin commented on code in PR #1476:
URL: https://github.com/apache/incubator-paimon/pull/1476#discussion_r1260537919


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/kafka/KafkaSchema.java:
##########
@@ -124,7 +126,13 @@ private static KafkaSchema parseCanalJson(String record, ObjectMapper objectMapp
                 String type = MySqlTypeUtils.getShortType(fieldType);
                 int precision = MySqlTypeUtils.getPrecision(fieldType);
                 int scale = MySqlTypeUtils.getScale(fieldType);
-                fields.put(fieldName, MySqlTypeUtils.toDataType(type, precision, scale));
+                fields.put(
+                        fieldName,
+                        MySqlTypeUtils.toDataType(
+                                type,
+                                precision,
+                                scale,
+                                MYSQL_CONVERTER_TINYINT1_BOOL.defaultValue()));

Review Comment:
   No need to modify here since the default `toDataType` method uses default value.



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/mysql/MySqlDebeziumJsonEventParser.java:
##########
@@ -170,7 +171,8 @@ public List<DataField> parseSchemaChange() {
                     MySqlTypeUtils.toDataType(
                             column.get("typeName").asText(),
                             length == null ? null : length.asInt(),
-                            scale == null ? null : scale.asInt());
+                            scale == null ? null : scale.asInt(),
+                            MYSQL_CONVERTER_TINYINT1_BOOL.defaultValue());

Review Comment:
   I think we should also pass this value to parser instead of using default value.
   I think in schema evolution case, the TINYINT(1) should also obey the option.
   You should add a test for the schema evolution test case.



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/mysql/MySqlActionUtils.java:
##########
@@ -66,12 +67,30 @@ class MySqlActionUtils {
                     .withDescription(
                             "Whether capture the scan the newly added tables or not, by default is true.");
 
+    public static final ConfigOption<Boolean> MYSQL_CONVERTER_TINYINT1_BOOL =
+            ConfigOptions.key("mysql.converter.tinyint1-to-bool")
+                    .booleanType()
+                    .defaultValue(true)
+                    .withDescription(
+                            "Mysql tinyint type will be converted to boolean type by default, if you want to convert to tinyint type, "
+                                    + "you can set this option to false.");
+
     static Connection getConnection(Configuration mySqlConfig) throws Exception {
-        return DriverManager.getConnection(
+        String url =
                 String.format(
-                        "jdbc:mysql://%s:%d/",
+                        "jdbc:mysql://%s:%d",
                         mySqlConfig.get(MySqlSourceOptions.HOSTNAME),
-                        mySqlConfig.get(MySqlSourceOptions.PORT)),
+                        mySqlConfig.get(MySqlSourceOptions.PORT));
+

Review Comment:
   It's better to add a link to mysql type doc to explain why you add this.



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/mysql/MySqlDebeziumJsonEventParser.java:
##########
@@ -250,7 +252,10 @@ private MySqlSchema buildMySqlSchema(String database, MySqlCreateTableStatement
                         name.getSimpleName(),
                         Tuple2.of(
                                 MySqlTypeUtils.toDataType(
-                                        column.getDataType().getName(), precision, scale),
+                                        column.getDataType().getName(),
+                                        precision,
+                                        scale,
+                                        MYSQL_CONVERTER_TINYINT1_BOOL.defaultValue()),

Review Comment:
   The same as above.



##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/action/cdc/mysql/MySqlSchema.java:
##########
@@ -74,7 +78,8 @@ public MySqlSchema(DatabaseMetaData metaData, String databaseName, String tableN
                 fields.put(
                         fieldName,
                         Tuple2.of(
-                                MySqlTypeUtils.toDataType(fieldType, precision, scale),
+                                MySqlTypeUtils.toDataType(
+                                        fieldType, precision, scale, convertTinyintToByte),

Review Comment:
   the argument name `convertTinyintToByte` should be `convertTinyintToBool`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] yuzelin commented on pull request #1476: [flink][cdc] add option to decide convert mysql tinyint(1) to bool or not

Posted by "yuzelin (via GitHub)" <gi...@apache.org>.
yuzelin commented on PR #1476:
URL: https://github.com/apache/incubator-paimon/pull/1476#issuecomment-1637982516

   Thanks for contributing~ LGTM. Please resolve conflicts.
   
   @JingsongLi please take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] legendtkl closed pull request #1476: [flink][cdc] add option to decide convert mysql tinyint(1) to bool or not

Posted by "legendtkl (via GitHub)" <gi...@apache.org>.
legendtkl closed pull request #1476: [flink][cdc] add option to decide convert mysql tinyint(1) to bool or not
URL: https://github.com/apache/incubator-paimon/pull/1476


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] JingsongLi merged pull request #1476: [flink][cdc] add option to decide convert mysql tinyint(1) to bool or not

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #1476:
URL: https://github.com/apache/incubator-paimon/pull/1476


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [incubator-paimon] legendtkl commented on pull request #1476: [flink][cdc] add option to decide convert mysql tinyint(1) to bool or not

Posted by "legendtkl (via GitHub)" <gi...@apache.org>.
legendtkl commented on PR #1476:
URL: https://github.com/apache/incubator-paimon/pull/1476#issuecomment-1617188167

   Hi, @JingsongLi @yuzelin , would you help review this tiny PR at your convenience?
   
   Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org