You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "SML0127 (via GitHub)" <gi...@apache.org> on 2024/10/22 02:40:18 UTC
[PR] [FLINK-36578][mysql] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
SML0127 opened a new pull request, #3658:
URL: https://github.com/apache/flink-cdc/pull/3658
The function to convert json to string is different in the snapshot step and the binlog step.
This causes the following differences.
```d
original data : {"key": "value", "key1": "value1"}
snapshot : {"key": "value", "key1": "value1"}
binlog : {"key":"value","key1":"value1"} // no whitespace
```
So I modified JsonStringFormatter and added it to flink-cdc
- modified code line
- line 105: Added whitespace before value
- line 207: Added whitespace after comma
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#issuecomment-2595339265
Sorry to make your rebase more painful, but seems there's still some conflicts between latest `master` branch. Could @SML0127 please double check it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1845347447
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/resources/ddl/column_type_test.sql:
##########
@@ -180,4 +180,22 @@ VALUES (DEFAULT,
23,
29,
31,
- 37);
\ No newline at end of file
+ 37);
+
+CREATE TABLE json_types
+(
+ id SERIAL,
+ json_c0 JSON,
+ json_c1 JSON,
+ json_c2 JSON,
+ int_c INTEGER,
+ PRIMARY KEY (id)
+) DEFAULT CHARSET=utf8;
+
+INSERT INTO json_types
+VALUES (DEFAULT,
+ '{"key1":"value1"}',
+ '{"key1":"value1","key2":"value2"}',
+ '[{"key1":"value1","key2":{"key2_1":"value2_1","key2_2":"value2_2"},"key3":["value3"],"key4":["value4_1","value4_2"]},{"key5":"value5"}]',
+ 1
+ );
Review Comment:
test table for various JSON type data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#issuecomment-2595348177
@yuxiqian
it's resolved 🤣🤣
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#issuecomment-2594735648
Could @SML0127 please rebase this to latest `master` branch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][mysql][BUG] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1836571642
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/json/JsonStringFormatter.java:
##########
Review Comment:
Can we make this enhancement an optional configuration and defaults to the legacy format, considering some users may have relied on this specific JSON format?
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java:
##########
@@ -412,7 +412,7 @@ private void testCommonDataTypes(UniqueDatabase database) throws Exception {
expectedSnapshot[30] = null;
// The json string from binlog will remove useless space
- expectedSnapshot[44] = BinaryStringData.fromString("{\"key1\":\"value1\"}");
+ expectedSnapshot[44] = BinaryStringData.fromString("{\"key1\": \"value1\"}");
Review Comment:
This additional assignment was meant to fix the format discrepancy and could be removed now.
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java:
##########
Review Comment:
Can we add some tests to verify if we have the same JSON format between snapshot and binlog stage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang merged PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#issuecomment-2568746928
@ruanhang1993 @leonardBang PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1845349530
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceOptions.java:
##########
@@ -272,4 +272,12 @@ public class MySqlDataSourceOptions {
+ "The difference between scan.newly-added-table.enabled and scan.binlog.newly-added-table.enabled options is: \n"
+ "scan.newly-added-table.enabled: do re-snapshot & binlog-reading for newly added table when restored; \n"
+ "scan.binlog.newly-added-table.enabled: only do binlog-reading for newly added table during binlog reading phase.");
+
+ @Experimental
+ public static final ConfigOption<Boolean> USE_LEGACY_JSON_FORMAT =
+ ConfigOptions.key("use.legacy.json.format")
+ .booleanType()
+ .defaultValue(true)
+ .withDescription(
+ "Whether to use legacy json format. The default value is true, which means there is no whitespace before value and after comma in json format.");
Review Comment:
`use.legacy.json.format` option for pipeline connector. default is `true`.
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceOptions.java:
##########
@@ -262,4 +262,12 @@ public class MySqlSourceOptions {
.defaultValue(false)
.withDescription(
"Whether to skip backfill in snapshot reading phase. If backfill is skipped, changes on captured tables during snapshot phase will be consumed later in binlog reading phase instead of being merged into the snapshot. WARNING: Skipping backfill might lead to data inconsistency because some binlog events happened within the snapshot phase might be replayed (only at-least-once semantic is promised). For example updating an already updated value in snapshot, or deleting an already deleted entry in snapshot. These replayed binlog events should be handled specially.");
+
+ @Experimental
+ public static final ConfigOption<Boolean> USE_LEGACY_JSON_FORMAT =
+ ConfigOptions.key("use.legacy.json.format")
+ .booleanType()
+ .defaultValue(true)
+ .withDescription(
+ "Whether to use legacy json format. The default value is true, which means there is no whitespace before value and after comma in json format.");
Review Comment:
`use.legacy.json.format` option for source connector. default is `true`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1845340637
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/json/JsonStringFormatter.java:
##########
Review Comment:
I added `useLegacyJsonFormat` option as a static variable in [MySqlSourceConfig](https://github.com/apache/flink-cdc/blob/337b5de64f58041fff6383f76c103af54b7eec11/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfig.java#L69), and shared it with legcay [MySqlSource](https://github.com/apache/flink-cdc/blob/337b5de64f58041fff6383f76c103af54b7eec11/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/MySqlSource.java#L161-L164) for legacy MySqlSource user.
Also the default value of this option is `true`, so users who want to continue using current format(legacy format) don't need to worry about this new option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1845342619
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java:
##########
Review Comment:
Added test codes for this new option.
https://github.com/apache/flink-cdc/blob/337b5de64f58041fff6383f76c103af54b7eec11/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java#L498-L551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1845347107
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/json/JsonStringFormatter.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.github.shyiko.mysql.binlog.event.deserialization.json;
+
+import com.github.shyiko.mysql.binlog.event.deserialization.ColumnType;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.Base64;
+
+import static org.apache.flink.cdc.connectors.mysql.source.config.MySqlSourceConfig.useLegacyJsonFormat;
+
+/**
+ * Copied from mysql-binlog-connector-java 0.27.2 to add whitespace before value and after comma.
+ *
+ * <p>Line 105: Added whitespace before value, Line 207: Added whitespace after comma
+ */
+public class JsonStringFormatter implements JsonFormatter {
+
+ /**
+ * Value used for lookup tables to indicate that matching characters do not need to be escaped.
+ */
+ private static final int ESCAPE_NONE = 0;
+
+ /**
+ * Value used for lookup tables to indicate that matching characters are to be escaped using
+ * standard escaping; for JSON this means (for example) using "backslash - u" escape method.
+ */
+ private static final int ESCAPE_GENERIC = -1;
+
+ /**
+ * A lookup table that determines which of the first 128 Unicode code points (single-byte UTF-8
+ * characters) must be escaped. A value of '0' means no escaping is required; positive values
+ * must be escaped with a preceding backslash; and negative values that generic escaping (e.g.,
+ */
+ private static final int[] ESCAPES;
+
+ static {
+ int[] escape = new int[128];
+ // Generic escape for control characters ...
+ for (int i = 0; i < 32; ++i) {
+ escape[i] = ESCAPE_GENERIC;
+ }
+ // Backslash escape for other specific characters ...
+ escape['"'] = '"';
+ escape['\\'] = '\\';
+ // Escaping of slash is optional, so let's not add it
+ escape[0x08] = 'b';
+ escape[0x09] = 't';
+ escape[0x0C] = 'f';
+ escape[0x0A] = 'n';
+ escape[0x0D] = 'r';
+ ESCAPES = escape;
+ }
+
+ private static final char[] HEX_CODES = "0123456789ABCDEF".toCharArray();
+
+ private final StringBuilder sb = new StringBuilder();
+
+ @Override
+ public String toString() {
+ return getString();
+ }
+
+ public String getString() {
+ return sb.toString();
+ }
+
+ @Override
+ public void beginObject(int numElements) {
+ sb.append('{');
+ }
+
+ @Override
+ public void beginArray(int numElements) {
+ sb.append('[');
+ }
+
+ @Override
+ public void endObject() {
+ sb.append('}');
+ }
+
+ @Override
+ public void endArray() {
+ sb.append(']');
+ }
+
+ @Override
+ public void name(String name) {
+ sb.append('"');
+ appendString(name);
+ if (useLegacyJsonFormat) {
+ sb.append("\":");
+ } else {
+ sb.append("\": ");
+ }
+ }
+
+ @Override
+ public void value(String value) {
+ sb.append('"');
+ appendString(value);
+ sb.append('"');
+ }
+
+ @Override
+ public void value(int value) {
+ sb.append(Integer.toString(value));
+ }
+
+ @Override
+ public void value(long value) {
+ sb.append(Long.toString(value));
+ }
+
+ @Override
+ public void value(double value) {
+ // Double's toString method will result in scientific notation and loss of precision
+ String str = Double.toString(value);
+ if (str.contains("E")) {
+ value(new BigDecimal(value));
+ } else {
+ sb.append(str);
+ }
+ }
+
+ @Override
+ public void value(BigInteger value) {
+ // Using the BigInteger.toString() method will result in scientific notation, so instead ...
+ value(new BigDecimal(value));
+ }
+
+ @Override
+ public void value(BigDecimal value) {
+ // Using the BigInteger.toString() method will result in scientific notation, so instead ...
+ sb.append(value.toPlainString());
+ }
+
+ @Override
+ public void value(boolean value) {
+ sb.append(Boolean.toString(value));
+ }
+
+ @Override
+ public void valueNull() {
+ sb.append("null");
+ }
+
+ @Override
+ public void valueYear(int year) {
+ sb.append(year);
+ }
+
+ @Override
+ public void valueDate(int year, int month, int day) {
+ sb.append('"');
+ appendDate(year, month, day);
+ sb.append('"');
+ }
+
+ @Override
+ // checkstyle, please ignore ParameterNumber for the next line
+ public void valueDatetime(
+ int year, int month, int day, int hour, int min, int sec, int microSeconds) {
+ sb.append('"');
+ appendDate(year, month, day);
+ sb.append(' ');
+ appendTime(hour, min, sec, microSeconds);
+ sb.append('"');
+ }
+
+ @Override
+ public void valueTime(int hour, int min, int sec, int microSeconds) {
+ sb.append('"');
+ if (hour < 0) {
+ sb.append('-');
+ hour = Math.abs(hour);
+ }
+ appendTime(hour, min, sec, microSeconds);
+ sb.append('"');
+ }
+
+ @Override
+ public void valueTimestamp(long secondsPastEpoch, int microSeconds) {
+ sb.append(secondsPastEpoch);
+ appendSixDigitUnsignedInt(microSeconds, false);
+ }
+
+ @Override
+ public void valueOpaque(ColumnType type, byte[] value) {
+ sb.append('"');
+ sb.append(Base64.getEncoder().encodeToString(value));
+ sb.append('"');
+ }
+
+ @Override
+ public void nextEntry() {
+ if (useLegacyJsonFormat) {
+ sb.append(",");
+ } else {
+ sb.append(", ");
+ }
Review Comment:
Now we can choose json type data format
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/json/JsonStringFormatter.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.github.shyiko.mysql.binlog.event.deserialization.json;
+
+import com.github.shyiko.mysql.binlog.event.deserialization.ColumnType;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.Base64;
+
+import static org.apache.flink.cdc.connectors.mysql.source.config.MySqlSourceConfig.useLegacyJsonFormat;
+
+/**
+ * Copied from mysql-binlog-connector-java 0.27.2 to add whitespace before value and after comma.
+ *
+ * <p>Line 105: Added whitespace before value, Line 207: Added whitespace after comma
+ */
+public class JsonStringFormatter implements JsonFormatter {
+
+ /**
+ * Value used for lookup tables to indicate that matching characters do not need to be escaped.
+ */
+ private static final int ESCAPE_NONE = 0;
+
+ /**
+ * Value used for lookup tables to indicate that matching characters are to be escaped using
+ * standard escaping; for JSON this means (for example) using "backslash - u" escape method.
+ */
+ private static final int ESCAPE_GENERIC = -1;
+
+ /**
+ * A lookup table that determines which of the first 128 Unicode code points (single-byte UTF-8
+ * characters) must be escaped. A value of '0' means no escaping is required; positive values
+ * must be escaped with a preceding backslash; and negative values that generic escaping (e.g.,
+ */
+ private static final int[] ESCAPES;
+
+ static {
+ int[] escape = new int[128];
+ // Generic escape for control characters ...
+ for (int i = 0; i < 32; ++i) {
+ escape[i] = ESCAPE_GENERIC;
+ }
+ // Backslash escape for other specific characters ...
+ escape['"'] = '"';
+ escape['\\'] = '\\';
+ // Escaping of slash is optional, so let's not add it
+ escape[0x08] = 'b';
+ escape[0x09] = 't';
+ escape[0x0C] = 'f';
+ escape[0x0A] = 'n';
+ escape[0x0D] = 'r';
+ ESCAPES = escape;
+ }
+
+ private static final char[] HEX_CODES = "0123456789ABCDEF".toCharArray();
+
+ private final StringBuilder sb = new StringBuilder();
+
+ @Override
+ public String toString() {
+ return getString();
+ }
+
+ public String getString() {
+ return sb.toString();
+ }
+
+ @Override
+ public void beginObject(int numElements) {
+ sb.append('{');
+ }
+
+ @Override
+ public void beginArray(int numElements) {
+ sb.append('[');
+ }
+
+ @Override
+ public void endObject() {
+ sb.append('}');
+ }
+
+ @Override
+ public void endArray() {
+ sb.append(']');
+ }
+
+ @Override
+ public void name(String name) {
+ sb.append('"');
+ appendString(name);
+ if (useLegacyJsonFormat) {
+ sb.append("\":");
+ } else {
+ sb.append("\": ");
+ }
Review Comment:
Now we can choose json type data format
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceOptions.java:
##########
@@ -262,4 +262,12 @@ public class MySqlSourceOptions {
.defaultValue(false)
.withDescription(
"Whether to skip backfill in snapshot reading phase. If backfill is skipped, changes on captured tables during snapshot phase will be consumed later in binlog reading phase instead of being merged into the snapshot. WARNING: Skipping backfill might lead to data inconsistency because some binlog events happened within the snapshot phase might be replayed (only at-least-once semantic is promised). For example updating an already updated value in snapshot, or deleting an already deleted entry in snapshot. These replayed binlog events should be handled specially.");
+
+ @Experimental
+ public static final ConfigOption<Boolean> USE_LEGACY_JSON_FORMAT =
+ ConfigOptions.key("use.legacy.json.format")
+ .booleanType()
+ .defaultValue(true)
+ .withDescription(
+ "Whether to use legacy json format. The default value is true, which means there is no whitespace before value and after comma in json format.");
Review Comment:
`use.legacy.json.format` option for source connector
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/source/config/MySqlSourceConfigFactory.java:
##########
@@ -276,6 +277,15 @@ public MySqlSourceConfigFactory skipSnapshotBackfill(boolean skipSnapshotBackfil
return this;
}
+ /**
+ * Whether to use legacy json format. The default value is true, which means there is no
+ * whitespace before value and after comma in json format.
+ */
+ public MySqlSourceConfigFactory useLegacyJsonFormat(boolean useLegacyJsonFormat) {
+ this.useLegacyJsonFormat = useLegacyJsonFormat;
+ return this;
+ }
+
Review Comment:
setter for `useLegacyJsonFormat` option
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/resources/ddl/column_type_test.sql:
##########
@@ -180,4 +180,22 @@ VALUES (DEFAULT,
23,
29,
31,
- 37);
\ No newline at end of file
+ 37);
+
+CREATE TABLE json_types
+(
+ id SERIAL,
+ json_c0 JSON,
+ json_c1 JSON,
+ json_c2 JSON,
+ int_c INTEGER,
+ PRIMARY KEY (id)
+) DEFAULT CHARSET=utf8;
+
+INSERT INTO json_types
+VALUES (DEFAULT,
+ '{"key1":"value1"}',
+ '{"key1":"value1","key2":"value2"}',
+ '[{"key1":"value1","key2":{"key2_1":"value2_1","key2_2":"value2_2"},"key3":["value3"],"key4":["value4_1","value4_2"]},{"key5":"value5"}]',
+ 1
+ );
Review Comment:
test table for useLegacyJsonFormat option
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/MySqlDataSourceOptions.java:
##########
@@ -272,4 +272,12 @@ public class MySqlDataSourceOptions {
+ "The difference between scan.newly-added-table.enabled and scan.binlog.newly-added-table.enabled options is: \n"
+ "scan.newly-added-table.enabled: do re-snapshot & binlog-reading for newly added table when restored; \n"
+ "scan.binlog.newly-added-table.enabled: only do binlog-reading for newly added table during binlog reading phase.");
+
+ @Experimental
+ public static final ConfigOption<Boolean> USE_LEGACY_JSON_FORMAT =
+ ConfigOptions.key("use.legacy.json.format")
+ .booleanType()
+ .defaultValue(true)
+ .withDescription(
+ "Whether to use legacy json format. The default value is true, which means there is no whitespace before value and after comma in json format.");
Review Comment:
`use.legacy.json.format` option for pipeline connector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1861726630
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java:
##########
@@ -457,6 +495,61 @@ private void testCommonDataTypes(UniqueDatabase database) throws Exception {
.isEqualTo(expectedStreamRecord);
}
+ private void testJsonDataType(UniqueDatabase database, Boolean useLegacyJsonFormat)
+ throws Exception {
+ database.createAndInitialize();
+ CloseableIterator<Event> iterator =
+ env.fromSource(
+ getFlinkSourceProvider(
+ new String[] {"json_types"},
+ database,
+ useLegacyJsonFormat)
+ .getSource(),
+ WatermarkStrategy.noWatermarks(),
+ "Event-Source")
+ .executeAndCollect();
+
+ Object[] expectedSnapshot =
+ new Object[] {
+ DecimalData.fromBigDecimal(new BigDecimal("1"), 20, 0),
+ BinaryStringData.fromString("{\"key1\": \"value1\"}"),
+ BinaryStringData.fromString("{\"key1\": \"value1\", \"key2\": \"value2\"}"),
+ BinaryStringData.fromString(
+ "[{\"key1\": \"value1\", \"key2\": {\"key2_1\": \"value2_1\", \"key2_2\": \"value2_2\"}, \"key3\": [\"value3\"], \"key4\": [\"value4_1\", \"value4_2\"]}, {\"key5\": \"value5\"}]"),
+ 1
+ };
+
+ // skip CreateTableEvent
+ List<Event> snapshotResults =
+ MySqSourceTestUtils.fetchResultsAndCreateTableEvent(iterator, 1).f0;
+ RecordData snapshotRecord = ((DataChangeEvent) snapshotResults.get(0)).after();
+ Assertions.assertThat(RecordDataTestUtils.recordFields(snapshotRecord, JSON_TYPES))
+ .isEqualTo(expectedSnapshot);
+
+ try (Connection connection = database.getJdbcConnection();
+ Statement statement = connection.createStatement()) {
+ statement.execute("UPDATE json_types SET int_c = null WHERE id = 1;");
+ }
+
+ Object[] expectedStreamRecord = expectedSnapshot;
+
+ if (useLegacyJsonFormat) {
+ expectedSnapshot[1] = BinaryStringData.fromString("{\"key1\":\"value1\"}");
+ expectedSnapshot[2] =
+ BinaryStringData.fromString("{\"key1\":\"value1\",\"key2\":\"value2\"}");
+ expectedSnapshot[3] =
+ BinaryStringData.fromString(
+ "[{\"key1\":\"value1\",\"key2\":{\"key2_1\":\"value2_1\",\"key2_2\":\"value2_2\"},\"key3\":[\"value3\"],\"key4\":[\"value4_1\",\"value4_2\"]},{\"key5\":\"value5\"}]");
+ }
+ expectedSnapshot[4] = null;
Review Comment:
I revised it more clearly as shown below.
```java
if (useLegacyJsonFormat) {
// removed whitespace before value and after comma in json format string value
Assertions.assertThat(RecordDataTestUtils.recordFields(streamRecord, JSON_TYPES))
.containsExactly(
DecimalData.fromBigDecimal(new BigDecimal("1"), 20, 0),
BinaryStringData.fromString("{\"key1\":\"value1\"}"),
BinaryStringData.fromString(
"{\"key1\":\"value1\",\"key2\":\"value2\"}"),
BinaryStringData.fromString(
"[{\"key1\":\"value1\",\"key2\":{\"key2_1\":\"value2_1\",\"key2_2\":\"value2_2\"},\"key3\":[\"value3\"],\"key4\":[\"value4_1\",\"value4_2\"]},{\"key5\":\"value5\"}]"),
null);
} else {
Assertions.assertThat(RecordDataTestUtils.recordFields(streamRecord, JSON_TYPES))
.containsExactly(expectedStreamRecord);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "leonardBang (via GitHub)" <gi...@apache.org>.
leonardBang commented on PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#issuecomment-2595978663
The MySQL related test job failed due to timeout, I append one commit to set a proper timeout for the most time-consuming step
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1845347058
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/json/JsonStringFormatter.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.github.shyiko.mysql.binlog.event.deserialization.json;
+
+import com.github.shyiko.mysql.binlog.event.deserialization.ColumnType;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.Base64;
+
+import static org.apache.flink.cdc.connectors.mysql.source.config.MySqlSourceConfig.useLegacyJsonFormat;
+
+/**
+ * Copied from mysql-binlog-connector-java 0.27.2 to add whitespace before value and after comma.
+ *
+ * <p>Line 105: Added whitespace before value, Line 207: Added whitespace after comma
+ */
+public class JsonStringFormatter implements JsonFormatter {
+
+ /**
+ * Value used for lookup tables to indicate that matching characters do not need to be escaped.
+ */
+ private static final int ESCAPE_NONE = 0;
+
+ /**
+ * Value used for lookup tables to indicate that matching characters are to be escaped using
+ * standard escaping; for JSON this means (for example) using "backslash - u" escape method.
+ */
+ private static final int ESCAPE_GENERIC = -1;
+
+ /**
+ * A lookup table that determines which of the first 128 Unicode code points (single-byte UTF-8
+ * characters) must be escaped. A value of '0' means no escaping is required; positive values
+ * must be escaped with a preceding backslash; and negative values that generic escaping (e.g.,
+ */
+ private static final int[] ESCAPES;
+
+ static {
+ int[] escape = new int[128];
+ // Generic escape for control characters ...
+ for (int i = 0; i < 32; ++i) {
+ escape[i] = ESCAPE_GENERIC;
+ }
+ // Backslash escape for other specific characters ...
+ escape['"'] = '"';
+ escape['\\'] = '\\';
+ // Escaping of slash is optional, so let's not add it
+ escape[0x08] = 'b';
+ escape[0x09] = 't';
+ escape[0x0C] = 'f';
+ escape[0x0A] = 'n';
+ escape[0x0D] = 'r';
+ ESCAPES = escape;
+ }
+
+ private static final char[] HEX_CODES = "0123456789ABCDEF".toCharArray();
+
+ private final StringBuilder sb = new StringBuilder();
+
+ @Override
+ public String toString() {
+ return getString();
+ }
+
+ public String getString() {
+ return sb.toString();
+ }
+
+ @Override
+ public void beginObject(int numElements) {
+ sb.append('{');
+ }
+
+ @Override
+ public void beginArray(int numElements) {
+ sb.append('[');
+ }
+
+ @Override
+ public void endObject() {
+ sb.append('}');
+ }
+
+ @Override
+ public void endArray() {
+ sb.append(']');
+ }
+
+ @Override
+ public void name(String name) {
+ sb.append('"');
+ appendString(name);
+ if (useLegacyJsonFormat) {
+ sb.append("\":");
+ } else {
+ sb.append("\": ");
+ }
Review Comment:
Now we can choose JSON type data format (whitespace in before value)
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/json/JsonStringFormatter.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.github.shyiko.mysql.binlog.event.deserialization.json;
+
+import com.github.shyiko.mysql.binlog.event.deserialization.ColumnType;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.Base64;
+
+import static org.apache.flink.cdc.connectors.mysql.source.config.MySqlSourceConfig.useLegacyJsonFormat;
+
+/**
+ * Copied from mysql-binlog-connector-java 0.27.2 to add whitespace before value and after comma.
+ *
+ * <p>Line 105: Added whitespace before value, Line 207: Added whitespace after comma
+ */
+public class JsonStringFormatter implements JsonFormatter {
+
+ /**
+ * Value used for lookup tables to indicate that matching characters do not need to be escaped.
+ */
+ private static final int ESCAPE_NONE = 0;
+
+ /**
+ * Value used for lookup tables to indicate that matching characters are to be escaped using
+ * standard escaping; for JSON this means (for example) using "backslash - u" escape method.
+ */
+ private static final int ESCAPE_GENERIC = -1;
+
+ /**
+ * A lookup table that determines which of the first 128 Unicode code points (single-byte UTF-8
+ * characters) must be escaped. A value of '0' means no escaping is required; positive values
+ * must be escaped with a preceding backslash; and negative values that generic escaping (e.g.,
+ */
+ private static final int[] ESCAPES;
+
+ static {
+ int[] escape = new int[128];
+ // Generic escape for control characters ...
+ for (int i = 0; i < 32; ++i) {
+ escape[i] = ESCAPE_GENERIC;
+ }
+ // Backslash escape for other specific characters ...
+ escape['"'] = '"';
+ escape['\\'] = '\\';
+ // Escaping of slash is optional, so let's not add it
+ escape[0x08] = 'b';
+ escape[0x09] = 't';
+ escape[0x0C] = 'f';
+ escape[0x0A] = 'n';
+ escape[0x0D] = 'r';
+ ESCAPES = escape;
+ }
+
+ private static final char[] HEX_CODES = "0123456789ABCDEF".toCharArray();
+
+ private final StringBuilder sb = new StringBuilder();
+
+ @Override
+ public String toString() {
+ return getString();
+ }
+
+ public String getString() {
+ return sb.toString();
+ }
+
+ @Override
+ public void beginObject(int numElements) {
+ sb.append('{');
+ }
+
+ @Override
+ public void beginArray(int numElements) {
+ sb.append('[');
+ }
+
+ @Override
+ public void endObject() {
+ sb.append('}');
+ }
+
+ @Override
+ public void endArray() {
+ sb.append(']');
+ }
+
+ @Override
+ public void name(String name) {
+ sb.append('"');
+ appendString(name);
+ if (useLegacyJsonFormat) {
+ sb.append("\":");
+ } else {
+ sb.append("\": ");
+ }
+ }
+
+ @Override
+ public void value(String value) {
+ sb.append('"');
+ appendString(value);
+ sb.append('"');
+ }
+
+ @Override
+ public void value(int value) {
+ sb.append(Integer.toString(value));
+ }
+
+ @Override
+ public void value(long value) {
+ sb.append(Long.toString(value));
+ }
+
+ @Override
+ public void value(double value) {
+ // Double's toString method will result in scientific notation and loss of precision
+ String str = Double.toString(value);
+ if (str.contains("E")) {
+ value(new BigDecimal(value));
+ } else {
+ sb.append(str);
+ }
+ }
+
+ @Override
+ public void value(BigInteger value) {
+ // Using the BigInteger.toString() method will result in scientific notation, so instead ...
+ value(new BigDecimal(value));
+ }
+
+ @Override
+ public void value(BigDecimal value) {
+ // Using the BigInteger.toString() method will result in scientific notation, so instead ...
+ sb.append(value.toPlainString());
+ }
+
+ @Override
+ public void value(boolean value) {
+ sb.append(Boolean.toString(value));
+ }
+
+ @Override
+ public void valueNull() {
+ sb.append("null");
+ }
+
+ @Override
+ public void valueYear(int year) {
+ sb.append(year);
+ }
+
+ @Override
+ public void valueDate(int year, int month, int day) {
+ sb.append('"');
+ appendDate(year, month, day);
+ sb.append('"');
+ }
+
+ @Override
+ // checkstyle, please ignore ParameterNumber for the next line
+ public void valueDatetime(
+ int year, int month, int day, int hour, int min, int sec, int microSeconds) {
+ sb.append('"');
+ appendDate(year, month, day);
+ sb.append(' ');
+ appendTime(hour, min, sec, microSeconds);
+ sb.append('"');
+ }
+
+ @Override
+ public void valueTime(int hour, int min, int sec, int microSeconds) {
+ sb.append('"');
+ if (hour < 0) {
+ sb.append('-');
+ hour = Math.abs(hour);
+ }
+ appendTime(hour, min, sec, microSeconds);
+ sb.append('"');
+ }
+
+ @Override
+ public void valueTimestamp(long secondsPastEpoch, int microSeconds) {
+ sb.append(secondsPastEpoch);
+ appendSixDigitUnsignedInt(microSeconds, false);
+ }
+
+ @Override
+ public void valueOpaque(ColumnType type, byte[] value) {
+ sb.append('"');
+ sb.append(Base64.getEncoder().encodeToString(value));
+ sb.append('"');
+ }
+
+ @Override
+ public void nextEntry() {
+ if (useLegacyJsonFormat) {
+ sb.append(",");
+ } else {
+ sb.append(", ");
+ }
Review Comment:
Now we can choose JSON type data format (whitespace in after comma)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1861726630
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java:
##########
@@ -457,6 +495,61 @@ private void testCommonDataTypes(UniqueDatabase database) throws Exception {
.isEqualTo(expectedStreamRecord);
}
+ private void testJsonDataType(UniqueDatabase database, Boolean useLegacyJsonFormat)
+ throws Exception {
+ database.createAndInitialize();
+ CloseableIterator<Event> iterator =
+ env.fromSource(
+ getFlinkSourceProvider(
+ new String[] {"json_types"},
+ database,
+ useLegacyJsonFormat)
+ .getSource(),
+ WatermarkStrategy.noWatermarks(),
+ "Event-Source")
+ .executeAndCollect();
+
+ Object[] expectedSnapshot =
+ new Object[] {
+ DecimalData.fromBigDecimal(new BigDecimal("1"), 20, 0),
+ BinaryStringData.fromString("{\"key1\": \"value1\"}"),
+ BinaryStringData.fromString("{\"key1\": \"value1\", \"key2\": \"value2\"}"),
+ BinaryStringData.fromString(
+ "[{\"key1\": \"value1\", \"key2\": {\"key2_1\": \"value2_1\", \"key2_2\": \"value2_2\"}, \"key3\": [\"value3\"], \"key4\": [\"value4_1\", \"value4_2\"]}, {\"key5\": \"value5\"}]"),
+ 1
+ };
+
+ // skip CreateTableEvent
+ List<Event> snapshotResults =
+ MySqSourceTestUtils.fetchResultsAndCreateTableEvent(iterator, 1).f0;
+ RecordData snapshotRecord = ((DataChangeEvent) snapshotResults.get(0)).after();
+ Assertions.assertThat(RecordDataTestUtils.recordFields(snapshotRecord, JSON_TYPES))
+ .isEqualTo(expectedSnapshot);
+
+ try (Connection connection = database.getJdbcConnection();
+ Statement statement = connection.createStatement()) {
+ statement.execute("UPDATE json_types SET int_c = null WHERE id = 1;");
+ }
+
+ Object[] expectedStreamRecord = expectedSnapshot;
+
+ if (useLegacyJsonFormat) {
+ expectedSnapshot[1] = BinaryStringData.fromString("{\"key1\":\"value1\"}");
+ expectedSnapshot[2] =
+ BinaryStringData.fromString("{\"key1\":\"value1\",\"key2\":\"value2\"}");
+ expectedSnapshot[3] =
+ BinaryStringData.fromString(
+ "[{\"key1\":\"value1\",\"key2\":{\"key2_1\":\"value2_1\",\"key2_2\":\"value2_2\"},\"key3\":[\"value3\"],\"key4\":[\"value4_1\",\"value4_2\"]},{\"key5\":\"value5\"}]");
+ }
+ expectedSnapshot[4] = null;
Review Comment:
I revised it more clearly as shown below.
```java
if (useLegacyJsonFormat) {
// removed whitespace before value and after comma in json format string value
Assertions.assertThat(RecordDataTestUtils.recordFields(streamRecord, JSON_TYPES))
.containsExactly(
DecimalData.fromBigDecimal(new BigDecimal("1"), 20, 0),
BinaryStringData.fromString("{\"key1\":\"value1\"}"),
BinaryStringData.fromString(
"{\"key1\":\"value1\",\"key2\":\"value2\"}"),
BinaryStringData.fromString(
"[{\"key1\":\"value1\",\"key2\":{\"key2_1\":\"value2_1\",\"key2_2\":\"value2_2\"},\"key3\":[\"value3\"],\"key4\":[\"value4_1\",\"value4_2\"]},{\"key5\":\"value5\"}]"),
null);
} else {
Assertions.assertThat(RecordDataTestUtils.recordFields(streamRecord, JSON_TYPES))
.containsExactly(expectedStreamRecord);
}
```
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/json/JsonStringFormatter.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.github.shyiko.mysql.binlog.event.deserialization.json;
+
+import com.github.shyiko.mysql.binlog.event.deserialization.ColumnType;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.Base64;
+
+import static org.apache.flink.cdc.connectors.mysql.source.config.MySqlSourceConfig.useLegacyJsonFormat;
+
+/**
+ * Copied from mysql-binlog-connector-java 0.27.2 to add whitespace before value and after comma.
+ *
+ * <p>Line 105: Added whitespace before value, Line 207: Added whitespace after comma
Review Comment:
resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#issuecomment-2550062723
@yuxiqian @ruanhang1993 could you please provide progress on this pr? thank you 🙇🏻♂️🙇🏻♂️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][mysql] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#issuecomment-2428093914
@leonardBang @PatrickRen PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1845347447
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/resources/ddl/column_type_test.sql:
##########
@@ -180,4 +180,22 @@ VALUES (DEFAULT,
23,
29,
31,
- 37);
\ No newline at end of file
+ 37);
+
+CREATE TABLE json_types
+(
+ id SERIAL,
+ json_c0 JSON,
+ json_c1 JSON,
+ json_c2 JSON,
+ int_c INTEGER,
+ PRIMARY KEY (id)
+) DEFAULT CHARSET=utf8;
+
+INSERT INTO json_types
+VALUES (DEFAULT,
+ '{"key1":"value1"}',
+ '{"key1":"value1","key2":"value2"}',
+ '[{"key1":"value1","key2":{"key2_1":"value2_1","key2_2":"value2_2"},"key3":["value3"],"key4":["value4_1","value4_2"]},{"key5":"value5"}]',
+ 1
+ );
Review Comment:
test table for various JSON format data
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1861728840
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/resources/ddl/column_type_test_mysql8.sql:
##########
Review Comment:
it was my mistake...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1861766911
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/table/MySqlTableSourceFactory.java:
##########
Review Comment:
I will add integrated test cases if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1845342619
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java:
##########
Review Comment:
I added tests for this new option. When `useLegacyJsonFormat = flase`, it verifies that the JSON type data format is the same between snapshot stage and binlog stage.
https://github.com/apache/flink-cdc/blob/337b5de64f58041fff6383f76c103af54b7eec11/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java#L498-L551
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1845337807
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java:
##########
@@ -412,7 +412,7 @@ private void testCommonDataTypes(UniqueDatabase database) throws Exception {
expectedSnapshot[30] = null;
// The json string from binlog will remove useless space
- expectedSnapshot[44] = BinaryStringData.fromString("{\"key1\":\"value1\"}");
+ expectedSnapshot[44] = BinaryStringData.fromString("{\"key1\": \"value1\"}");
Review Comment:
Resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Add table information of binlog offsets when checkpointing added modified JsonStringFormatter [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1845347058
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/json/JsonStringFormatter.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.github.shyiko.mysql.binlog.event.deserialization.json;
+
+import com.github.shyiko.mysql.binlog.event.deserialization.ColumnType;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.Base64;
+
+import static org.apache.flink.cdc.connectors.mysql.source.config.MySqlSourceConfig.useLegacyJsonFormat;
+
+/**
+ * Copied from mysql-binlog-connector-java 0.27.2 to add whitespace before value and after comma.
+ *
+ * <p>Line 105: Added whitespace before value, Line 207: Added whitespace after comma
+ */
+public class JsonStringFormatter implements JsonFormatter {
+
+ /**
+ * Value used for lookup tables to indicate that matching characters do not need to be escaped.
+ */
+ private static final int ESCAPE_NONE = 0;
+
+ /**
+ * Value used for lookup tables to indicate that matching characters are to be escaped using
+ * standard escaping; for JSON this means (for example) using "backslash - u" escape method.
+ */
+ private static final int ESCAPE_GENERIC = -1;
+
+ /**
+ * A lookup table that determines which of the first 128 Unicode code points (single-byte UTF-8
+ * characters) must be escaped. A value of '0' means no escaping is required; positive values
+ * must be escaped with a preceding backslash; and negative values that generic escaping (e.g.,
+ */
+ private static final int[] ESCAPES;
+
+ static {
+ int[] escape = new int[128];
+ // Generic escape for control characters ...
+ for (int i = 0; i < 32; ++i) {
+ escape[i] = ESCAPE_GENERIC;
+ }
+ // Backslash escape for other specific characters ...
+ escape['"'] = '"';
+ escape['\\'] = '\\';
+ // Escaping of slash is optional, so let's not add it
+ escape[0x08] = 'b';
+ escape[0x09] = 't';
+ escape[0x0C] = 'f';
+ escape[0x0A] = 'n';
+ escape[0x0D] = 'r';
+ ESCAPES = escape;
+ }
+
+ private static final char[] HEX_CODES = "0123456789ABCDEF".toCharArray();
+
+ private final StringBuilder sb = new StringBuilder();
+
+ @Override
+ public String toString() {
+ return getString();
+ }
+
+ public String getString() {
+ return sb.toString();
+ }
+
+ @Override
+ public void beginObject(int numElements) {
+ sb.append('{');
+ }
+
+ @Override
+ public void beginArray(int numElements) {
+ sb.append('[');
+ }
+
+ @Override
+ public void endObject() {
+ sb.append('}');
+ }
+
+ @Override
+ public void endArray() {
+ sb.append(']');
+ }
+
+ @Override
+ public void name(String name) {
+ sb.append('"');
+ appendString(name);
+ if (useLegacyJsonFormat) {
+ sb.append("\":");
+ } else {
+ sb.append("\": ");
+ }
Review Comment:
Now we can choose JSON type data format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#discussion_r1860226446
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/resources/ddl/column_type_test_mysql8.sql:
##########
Review Comment:
Accidental change?
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/com/github/shyiko/mysql/binlog/event/deserialization/json/JsonStringFormatter.java:
##########
@@ -0,0 +1,337 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.github.shyiko.mysql.binlog.event.deserialization.json;
+
+import com.github.shyiko.mysql.binlog.event.deserialization.ColumnType;
+
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.util.Base64;
+
+import static org.apache.flink.cdc.connectors.mysql.source.config.MySqlSourceConfig.useLegacyJsonFormat;
+
+/**
+ * Copied from mysql-binlog-connector-java 0.27.2 to add whitespace before value and after comma.
+ *
+ * <p>Line 105: Added whitespace before value, Line 207: Added whitespace after comma
Review Comment:
Seems line numbers have been differed from actual changes.
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/table/MySqlTableSourceFactory.java:
##########
Review Comment:
Since DataStream / Table source connectors' behavior also changed, corresponding integrated test cases might be necessary.
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlFullTypesITCase.java:
##########
@@ -457,6 +495,61 @@ private void testCommonDataTypes(UniqueDatabase database) throws Exception {
.isEqualTo(expectedStreamRecord);
}
+ private void testJsonDataType(UniqueDatabase database, Boolean useLegacyJsonFormat)
+ throws Exception {
+ database.createAndInitialize();
+ CloseableIterator<Event> iterator =
+ env.fromSource(
+ getFlinkSourceProvider(
+ new String[] {"json_types"},
+ database,
+ useLegacyJsonFormat)
+ .getSource(),
+ WatermarkStrategy.noWatermarks(),
+ "Event-Source")
+ .executeAndCollect();
+
+ Object[] expectedSnapshot =
+ new Object[] {
+ DecimalData.fromBigDecimal(new BigDecimal("1"), 20, 0),
+ BinaryStringData.fromString("{\"key1\": \"value1\"}"),
+ BinaryStringData.fromString("{\"key1\": \"value1\", \"key2\": \"value2\"}"),
+ BinaryStringData.fromString(
+ "[{\"key1\": \"value1\", \"key2\": {\"key2_1\": \"value2_1\", \"key2_2\": \"value2_2\"}, \"key3\": [\"value3\"], \"key4\": [\"value4_1\", \"value4_2\"]}, {\"key5\": \"value5\"}]"),
+ 1
+ };
+
+ // skip CreateTableEvent
+ List<Event> snapshotResults =
+ MySqSourceTestUtils.fetchResultsAndCreateTableEvent(iterator, 1).f0;
+ RecordData snapshotRecord = ((DataChangeEvent) snapshotResults.get(0)).after();
+ Assertions.assertThat(RecordDataTestUtils.recordFields(snapshotRecord, JSON_TYPES))
+ .isEqualTo(expectedSnapshot);
+
+ try (Connection connection = database.getJdbcConnection();
+ Statement statement = connection.createStatement()) {
+ statement.execute("UPDATE json_types SET int_c = null WHERE id = 1;");
+ }
+
+ Object[] expectedStreamRecord = expectedSnapshot;
+
+ if (useLegacyJsonFormat) {
+ expectedSnapshot[1] = BinaryStringData.fromString("{\"key1\":\"value1\"}");
+ expectedSnapshot[2] =
+ BinaryStringData.fromString("{\"key1\":\"value1\",\"key2\":\"value2\"}");
+ expectedSnapshot[3] =
+ BinaryStringData.fromString(
+ "[{\"key1\":\"value1\",\"key2\":{\"key2_1\":\"value2_1\",\"key2_2\":\"value2_2\"},\"key3\":[\"value3\"],\"key4\":[\"value4_1\",\"value4_2\"]},{\"key5\":\"value5\"}]");
+ }
+ expectedSnapshot[4] = null;
Review Comment:
Modifying expected data is a little confusing. Will something like
```java
if (useLegacyJsonFormat) {
assertThat(RecordDataTestUtils.recordFields(...))
.containsExactly(
BinaryStringData.fromString("{\"key1\":\"value1\"}"),
BinaryStringData.fromString("{\"key2\":\"value2\"}"),
...
);
} else {
assertThat(RecordDataTestUtils.recordFields(...))
.containsExactly(...);
}
```
be clearer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-36578][MySQL] Added modified JsonStringFormatter and option use.legacy.json.format [flink-cdc]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on PR #3658:
URL: https://github.com/apache/flink-cdc/pull/3658#issuecomment-2538529346
@ruanhang1993 PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org