You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "wolfboys (via GitHub)" <gi...@apache.org> on 2025/03/18 13:33:19 UTC
[PR] [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS [flink-cdc]
wolfboys opened a new pull request, #3955:
URL: https://github.com/apache/flink-cdc/pull/3955
## [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS
In newer versions of MySQL, the `SHOW MASTER STATUS` syntax has been deprecated and replaced with [show binary log status](https://dev.mysql.com/doc/refman/8.4/en/show-master-status.html). Since this syntax is used in MySQL CDC, compatibility adjustments are required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on PR #3955:
URL: https://github.com/apache/flink-cdc/pull/3955#issuecomment-3003411407
Could @wolfboys please rebase this PR with latest `master` branch?
I just created a PR https://github.com/wolfboys/flink-cdc/pull/1 to run UT test cases in multiple MySQL versions, please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS [flink-cdc]
Posted by "yuxiqian (via GitHub)" <gi...@apache.org>.
yuxiqian commented on code in PR #3955:
URL: https://github.com/apache/flink-cdc/pull/3955#discussion_r2001368371
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/debezium/DebeziumUtils.java:
##########
@@ -118,33 +121,57 @@ public static MySqlDatabaseSchema createMySqlDatabaseSchema(
/** Fetch current binlog offsets in MySql Server. */
public static BinlogOffset currentBinlogOffset(JdbcConnection jdbc) {
- final String showMasterStmt = "SHOW MASTER STATUS";
try {
- return jdbc.queryAndMap(
- showMasterStmt,
+ return queryBinlogStatus(
+ jdbc,
rs -> {
- if (rs.next()) {
- final String binlogFilename = rs.getString(1);
- final long binlogPosition = rs.getLong(2);
- final String gtidSet =
- rs.getMetaData().getColumnCount() > 4 ? rs.getString(5) : null;
- return BinlogOffset.builder()
- .setBinlogFilePosition(binlogFilename, binlogPosition)
- .setGtidSet(gtidSet)
- .build();
- } else {
- throw new FlinkRuntimeException(
- "Cannot read the binlog filename and position via '"
- + showMasterStmt
- + "'. Make sure your server is correctly configured");
+ try {
+ if (rs.next()) {
+ final String binlogFilename = rs.getString(1);
+ final long binlogPosition = rs.getLong(2);
+ final String gtidSet =
+ rs.getMetaData().getColumnCount() > 4
+ ? rs.getString(5)
+ : null;
+ return BinlogOffset.builder()
+ .setBinlogFilePosition(binlogFilename, binlogPosition)
+ .setGtidSet(gtidSet)
+ .build();
+ } else {
+ throw new ConfigurationException(
+ "Cannot read the binlog filename and position. Make sure your server is correctly configured");
+ }
+ } catch (Exception e) {
+ throw new FlinkRuntimeException(e);
}
});
} catch (SQLException e) {
- throw new FlinkRuntimeException(
- "Cannot read the binlog filename and position via '"
- + showMasterStmt
- + "'. Make sure your server is correctly configured",
- e);
+ throw new FlinkRuntimeException(e);
+ }
+ }
+
+ public static <T> T queryBinlogStatus(
+ JdbcConnection connection, Function<ResultSet, T> callback) throws SQLException {
+ String showMasterStmt = "SHOW MASTER STATUS";
Review Comment:
Could we detect the MySQL version just once, and avoid the trial-and-error check every time?
##########
docs/content.zh/docs/faq/faq.md:
##########
Review Comment:
MySQL 8.4+ support would be great, but validation is still required. Shall we parameterize MySQL UT & E2e test cases with multiple LTS versions (including 5.7, 8.0, 8.4), just like MongoDB CDC?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS [flink-cdc]
Posted by "joyCurry30 (via GitHub)" <gi...@apache.org>.
joyCurry30 commented on PR #3955:
URL: https://github.com/apache/flink-cdc/pull/3955#issuecomment-2753368189
During the Source initialization, we've included a MySQL version check. Could we leverage this phase to determine which SQL syntax variant to use?
<img width="1130" alt="IntelliJ IDEA 2025-03-26 14 25 31" src="https://github.com/user-attachments/assets/da67b626-1f36-4bfc-9cbd-2f1ddeffa49b" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS [flink-cdc]
Posted by "wolfboys (via GitHub)" <gi...@apache.org>.
wolfboys commented on code in PR #3955:
URL: https://github.com/apache/flink-cdc/pull/3955#discussion_r2001483266
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/debezium/DebeziumUtils.java:
##########
@@ -118,33 +121,57 @@ public static MySqlDatabaseSchema createMySqlDatabaseSchema(
/** Fetch current binlog offsets in MySql Server. */
public static BinlogOffset currentBinlogOffset(JdbcConnection jdbc) {
- final String showMasterStmt = "SHOW MASTER STATUS";
try {
- return jdbc.queryAndMap(
- showMasterStmt,
+ return queryBinlogStatus(
+ jdbc,
rs -> {
- if (rs.next()) {
- final String binlogFilename = rs.getString(1);
- final long binlogPosition = rs.getLong(2);
- final String gtidSet =
- rs.getMetaData().getColumnCount() > 4 ? rs.getString(5) : null;
- return BinlogOffset.builder()
- .setBinlogFilePosition(binlogFilename, binlogPosition)
- .setGtidSet(gtidSet)
- .build();
- } else {
- throw new FlinkRuntimeException(
- "Cannot read the binlog filename and position via '"
- + showMasterStmt
- + "'. Make sure your server is correctly configured");
+ try {
+ if (rs.next()) {
+ final String binlogFilename = rs.getString(1);
+ final long binlogPosition = rs.getLong(2);
+ final String gtidSet =
+ rs.getMetaData().getColumnCount() > 4
+ ? rs.getString(5)
+ : null;
+ return BinlogOffset.builder()
+ .setBinlogFilePosition(binlogFilename, binlogPosition)
+ .setGtidSet(gtidSet)
+ .build();
+ } else {
+ throw new ConfigurationException(
+ "Cannot read the binlog filename and position. Make sure your server is correctly configured");
+ }
+ } catch (Exception e) {
+ throw new FlinkRuntimeException(e);
}
});
} catch (SQLException e) {
- throw new FlinkRuntimeException(
- "Cannot read the binlog filename and position via '"
- + showMasterStmt
- + "'. Make sure your server is correctly configured",
- e);
+ throw new FlinkRuntimeException(e);
+ }
+ }
+
+ public static <T> T queryBinlogStatus(
+ JdbcConnection connection, Function<ResultSet, T> callback) throws SQLException {
+ String showMasterStmt = "SHOW MASTER STATUS";
Review Comment:
Thanks for your review. If this method is called frequently, we should look for ways to improve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS [flink-cdc]
Posted by "joyCurry30 (via GitHub)" <gi...@apache.org>.
joyCurry30 commented on code in PR #3955:
URL: https://github.com/apache/flink-cdc/pull/3955#discussion_r2013462573
##########
flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/debezium/DebeziumUtils.java:
##########
@@ -62,6 +65,8 @@ public class DebeziumUtils {
private static final Logger LOG = LoggerFactory.getLogger(DebeziumUtils.class);
+ private static String showMasterStmt = null;
Review Comment:
Why use 'static String'?Is there a special situation for this use?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3955:
URL: https://github.com/apache/flink-cdc/pull/3955#issuecomment-3439946979
This pull request has been automatically marked as stale because it has not had recent activity for 120 days. It will be closed in 60 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS [flink-cdc]
Posted by "joyCurry30 (via GitHub)" <gi...@apache.org>.
joyCurry30 commented on code in PR #3955:
URL: https://github.com/apache/flink-cdc/pull/3955#discussion_r2013458011
##########
docs/content/docs/faq/faq.md:
##########
@@ -137,7 +137,7 @@ Flink CDC provides DataStream API `MysqlSource` since version 2.1. Users can con
### Q8: In the same MySQL instance, the table of one database cannot synchronize incremental data, but other databases works fine. Why?
-Users can check Binlog_Ignore_DB and Binlog_Do_DB through the `show master status` command
+Users can check Binlog_Ignore_DB and Binlog_Do_DB through the `show master status` or `` command
Review Comment:
Or `` ? There seems to be a lack of concrete content.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3955:
URL: https://github.com/apache/flink-cdc/pull/3955#issuecomment-3684592519
This pull request has been closed because it has not had recent activity. You could reopen it if you try to continue your work, and anyone who are interested in it are encouraged to continue work on this pull request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS [flink-cdc]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3955: [FLINK-37503] Compatibility Issue with MySQL Syntax SHOW MASTER STATUS
URL: https://github.com/apache/flink-cdc/pull/3955
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org