You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@inlong.apache.org by "liaorui (via GitHub)" <gi...@apache.org> on 2023/06/29 04:23:45 UTC
[GitHub] [inlong] liaorui opened a new issue, #8363: [Improve][Sort] MySQL connector captures binlog in snapshot phase all the time even if tables have been removed in flink sql
liaorui opened a new issue, #8363:
URL: https://github.com/apache/inlong/issues/8363
### Description
MySQL connector captures binlog in snapshot phase all the time even if tables have been removed in flink sql.
For example, flink sql has captured table1, table2 and table3. Table1 has large scale data. MySQL connector chunk splitter is splitting table1 into many chunks, and reading binlog file by sequence. It will take a lot of time to finish reading table1.
If we don't want to capture table1 and remove it from flink sql, we pause the flink job with checkpoint, then start the job again. Even if table1 don't exist in flink sql, MySQL connector still captures table1, as flink job restores states from last checkpoint and table1 is still in checkpoint.
Maybe we need to do something when job is restoring from checkpoint.
### InLong Component
InLong Sort
### Are you willing to submit PR?
- [X] Yes, I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@inlong.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap closed issue #8363: [Improve][Sort] MySQL connector captures binlog in snapshot phase all the time even if tables have been removed in flink sql
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap closed issue #8363: [Improve][Sort] MySQL connector captures binlog in snapshot phase all the time even if tables have been removed in flink sql
URL: https://github.com/apache/inlong/issues/8363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org