You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@inlong.apache.org by "e-mhui (via GitHub)" <gi...@apache.org> on 2023/05/17 09:52:11 UTC
[GitHub] [inlong] e-mhui opened a new pull request, #8044: [INLONG-8038][Sort] Optimize MySQL CDC chunk splitting logic
e-mhui opened a new pull request, #8044:
URL: https://github.com/apache/inlong/pull/8044
### Prepare a Pull Request
[INLONG-8038][Sort] Optimize MySQL CDC chunk splitting logic
- Fixes #8038
### Motivation
When MySQL CDC splits chunks during the snapshot phase, the last chunk is usually split by "where id > xxxx, [splitKey, infinity)". When the snapshot phase takes a long time and reaches the last chunk, the primary key may have grown significantly, resulting in a large amount of data being retrieved through the chunking condition, which may cause TM OOM exceptions.
To optimize the chunking logic, we can set the condition of the last chunk to "where id > xxxx and id < max(id)".
### Modifications
Change the condition of the last chunk to "WHERE id >= ? AND id <= ?".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap merged pull request #8044: [INLONG-8038][Sort] Optimize MySQL CDC chunk splitting
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap merged PR #8044:
URL: https://github.com/apache/inlong/pull/8044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on pull request #8044: [INLONG-8038][Sort] Optimize MySQL CDC chunk splitting logic
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on PR #8044:
URL: https://github.com/apache/inlong/pull/8044#issuecomment-1551141732
@e-mhui Hi, Will be binlog offset effected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] e-mhui commented on a diff in pull request #8044: [INLONG-8038][Sort] Optimize MySQL CDC chunk splitting
Posted by "e-mhui (via GitHub)" <gi...@apache.org>.
e-mhui commented on code in PR #8044:
URL: https://github.com/apache/inlong/pull/8044#discussion_r1209650776
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/mysql-cdc/src/main/java/org/apache/inlong/sort/cdc/mysql/source/assigners/MySqlSnapshotSplitAssigner.java:
##########
@@ -228,7 +228,10 @@ private void startAsynchronouslySplit() {
.map(MySqlSnapshotSplit::toSchemaLessSnapshotSplit)
.collect(Collectors.toList());
synchronized (lock) {
- remainingSplits.addAll(schemaLessSnapshotSplits);
+ int size = schemaLessSnapshotSplits.size();
Review Comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] gong commented on a diff in pull request #8044: [INLONG-8038][Sort] Optimize MySQL CDC chunk splitting
Posted by "gong (via GitHub)" <gi...@apache.org>.
gong commented on code in PR #8044:
URL: https://github.com/apache/inlong/pull/8044#discussion_r1209672869
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/mysql-cdc/src/main/java/org/apache/inlong/sort/cdc/mysql/source/assigners/MySqlSnapshotSplitAssigner.java:
##########
@@ -397,6 +397,14 @@ private void closeExecutorService() {
}
}
+ private void addNewlyAddedSplits(List<MySqlSchemalessSnapshotSplit> schemaLessSnapshotSplits) {
+ int size = schemaLessSnapshotSplits.size();
+ // move the last snapshot split to the front of the remaining splits to prevent OOM
+ // caused by the excessive data of the last snapshot split.
+ remainingSplits.add(0, schemaLessSnapshotSplits.get(size - 1));
+ remainingSplits.addAll(schemaLessSnapshotSplits.subList(0, size - 1));
Review Comment:
`remainSplits` is all splits, `remainingSplits.add(0, schemaLessSnapshotSplits.get(size - 1));` will cause other table split lost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] e-mhui commented on pull request #8044: [INLONG-8038][Sort] Optimize MySQL CDC chunk splitting logic
Posted by "e-mhui (via GitHub)" <gi...@apache.org>.
e-mhui commented on PR #8044:
URL: https://github.com/apache/inlong/pull/8044#issuecomment-1551176226
> @e-mhui Hi, Will be binlog offset effected?
It will not affect binlog offset, because the binlog offset will start from the smallest high watermark in the chunk splits.
<img width="1761" alt="image" src="https://github.com/apache/inlong/assets/111486498/1252c14e-dcbb-4fc9-a67a-5bdbd1dfe824">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] EMsnap commented on a diff in pull request #8044: [INLONG-8038][Sort] Optimize MySQL CDC chunk splitting
Posted by "EMsnap (via GitHub)" <gi...@apache.org>.
EMsnap commented on code in PR #8044:
URL: https://github.com/apache/inlong/pull/8044#discussion_r1209642323
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/mysql-cdc/src/main/java/org/apache/inlong/sort/cdc/mysql/source/assigners/MySqlSnapshotSplitAssigner.java:
##########
@@ -228,7 +228,10 @@ private void startAsynchronouslySplit() {
.map(MySqlSnapshotSplit::toSchemaLessSnapshotSplit)
.collect(Collectors.toList());
synchronized (lock) {
- remainingSplits.addAll(schemaLessSnapshotSplits);
+ int size = schemaLessSnapshotSplits.size();
Review Comment:
I guess the steps can be held inside one method. ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [inlong] e-mhui commented on a diff in pull request #8044: [INLONG-8038][Sort] Optimize MySQL CDC chunk splitting
Posted by "e-mhui (via GitHub)" <gi...@apache.org>.
e-mhui commented on code in PR #8044:
URL: https://github.com/apache/inlong/pull/8044#discussion_r1209712264
##########
inlong-sort/sort-flink/sort-flink-v1.13/sort-connectors/mysql-cdc/src/main/java/org/apache/inlong/sort/cdc/mysql/source/assigners/MySqlSnapshotSplitAssigner.java:
##########
@@ -397,6 +397,14 @@ private void closeExecutorService() {
}
}
+ private void addNewlyAddedSplits(List<MySqlSchemalessSnapshotSplit> schemaLessSnapshotSplits) {
+ int size = schemaLessSnapshotSplits.size();
+ // move the last snapshot split to the front of the remaining splits to prevent OOM
+ // caused by the excessive data of the last snapshot split.
+ remainingSplits.add(0, schemaLessSnapshotSplits.get(size - 1));
+ remainingSplits.addAll(schemaLessSnapshotSplits.subList(0, size - 1));
Review Comment:
The `remainingSplits.add(0, schemaLessSnapshotSplits.get(size - 1));` means Inserts the specified element at the specified position in this list, it's not cause other split lost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@inlong.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org