You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2022/10/26 11:22:41 UTC
[GitHub] [shardingsphere] sandynz opened a new pull request, #21774: Improve migration job stopping
sandynz opened a new pull request, #21774:
URL: https://github.com/apache/shardingsphere/pull/21774
Changes proposed in this pull request:
- Improve migration job stopping. 1) Dumping data on DB, 2) Importing data on DB, 3) Blocking on queue, 4) Job onFailure on any of task failure, 5) Stop whole job in cluster on failure but not just current instance.
---
Before committing this PR, I'm sure that I have checked the following options:
- [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
- [ ] I have self-reviewed the commit code.
- [ ] I have (or in comment I request) added corresponding labels for the pull request.
- [ ] I have passed maven check locally : `mvn clean install -B -T2C -DskipTests -Dmaven.javadoc.skip=true -e`.
- [ ] I have made corresponding changes to the documentation.
- [ ] I have added corresponding unit tests for my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #21774: Improve migration job stopping
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #21774:
URL: https://github.com/apache/shardingsphere/pull/21774#issuecomment-1291932906
Recover steps:
1, Start MySQL server.
2, Start migration job, job will continue (active is true, and no error_message).
Example result:
```
mysql> start migration 'j0101270cbb6714cff0d7d13db9fa23c7c0a1';
Query OK, 0 rows affected (0.63 sec)
mysql> show migration list;
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| id | tables | sharding_total_count | active | create_time | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
| j0101270cbb6714cff0d7d13db9fa23c7c0a1 | sbtest1 | 1 | true | 2022-10-22 18:33:19 | NULL |
+---------------------------------------+---------+----------------------+--------+---------------------+-----------+
1 row in set (0.10 sec)
mysql> show migration status 'j0101270cbb6714cff0d7d13db9fa23c7c0a1';
+------+-------------+------------------------+--------+-------------------------+-------------------------------+--------------------------+---------------+
| item | data_source | status | active | processed_records_count | inventory_finished_percentage | incremental_idle_seconds | error_message |
+------+-------------+------------------------+--------+-------------------------+-------------------------------+--------------------------+---------------+
| 0 | mysql_ds_0 | EXECUTE_INVENTORY_TASK | true | 39408000 | 0 | 0 | |
+------+-------------+------------------------+--------+-------------------------+-------------------------------+--------------------------+---------------+
1 row in set (0.21 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #21774: Improve migration job stopping
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #21774:
URL: https://github.com/apache/shardingsphere/pull/21774#issuecomment-1291888741
Test steps:
1, Start a migration job on sbtest1 table, with 140 millions records, the migration might cost more than 10 minutes on local machine.
2, Stop underlying MySQL server.
3, Verify migration job is stopped, `active` is false, and `error_message` is not blank in migration status DistSQL result.
Example result:
```
mysql> show migration list;
+---------------------------------------+---------+----------------------+--------+---------------------+---------------------+
| id | tables | sharding_total_count | active | create_time | stop_time |
+---------------------------------------+---------+----------------------+--------+---------------------+---------------------+
| j0101270cbb6714cff0d7d13db9fa23c7c0a1 | sbtest1 | 1 | false | 2022-10-26 18:33:19 | 2022-10-26 19:09:08 |
+---------------------------------------+---------+----------------------+--------+---------------------+---------------------+
1 row in set (0.29 sec)
mysql> show migration status 'j0101270cbb6714cff0d7d13db9fa23c7c0a1';
+------+-------------+--------------------------------+--------+-------------------------+-------------------------------+--------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------+
| item | data_source | status | active | processed_records_count | inventory_finished_percentage | incremental_idle_seconds | error_message
|
+------+-------------+--------------------------------+--------+-------------------------+-------------------------------+--------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------+
| 0 | mysql_ds_0 | EXECUTE_INVENTORY_TASK_FAILURE | false | 39248000 | 0 | 0 | java.util.concurrent.CompletionException: org.apache.shardingsphere.data.pipeline.core.ingest.exception.IngestException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Server shutdown in progress
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1643)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.shardingsphere.data.pipeline.core.ingest.exception.IngestException: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Server shutdown in progress
at org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper.runBlocking(InventoryDumper.java:116)
at org.apache.shardingsphere.data.pipeline.api.executor.AbstractLifecycleExecutor.start(AbstractLifecycleExecutor.java:53)
at org.apache.shardingsphere.data.pipeline.api.executor.AbstractLifecycleExecutor.run(AbstractLifecycleExecutor.java:91)
at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1640)
... 4 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Server shutdown in progress
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.Util.getInstance(Util.java:408)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)
at com.mysql.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:1288)
at com.mysql.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:788)
at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2011)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
at org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper.dump(InventoryDumper.java:142)
at org.apache.shardingsphere.data.pipeline.core.ingest.dumper.InventoryDumper.runBlocking(InventoryDumper.java:106)
... 7 more
|
+------+-------------+--------------------------------+--------+-------------------------+-------------------------------+--------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------+
1 row in set (0.02 sec)
```
And also verify pipeline threads quit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] tristaZero merged pull request #21774: Improve migration job stopping
Posted by GitBox <gi...@apache.org>.
tristaZero merged PR #21774:
URL: https://github.com/apache/shardingsphere/pull/21774
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org