You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2022/11/04 05:13:47 UTC
[GitHub] [shardingsphere] sandynz opened a new pull request, #21949: sharding column could be updated in pipeline
sandynz opened a new pull request, #21949:
URL: https://github.com/apache/shardingsphere/pull/21949
Changes proposed in this pull request:
- Remove shardingColumnsMap in SQL builder, so sharding column could be updated in `UPDATE tbl SET shardingColumn1=v ...` and upsert SQL (e.g. `INSERT INTO ... ON DUPLICATE KEY UPDATE ...`)
TODO
kernel doesn't support sharding column updating for now
---
Before committing this PR, I'm sure that I have checked the following options:
- [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
- [ ] I have self-reviewed the commit code.
- [ ] I have (or in comment I request) added corresponding labels for the pull request.
- [ ] I have passed maven check locally : `mvn clean install -B -T2C -DskipTests -Dmaven.javadoc.skip=true -e`.
- [ ] I have made corresponding changes to the documentation.
- [ ] I have added corresponding unit tests for my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] taojintianxia merged pull request #21949: sharding column could be updated in pipeline
Posted by GitBox <gi...@apache.org>.
taojintianxia merged PR #21949:
URL: https://github.com/apache/shardingsphere/pull/21949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on pull request #21949: sharding column could be updated in pipeline
Posted by GitBox <gi...@apache.org>.
sandynz commented on PR #21949:
URL: https://github.com/apache/shardingsphere/pull/21949#issuecomment-1302999189
Steps is the same as [usage doc]( https://shardingsphere.apache.org/document/5.2.1/en/user-manual/shardingsphere-proxy/migration/usage/#complete-procedure-example ).
After job status reach `EXECUTE_INCREMENTAL_TASK`, update sharding column in source table:
```
mysql> update t_order set order_id=7,status='updated1' where order_id=6;
Query OK, 1 row affected (0.05 sec)
Rows matched: 1 Changed: 1 Warnings: 0
```
Proxy log:
```
[ShardingSphere-pipeline-Incremental-j01016e501b498ed1bdb2c373a2e85e2529a6-2] o.a.s.d.p.c.importer.DefaultImporter - flush failed 3/3 times.
java.sql.SQLException: Can not update sharding value for table `t_order`.
at org.apache.shardingsphere.infra.util.exception.external.sql.ShardingSphereSQLException.toSQLException(ShardingSphereSQLException.java:59)
at org.apache.shardingsphere.dialect.SQLExceptionTransformEngine.toSQLException(SQLExceptionTransformEngine.java:51)
at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSpherePreparedStatement.executeUpdate(ShardingSpherePreparedStatement.java:346)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.executeUpdate(DefaultImporter.java:236)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.executeUpdate(DefaultImporter.java:215)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.doFlush(DefaultImporter.java:176)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.tryFlush(DefaultImporter.java:152)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.flushInternal(DefaultImporter.java:145)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.flush(DefaultImporter.java:136)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.write(DefaultImporter.java:108)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.runBlocking(DefaultImporter.java:94)
at org.apache.shardingsphere.data.pipeline.api.executor.AbstractLifecycleExecutor.start(AbstractLifecycleExecutor.java:53)
at org.apache.shardingsphere.data.pipeline.api.executor.AbstractLifecycleExecutor.run(AbstractLifecycleExecutor.java:91)
```
Job failed:
```
mysql> SHOW MIGRATION LIST;
+---------------------------------------+---------+----------------+--------+---------------------+---------------------+
| id | tables | job_item_count | active | create_time | stop_time |
+---------------------------------------+---------+----------------+--------+---------------------+---------------------+
| j01016e501b498ed1bdb2c373a2e85e2529a6 | t_order | 1 | false | 2022-11-04 13:25:59 | 2022-11-04 13:29:06 |
+---------------------------------------+---------+----------------+--------+---------------------+---------------------+
1 row in set (0.03 sec)
mysql> SHOW MIGRATION STATUS 'j01016e501b498ed1bdb2c373a2e85e2529a6';
+------+-------------+----------------------------------+--------+-------------------------+-------------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| item | data_source | status | active | processed_records_count | inventory_finished_percentage | incremental_idle_seconds | error_message
|
+------+-------------+----------------------------------+--------+-------------------------+-------------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 0 | ds_0 | EXECUTE_INCREMENTAL_TASK_FAILURE | false | 6 | 100 | 43 | java.util.concurrent.CompletionException: org.apache.shardingsphere.data.pipeline.core.exception.job.PipelineImporterJobWriteException: Importer job write data failed.
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1643)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.shardingsphere.data.pipeline.core.exception.job.PipelineImporterJobWriteException: Importer job write data failed.
at org.apache.shardingsphere.infra.util.exception.ShardingSpherePreconditions.checkState(ShardingSpherePreconditions.java:41)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.flushInternal(DefaultImporter.java:146)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.flush(DefaultImporter.java:136)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.write(DefaultImporter.java:108)
at org.apache.shardingsphere.data.pipeline.core.importer.DefaultImporter.runBlocking(DefaultImporter.java:94)
at org.apache.shardingsphere.data.pipeline.api.executor.AbstractLifecycleExecutor.start(AbstractLifecycleExecutor.java:53)
at org.apache.shardingsphere.data.pipeline.api.executor.AbstractLifecycleExecutor.run(AbstractLifecycleExecutor.java:91)
at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1640)
... 4 more
|
+------+-------------+----------------------------------+--------+-------------------------+-------------------------------+--------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.02 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter commented on pull request #21949: sharding column could be updated in pipeline
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #21949:
URL: https://github.com/apache/shardingsphere/pull/21949#issuecomment-1303010473
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/21949?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#21949](https://codecov.io/gh/apache/shardingsphere/pull/21949?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2e135cd) into [master](https://codecov.io/gh/apache/shardingsphere/commit/d9586648d47b889fec35bf5eca7abb374ff6e8cd?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d958664) will **increase** coverage by `0.41%`.
> The diff coverage is `18.75%`.
```diff
@@ Coverage Diff @@
## master #21949 +/- ##
============================================
+ Coverage 61.08% 61.50% +0.41%
- Complexity 2465 2468 +3
============================================
Files 4100 4106 +6
Lines 56991 57199 +208
Branches 9636 9679 +43
============================================
+ Hits 34812 35178 +366
+ Misses 19255 19158 -97
+ Partials 2924 2863 -61
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/21949?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ng/data/pipeline/ShardingColumnsExtractorImpl.java](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmVhdHVyZXMvc2hhcmRpbmcvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvc2hhcmRpbmcvZGF0YS9waXBlbGluZS9TaGFyZGluZ0NvbHVtbnNFeHRyYWN0b3JJbXBsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...base/schema/loader/SchemaMetaDataLoaderEngine.java](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9tZXRhZGF0YS9kYXRhYmFzZS9zY2hlbWEvbG9hZGVyL1NjaGVtYU1ldGFEYXRhTG9hZGVyRW5naW5lLmphdmE=) | `75.75% <0.00%> (ø)` | |
| [...eator/ShardingSpherePipelineDataSourceCreator.java](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-amRiYy9jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9kcml2ZXIvZGF0YS9waXBlbGluZS9kYXRhc291cmNlL2NyZWF0b3IvU2hhcmRpbmdTcGhlcmVQaXBlbGluZURhdGFTb3VyY2VDcmVhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../spi/sharding/ShardingColumnsExtractorFactory.java](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9kYXRhL3BpcGVsaW5lL3NwaS9zaGFyZGluZy9TaGFyZGluZ0NvbHVtbnNFeHRyYWN0b3JGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ta/pipeline/spi/sqlbuilder/PipelineSQLBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9kYXRhL3BpcGVsaW5lL3NwaS9zcWxidWlsZGVyL1BpcGVsaW5lU1FMQnVpbGRlci5qYXZh) | `0.00% <ø> (ø)` | |
| [...hm/DataMatchDataConsistencyCalculateAlgorithm.java](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jb3JlL2NoZWNrL2NvbnNpc3RlbmN5L2FsZ29yaXRobS9EYXRhTWF0Y2hEYXRhQ29uc2lzdGVuY3lDYWxjdWxhdGVBbGdvcml0aG0uamF2YQ==) | `11.34% <0.00%> (-0.12%)` | :arrow_down: |
| [...e/data/pipeline/core/importer/DefaultImporter.java](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jb3JlL2ltcG9ydGVyL0RlZmF1bHRJbXBvcnRlci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...ne/core/sqlbuilder/AbstractPipelineSQLBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jb3JlL3NxbGJ1aWxkZXIvQWJzdHJhY3RQaXBlbGluZVNRTEJ1aWxkZXIuamF2YQ==) | `3.27% <0.00%> (+0.10%)` | :arrow_up: |
| [...line/core/sqlbuilder/OraclePipelineSQLBuilder.java](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jb3JlL3NxbGJ1aWxkZXIvT3JhY2xlUGlwZWxpbmVTUUxCdWlsZGVyLmphdmE=) | `5.26% <0.00%> (ø)` | |
| [...peline/scenario/migration/MigrationJobAPIImpl.java](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9zY2VuYXJpby9taWdyYXRpb24vTWlncmF0aW9uSm9iQVBJSW1wbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| ... and [61 more](https://codecov.io/gh/apache/shardingsphere/pull/21949/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org