You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "6westboy9 (via GitHub)" <gi...@apache.org> on 2024/01/10 03:43:35 UTC
[I] ShardingSphere-Proxy migration verification cannot be executed [shardingsphere]
6westboy9 opened a new issue, #29690:
URL: https://github.com/apache/shardingsphere/issues/29690
## Bug Report
**When I used ShardingSphere-Proxy for data migration, I found that the migration was normal, but the validation could not be performed, and the error message was as follows:**
cat ${ShardingSphere-Proxy-HOME}/logs/stdout.log
<img width="933" alt="3e324a83970baed8939d81b77939a71" src="https://github.com/apache/shardingsphere/assets/39526241/b9f13e19-649f-405f-a627-91c3a25d5e58">
**ShardingSphere-Proxy server configuration is as follows:**
cat ${ShardingSphere-Proxy-HOME}/conf/server.yaml
```yaml
mode:
type: Cluster
repository:
type: ZooKeeper
props:
namespace: governance_ds
server-lists: xxx:2181
retryIntervalMilliseconds: 500
timeToLiveSeconds: 60
maxRetries: 3
operationTimeoutMilliseconds: 500
authority:
users:
- user: root@%
password: root
- user: sharding
password: sharding
privilege:
type: ALL_PERMITTED
```
### Which version of ShardingSphere did you use?
apache-shardingsphere-5.4.1-shardingsphere-proxy-bin.tar.gz
### Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?
ShardingSphere-Proxy
### Expected behavior
Successfully executed verification
### Actual behavior
Failed to perform verification
### Reason analyze (If you can)
After reading the source code, I found that the type attribute in YamProxyConfiguration is Cluster when starting, but when creating ContextManager, the type attribute in the modeConfiguration parameter in the ContextManagerBuilderParameter is Standalone, indicating that I don't quite understand it
### Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.
```SQL
CREATE DATABASE windranger_emr;
USE windranger_emr;
REGISTER STORAGE UNIT IF NOT EXISTS ds_0 (
URL="jdbc:mysql://xxx:3306/windranger_emr?characterEncoding=utf8&serverTimezone=GMT%2B8&allowMultiQueries=true",
USER="xxx",
PASSWORD="xxx",
PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000")
);
CREATE SHARDING TABLE RULE IF NOT EXISTS pat_inhos_order_group (
STORAGE_UNITS(ds_0),
SHARDING_COLUMN=inhos_code,TYPE(NAME="hash_mod",PROPERTIES("sharding-count"="4"))
);
CREATE TABLE IF NOT EXISTS pat_inhos_order_group (
// ...
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
```
```SQL
REGISTER MIGRATION SOURCE STORAGE UNIT windranger_emr_source (
URL="jdbc:mysql://xxx:3306/windranger_emr?characterEncoding=utf8&serverTimezone=GMT%2B8&allowMultiQueries=true",
USER="xxx",
PASSWORD="xxx",
PROPERTIES("minPoolSize"="1","maxPoolSize"="20","idleTimeout"="60000")
);
```
```SQL
MIGRATE TABLE windranger_emr_source.pat_inhos_order_group INTO windranger_emr.pat_inhos_order_group;
SHOW MIGRATION LIST;
+--------------------------------------------+---------------------------------------------+----------------+--------+---------------------+-----------+
| id | tables | job_item_count | active | create_time | stop_time |
+--------------------------------------------+---------------------------------------------+----------------+--------+---------------------+-----------+
| j0102p0000013efc3a9791b6876cc092ed25155f84 | windranger_emr_source.pat_inhos_order_group | 1 | true | 2024-01-09 17:04:25 | NULL |
+--------------------------------------------+---------------------------------------------+----------------+--------+---------------------+-----------+
SHOW MIGRATION STATUS 'j0102p0000013efc3a9791b6876cc092ed25155f84';
+------+-----------------------+---------------------------------------------+--------------------------+--------+-------------------------+-------------------------------+--------------------------+---------------+
| item | data_source | tables | status | active | processed_records_count | inventory_finished_percentage | incremental_idle_seconds | error_message |
+------+-----------------------+---------------------------------------------+--------------------------+--------+-------------------------+-------------------------------+--------------------------+---------------+
| 0 | windranger_emr_source | windranger_emr_source.pat_inhos_order_group | EXECUTE_INCREMENTAL_TASK | true | 28735078 | 100 | | |
+------+-----------------------+---------------------------------------------+--------------------------+--------+-------------------------+-------------------------------+--------------------------+---------------+
```
```SQL
CHECK MIGRATION 'j0102p0000013efc3a9791b6876cc092ed25155f84' BY TYPE (NAME='CRC32_MATCH');
SHOW MIGRATION CHECK STATUS 'j0102p0000013efc3a9791b6876cc092ed25155f84';
+---------------------------------------------+--------+---------------------+--------+-------------------------------+-----------------------------+--------------------------+-------------------------+----------------+------------------+----------------+-----------------+---------------+
| tables | result | check_failed_tables | active | inventory_finished_percentage | inventory_remaining_seconds | incremental_idle_seconds | check_begin_time | check_end_time | duration_seconds | algorithm_type | algorithm_props | error_message |
+---------------------------------------------+--------+---------------------+--------+-------------------------------+-----------------------------+--------------------------+-------------------------+----------------+------------------+----------------+-----------------+---------------+
| windranger_emr_source.pat_inhos_order_group | | | true | 0 | 0 | | 2024-01-10 10:53:42.762 | | 0 | CRC32_MATCH | | |
+---------------------------------------------+--------+---------------------+--------+-------------------------------+-----------------------------+--------------------------+-------------------------+----------------+------------------+----------------+-----------------+---------------+
```
### Example codes for reproduce this issue (such as a github link).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShardingSphere-Proxy migration verification cannot be executed [shardingsphere]
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on issue #29690:
URL: https://github.com/apache/shardingsphere/issues/29690#issuecomment-4240912190
Looks it's not bug. I'll close it now. You could re-open it later if necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShardingSphere-Proxy migration verification cannot be executed [shardingsphere]
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on issue #29690:
URL: https://github.com/apache/shardingsphere/issues/29690#issuecomment-3538823069
And you could try `DATA_MATCH` algorithm instead of `CRC32_MATCH`.
Since `CRC32_MATCH` will calculate all records on underlying database, so `inventory_finished_percentage` might be `0` for a long time if there're much records in table. But `DATA_MATCH` has different implementation, it'll update `inventory_finished_percentage` quickly.
Example DistSQL:
CHECK MIGRATION 'j0102p0000013efc3a9791b6876cc092ed25155f84' BY TYPE (NAME='DATA_MATCH');
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShardingSphere-Proxy migration verification cannot be executed [shardingsphere]
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz closed issue #29690: ShardingSphere-Proxy migration verification cannot be executed
URL: https://github.com/apache/shardingsphere/issues/29690
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShardingSphere-Proxy migration verification cannot be executed [shardingsphere]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #29690:
URL: https://github.com/apache/shardingsphere/issues/29690#issuecomment-1937854991
There hasn't been any activity on this issue recently, and in order to prioritize active issues, it will be marked as stale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] ShardingSphere-Proxy migration verification cannot be executed [shardingsphere]
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on issue #29690:
URL: https://github.com/apache/shardingsphere/issues/29690#issuecomment-2213924277
Hi @6westboy9 , migration and check will create internal Standalone data source, that's ok.
From `SHOW MIGRATION CHECK STATUS`, looks check job is still running, you could try to verify it on underlying MySQL `show processlist`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org