You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2021/12/10 13:24:09 UTC

[GitHub] [shardingsphere] sandynz opened a new pull request #14045: Separate sharding related code from scaling-core module part 3 and refactor job configuration structure

sandynz opened a new pull request #14045:
URL: https://github.com/apache/shardingsphere/pull/14045


   Fixes #13875 part 8.
   
   Changes proposed in this pull request:
   - Remove SnowflakeKeyGenerateAlgorithm dependency from JobConfiguration, which is in `sharding-core`
   - Add getActualDataSourceConfig for ShardingSphereJDBCDataSourceConfiguration, prepare for DataSourcePreparer refactoring
   - Add JobDataNodeLine
   - Add tablesFirstDataNodes in job handleConfig, prepare for DataSourcePreparer refactoring
   - Refactor fields in job handleConfig
   - Refactor DataSourcePreparer, separate sharding related code from scaling-core
   - Refactor jobShardingDataNodes type in handleConfig
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] sandynz commented on pull request #14045: Separate sharding related code from scaling-core module part 3 and refactor job configuration structure

Posted by GitBox <gi...@apache.org>.
sandynz commented on pull request #14045:
URL: https://github.com/apache/shardingsphere/pull/14045#issuecomment-991018311


   There's one CI run failed, still embed ZK related:
   ```
   [INFO ] 2021-12-10 13:43:48.944 [main] o.a.s.s.c.fixture.EmbedTestingServer - Embed zookeeper server already exists 1, on 127.0.0.1:3181
   [INFO ] 2021-12-10 13:43:49.132 [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:3181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@42504d1
   [WARN ] 2021-12-10 13:43:51.315 [SyncThread:0] o.a.z.server.persistence.FileTxnLog - fsync-ing the write ahead log in SyncThread:0 took 2176ms which will adversely effect operation latency.File size is 67108880 bytes. See the ZooKeeper troubleshooting guide
   [INFO ] 2021-12-10 13:43:51.631 [main] org.apache.zookeeper.ZooKeeper - Session: 0x100001307ce0001 closed
   Error:  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.716 s <<< FAILURE! - in org.apache.shardingsphere.scaling.core.job.ScalingJobTest
   Error:  assertExecute(org.apache.shardingsphere.scaling.core.job.ScalingJobTest)  Time elapsed: 2.532 s  <<< ERROR!
   org.apache.shardingsphere.mode.repository.cluster.ClusterPersistRepositoryException: org.apache.zookeeper.KeeperException$OperationTimeoutException: KeeperErrorCode = OperationTimeout
   	at org.apache.shardingsphere.scaling.core.job.ScalingJobTest.assertExecute(ScalingJobTest.java:56)
   Caused by: org.apache.zookeeper.KeeperException$OperationTimeoutException: KeeperErrorCode = OperationTimeout
   	at org.apache.shardingsphere.scaling.core.job.ScalingJobTest.assertExecute(ScalingJobTest.java:56)
   ```
   It might be caused by `fsync-ing`, it cost 2176ms, more than operation timeout.
   
   Plan to implement Memory mode registry center implementation and use it for unit test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] codecov-commenter commented on pull request #14045: Separate sharding related code from scaling-core module part 3 and refactor job configuration structure

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #14045:
URL: https://github.com/apache/shardingsphere/pull/14045#issuecomment-991134881


   # [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/14045?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#14045](https://codecov.io/gh/apache/shardingsphere/pull/14045?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (73e71a7) into [master](https://codecov.io/gh/apache/shardingsphere/commit/4f01e866431564a5b52c154ec5b0da0690ddc801?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4f01e86) will **decrease** coverage by `0.05%`.
   > The diff coverage is `56.36%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/shardingsphere/pull/14045/graphs/tree.svg?width=650&height=150&src=pr&token=ZvlXpWa7so&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/shardingsphere/pull/14045?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #14045      +/-   ##
   ============================================
   - Coverage     63.48%   63.43%   -0.06%     
   - Complexity     1771     1780       +9     
   ============================================
     Files          2810     2819       +9     
     Lines         42027    42150     +123     
     Branches       7172     7195      +23     
   ============================================
   + Hits          26682    26736      +54     
   - Misses        13303    13359      +56     
   - Partials       2042     2055      +13     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/14045?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ped/ShardingSphereJDBCDataSourceConfiguration.java](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9jb25maWcvZGF0YXNvdXJjZS90eXBlZC9TaGFyZGluZ1NwaGVyZUpEQkNEYXRhU291cmNlQ29uZmlndXJhdGlvbi5qYXZh) | `51.85% <0.00%> (-6.49%)` | :arrow_down: |
   | [...datasource/typed/TypedDataSourceConfiguration.java](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtaW5mcmEvc2hhcmRpbmdzcGhlcmUtaW5mcmEtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9pbmZyYS9jb25maWcvZGF0YXNvdXJjZS90eXBlZC9UeXBlZERhdGFTb3VyY2VDb25maWd1cmF0aW9uLmphdmE=) | `16.66% <0.00%> (-10.61%)` | :arrow_down: |
   | [...ta/pipeline/core/datasource/DataSourceManager.java](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUta2VybmVsL3NoYXJkaW5nc3BoZXJlLWRhdGEtcGlwZWxpbmUvc2hhcmRpbmdzcGhlcmUtZGF0YS1waXBlbGluZS1jb3JlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9kYXRhL3BpcGVsaW5lL2NvcmUvZGF0YXNvdXJjZS9EYXRhU291cmNlTWFuYWdlci5qYXZh) | `0.00% <ø> (ø)` | |
   | [...dingsphere/migration/common/api/ScalingWorker.java](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtc2NhbGluZy9zaGFyZGluZ3NwaGVyZS1zY2FsaW5nLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL21pZ3JhdGlvbi9jb21tb24vYXBpL1NjYWxpbmdXb3JrZXIuamF2YQ==) | `0.00% <ø> (ø)` | |
   | [...ommon/job/preparer/AbstractDataSourcePreparer.java](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtc2NhbGluZy9zaGFyZGluZ3NwaGVyZS1zY2FsaW5nLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL21pZ3JhdGlvbi9jb21tb24vam9iL3ByZXBhcmVyL0Fic3RyYWN0RGF0YVNvdXJjZVByZXBhcmVyLmphdmE=) | `54.54% <0.00%> (+21.21%)` | :arrow_up: |
   | [.../check/consistency/DataConsistencyCheckerImpl.java](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtc2NhbGluZy9zaGFyZGluZ3NwaGVyZS1zY2FsaW5nLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3NjYWxpbmcvY29yZS9qb2IvY2hlY2svY29uc2lzdGVuY3kvRGF0YUNvbnNpc3RlbmN5Q2hlY2tlckltcGwuamF2YQ==) | `85.91% <ø> (ø)` | |
   | [...rdingsphere/scaling/core/util/ScalingTaskUtil.java](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtc2NhbGluZy9zaGFyZGluZ3NwaGVyZS1zY2FsaW5nLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3NjYWxpbmcvY29yZS91dGlsL1NjYWxpbmdUYXNrVXRpbC5qYXZh) | `55.00% <0.00%> (+45.00%)` | :arrow_up: |
   | [...component/checker/OpenGaussDataSourcePreparer.java](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtc2NhbGluZy9zaGFyZGluZ3NwaGVyZS1zY2FsaW5nLWRpYWxlY3Qvc2hhcmRpbmdzcGhlcmUtc2NhbGluZy1vcGVuZ2F1c3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3NjYWxpbmcvb3BlbmdhdXNzL2NvbXBvbmVudC9jaGVja2VyL09wZW5HYXVzc0RhdGFTb3VyY2VQcmVwYXJlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ngsphere/scaling/core/config/JobConfiguration.java](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtc2NhbGluZy9zaGFyZGluZ3NwaGVyZS1zY2FsaW5nLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3NjYWxpbmcvY29yZS9jb25maWcvSm9iQ29uZmlndXJhdGlvbi5qYXZh) | `43.75% <16.66%> (-46.88%)` | :arrow_down: |
   | [...schedule/ShardingRuleJobConfigurationPreparer.java](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2hhcmRpbmdzcGhlcmUtc2NhbGluZy9zaGFyZGluZ3NwaGVyZS1zY2FsaW5nLWNvcmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3NoYXJkaW5nc3BoZXJlL3NoYXJkaW5nL3NjaGVkdWxlL1NoYXJkaW5nUnVsZUpvYkNvbmZpZ3VyYXRpb25QcmVwYXJlci5qYXZh) | `58.08% <31.81%> (-23.26%)` | :arrow_down: |
   | ... and [63 more](https://codecov.io/gh/apache/shardingsphere/pull/14045/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/shardingsphere/pull/14045?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/14045?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [4f01e86...73e71a7](https://codecov.io/gh/apache/shardingsphere/pull/14045?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] tristaZero merged pull request #14045: Separate sharding related code from scaling-core module part 3 and refactor job configuration structure

Posted by GitBox <gi...@apache.org>.
tristaZero merged pull request #14045:
URL: https://github.com/apache/shardingsphere/pull/14045


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [shardingsphere] tristaZero merged pull request #14045: Separate sharding related code from scaling-core module part 3 and refactor job configuration structure

Posted by GitBox <gi...@apache.org>.
tristaZero merged pull request #14045:
URL: https://github.com/apache/shardingsphere/pull/14045


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org