You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by GitBox <gi...@apache.org> on 2022/10/25 11:43:33 UTC
[GitHub] [shardingsphere] azexcy opened a new pull request, #21750: Improve stop pipeline job, extract method to the upper level
azexcy opened a new pull request, #21750:
URL: https://github.com/apache/shardingsphere/pull/21750
Changes proposed in this pull request:
- Improve pipeline job sync to stop, extract method to the upper level
-
-
---
Before committing this PR, I'm sure that I have checked the following options:
- [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
- [ ] I have self-reviewed the commit code.
- [ ] I have (or in comment I request) added corresponding labels for the pull request.
- [ ] I have passed maven check locally : `mvn clean install -B -T2C -DskipTests -Dmaven.javadoc.skip=true -e`.
- [ ] I have made corresponding changes to the documentation.
- [ ] I have added corresponding unit tests for my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] codecov-commenter commented on pull request #21750: Improve stop pipeline job, extract method to the upper level
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #21750:
URL: https://github.com/apache/shardingsphere/pull/21750#issuecomment-1291456877
# [Codecov](https://codecov.io/gh/apache/shardingsphere/pull/21750?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#21750](https://codecov.io/gh/apache/shardingsphere/pull/21750?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (da68278) into [master](https://codecov.io/gh/apache/shardingsphere/commit/5c247ec28388c26818651a5269403567ccbb76b0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5c247ec) will **decrease** coverage by `0.27%`.
> The diff coverage is `51.82%`.
```diff
@@ Coverage Diff @@
## master #21750 +/- ##
============================================
- Coverage 61.06% 60.78% -0.28%
+ Complexity 2497 2446 -51
============================================
Files 4110 4117 +7
Lines 57386 57455 +69
Branches 9697 9697
============================================
- Hits 35042 34926 -116
- Misses 19433 19622 +189
+ Partials 2911 2907 -4
```
| [Impacted Files](https://codecov.io/gh/apache/shardingsphere/pull/21750?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...erter/DatabaseDiscoveryRuleStatementConverter.java](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmVhdHVyZXMvZGItZGlzY292ZXJ5L2Rpc3RzcWwvaGFuZGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGJkaXNjb3ZlcnkvZGlzdHNxbC9oYW5kbGVyL2NvbnZlcnRlci9EYXRhYmFzZURpc2NvdmVyeVJ1bGVTdGF0ZW1lbnRDb252ZXJ0ZXIuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...te/AlterDatabaseDiscoveryRuleStatementUpdater.java](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmVhdHVyZXMvZGItZGlzY292ZXJ5L2Rpc3RzcWwvaGFuZGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGJkaXNjb3ZlcnkvZGlzdHNxbC9oYW5kbGVyL3VwZGF0ZS9BbHRlckRhdGFiYXNlRGlzY292ZXJ5UnVsZVN0YXRlbWVudFVwZGF0ZXIuamF2YQ==) | `86.95% <ø> (+0.16%)` | :arrow_up: |
| [...e/CreateDatabaseDiscoveryRuleStatementUpdater.java](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmVhdHVyZXMvZGItZGlzY292ZXJ5L2Rpc3RzcWwvaGFuZGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGJkaXNjb3ZlcnkvZGlzdHNxbC9oYW5kbGVyL3VwZGF0ZS9DcmVhdGVEYXRhYmFzZURpc2NvdmVyeVJ1bGVTdGF0ZW1lbnRVcGRhdGVyLmphdmE=) | `74.28% <ø> (-4.29%)` | :arrow_down: |
| [...core/DatabaseDiscoveryDistSQLStatementVisitor.java](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZmVhdHVyZXMvZGItZGlzY292ZXJ5L2Rpc3RzcWwvcGFyc2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9zaGFyZGluZ3NwaGVyZS9kYmRpc2NvdmVyeS9kaXN0c3FsL3BhcnNlci9jb3JlL0RhdGFiYXNlRGlzY292ZXJ5RGlzdFNRTFN0YXRlbWVudFZpc2l0b3IuamF2YQ==) | `50.00% <ø> (ø)` | |
| [...esher/type/AlterTableStatementSchemaRefresher.java](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvY29udGV4dC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvaW5mcmEvY29udGV4dC9yZWZyZXNoZXIvdHlwZS9BbHRlclRhYmxlU3RhdGVtZW50U2NoZW1hUmVmcmVzaGVyLmphdmE=) | `7.40% <0.00%> (ø)` | |
| [...resher/type/AlterViewStatementSchemaRefresher.java](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvY29udGV4dC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvaW5mcmEvY29udGV4dC9yZWZyZXNoZXIvdHlwZS9BbHRlclZpZXdTdGF0ZW1lbnRTY2hlbWFSZWZyZXNoZXIuamF2YQ==) | `5.71% <0.00%> (ø)` | |
| [...sher/type/CreateTableStatementSchemaRefresher.java](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvY29udGV4dC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvaW5mcmEvY29udGV4dC9yZWZyZXNoZXIvdHlwZS9DcmVhdGVUYWJsZVN0YXRlbWVudFNjaGVtYVJlZnJlc2hlci5qYXZh) | `11.76% <0.00%> (ø)` | |
| [...esher/type/CreateViewStatementSchemaRefresher.java](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvY29udGV4dC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvaW5mcmEvY29udGV4dC9yZWZyZXNoZXIvdHlwZS9DcmVhdGVWaWV3U3RhdGVtZW50U2NoZW1hUmVmcmVzaGVyLmphdmE=) | `10.00% <0.00%> (ø)` | |
| [...rdingsphere/infra/util/reflect/ReflectiveUtil.java](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-aW5mcmEvdXRpbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvaW5mcmEvdXRpbC9yZWZsZWN0L1JlZmxlY3RpdmVVdGlsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...line/core/api/impl/AbstractPipelineJobAPIImpl.java](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a2VybmVsL2RhdGEtcGlwZWxpbmUvY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvc2hhcmRpbmdzcGhlcmUvZGF0YS9waXBlbGluZS9jb3JlL2FwaS9pbXBsL0Fic3RyYWN0UGlwZWxpbmVKb2JBUElJbXBsLmphdmE=) | `0.00% <0.00%> (ø)` | |
| ... and [113 more](https://codecov.io/gh/apache/shardingsphere/pull/21750/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on a diff in pull request #21750: Improve stop pipeline job, extract method to the upper level
Posted by GitBox <gi...@apache.org>.
azexcy commented on code in PR #21750:
URL: https://github.com/apache/shardingsphere/pull/21750#discussion_r1005190831
##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/scenario/consistencycheck/ConsistencyCheckJob.java:
##########
@@ -74,12 +70,9 @@ public void stop() {
log.info("stop consistency check job, jobId is null, ignore");
return;
}
- for (PipelineTasksRunner each : getTasksRunnerMap().values()) {
+ for (PipelineTasksRunner each : getTaskRunners()) {
each.stop();
}
- getTasksRunnerMap().clear();
- String jobBarrierDisablePath = PipelineMetaDataNode.getJobBarrierDisablePath(getJobId());
- pipelineDistributedBarrier.persistEphemeralChildrenNode(jobBarrierDisablePath, 0);
- PipelineJobProgressPersistService.removeJobProgressPersistContext(getJobId());
Review Comment:
Ok, I will extract `removeJobProgressPersistContext` method to parent class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #21750: Improve stop pipeline job, extract method to the upper level
Posted by GitBox <gi...@apache.org>.
sandynz commented on code in PR #21750:
URL: https://github.com/apache/shardingsphere/pull/21750#discussion_r1005141370
##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/scenario/migration/MigrationJob.java:
##########
@@ -118,10 +118,8 @@ public void stop() {
return;
}
log.info("stop tasks runner, jobId={}", jobId);
- String jobBarrierDisablePath = PipelineMetaDataNode.getJobBarrierDisablePath(jobId);
for (PipelineTasksRunner each : getTasksRunnerMap().values()) {
each.stop();
- pipelineDistributedBarrier.persistEphemeralChildrenNode(jobBarrierDisablePath, each.getJobItemContext().getShardingItem());
Review Comment:
There's still `persistEphemeralChildrenNode` invocation in `execute`, could it be put to common method?
##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/core/job/PipelineJobCenter.java:
##########
@@ -59,14 +63,20 @@ public static boolean isJobExisting(final String jobId) {
* Stop job.
*
* @param jobId job id
+ * @param isAsync is async to stop
*/
- public static void stop(final String jobId) {
+ public static void stop(final String jobId, final boolean isAsync) {
Review Comment:
It's better not add new parameter, since:
1, Most of cases use `true` for `isAsync` param;
2, We could do `persistEphemeralChildrenNode` just on where it call this `stop` and use `false` for `isAysnc`;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz merged pull request #21750: Improve stop pipeline job, extract method to the upper level
Posted by GitBox <gi...@apache.org>.
sandynz merged PR #21750:
URL: https://github.com/apache/shardingsphere/pull/21750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #21750: Improve stop pipeline job, extract method to the upper level
Posted by GitBox <gi...@apache.org>.
sandynz commented on code in PR #21750:
URL: https://github.com/apache/shardingsphere/pull/21750#discussion_r1005180760
##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/core/task/IncrementalTask.java:
##########
@@ -82,7 +82,7 @@ public IncrementalTask(final int concurrency, final DumperConfiguration dumperCo
private IncrementalTaskProgress createIncrementalTaskProgress(final IngestPosition<?> position, final InventoryIncrementalJobItemProgress jobItemProgress) {
IncrementalTaskProgress incrementalTaskProgress = new IncrementalTaskProgress();
incrementalTaskProgress.setPosition(position);
- if (null != jobItemProgress) {
+ if (null != jobItemProgress && null != jobItemProgress.getIncremental().getIncrementalTaskProgress()) {
incrementalTaskProgress.setIncrementalTaskDelay(jobItemProgress.getIncremental().getIncrementalTaskProgress().getIncrementalTaskDelay());
Review Comment:
`jobItemProgress.getIncremental()` might be null
##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/scenario/consistencycheck/ConsistencyCheckJob.java:
##########
@@ -74,12 +70,9 @@ public void stop() {
log.info("stop consistency check job, jobId is null, ignore");
return;
}
- for (PipelineTasksRunner each : getTasksRunnerMap().values()) {
+ for (PipelineTasksRunner each : getTaskRunners()) {
each.stop();
}
- getTasksRunnerMap().clear();
- String jobBarrierDisablePath = PipelineMetaDataNode.getJobBarrierDisablePath(getJobId());
- pipelineDistributedBarrier.persistEphemeralChildrenNode(jobBarrierDisablePath, 0);
- PipelineJobProgressPersistService.removeJobProgressPersistContext(getJobId());
Review Comment:
`removeJobProgressPersistContext` is removed here, but not in migraiton job.
1, Is `removeJobProgressPersistContext` not needed for consistency job?
2, Could `removeJobProgressPersistContext` be added in AbstractPipelineJob?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org