You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@shardingsphere.apache.org by "azexcy (via GitHub)" <gi...@apache.org> on 2023/09/04 12:25:52 UTC
[GitHub] [shardingsphere] azexcy opened a new pull request, #28356: Fix get inventory position not correctly at Pipeline
azexcy opened a new pull request, #28356:
URL: https://github.com/apache/shardingsphere/pull/28356
Changes proposed in this pull request:
- Fix get inventory position not correctly at Pipeline
---
Before committing this PR, I'm sure that I have checked the following options:
- [ ] My code follows the [code of conduct](https://shardingsphere.apache.org/community/en/involved/conduct/code/) of this project.
- [ ] I have self-reviewed the commit code.
- [ ] I have (or in comment I request) added corresponding labels for the pull request.
- [ ] I have passed maven check locally : `./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e`.
- [ ] I have made corresponding changes to the documentation.
- [ ] I have added corresponding unit tests for my changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on a diff in pull request #28356: Fix get inventory position not correctly at Pipeline
Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on code in PR #28356:
URL: https://github.com/apache/shardingsphere/pull/28356#discussion_r1315329042
##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/common/job/progress/JobItemInventoryTasksProgress.java:
##########
@@ -43,7 +43,7 @@ public final class JobItemInventoryTasksProgress {
* @return inventory position
*/
public Map<String, IngestPosition> getInventoryPosition(final String tableName) {
- Pattern pattern = Pattern.compile(String.format("%s(#\\d+)?", tableName));
+ Pattern pattern = Pattern.compile(String.format("\\.%s#(\\d+)?", tableName));
Review Comment:
If using `InventoryTaskSplitter`, the shardingItem wouldn't be null, scenario not found where shardingItem is null, so I change Integer to int
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] azexcy commented on a diff in pull request #28356: Fix get inventory position not correctly at Pipeline
Posted by "azexcy (via GitHub)" <gi...@apache.org>.
azexcy commented on code in PR #28356:
URL: https://github.com/apache/shardingsphere/pull/28356#discussion_r1315329042
##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/common/job/progress/JobItemInventoryTasksProgress.java:
##########
@@ -43,7 +43,7 @@ public final class JobItemInventoryTasksProgress {
* @return inventory position
*/
public Map<String, IngestPosition> getInventoryPosition(final String tableName) {
- Pattern pattern = Pattern.compile(String.format("%s(#\\d+)?", tableName));
+ Pattern pattern = Pattern.compile(String.format("\\.%s#(\\d+)?", tableName));
Review Comment:
If using `InventoryTaskSplitter`, the shardingItem wouldn't be null, scenario not found where shardingItem is null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz merged pull request #28356: Fix get inventory position not correctly at Pipeline
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz merged PR #28356:
URL: https://github.com/apache/shardingsphere/pull/28356
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #28356: Fix get inventory position not correctly at Pipeline
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #28356:
URL: https://github.com/apache/shardingsphere/pull/28356#discussion_r1315775959
##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/common/job/progress/JobItemInventoryTasksProgress.java:
##########
@@ -43,7 +43,7 @@ public final class JobItemInventoryTasksProgress {
* @return inventory position
*/
public Map<String, IngestPosition> getInventoryPosition(final String tableName) {
- Pattern pattern = Pattern.compile(String.format("%s(#\\d+)?", tableName));
+ Pattern pattern = Pattern.compile(String.format("\\.%s#(\\d+)?", tableName));
Review Comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [shardingsphere] sandynz commented on a diff in pull request #28356: Fix get inventory position not correctly at Pipeline
Posted by "sandynz (via GitHub)" <gi...@apache.org>.
sandynz commented on code in PR #28356:
URL: https://github.com/apache/shardingsphere/pull/28356#discussion_r1315314770
##########
kernel/data-pipeline/core/src/test/java/org/apache/shardingsphere/data/pipeline/common/job/progress/InventoryIncrementalJobItemProgressTest.java:
##########
@@ -76,6 +79,24 @@ void assertGetIncrementalDataLatestActiveTimeMillis() {
assertThat(getJobItemProgress(ConfigurationFileUtils.readFile("job-progress-all-finished.yaml")).getIncremental().getIncrementalLatestActiveTimeMillis(), is(50L));
}
+ @Test
+ void assertGetProgressesCorrectly() {
+ Map<String, InventoryTaskProgress> progresses = new HashMap<>();
+ progresses.put("ds.order_item#0", new InventoryTaskProgress(new IntegerPrimaryKeyPosition(1, 100)));
+ progresses.put("ds.order_item#1", new InventoryTaskProgress(new IntegerPrimaryKeyPosition(100, 200)));
+ progresses.put("ds.order#0", new InventoryTaskProgress(new FinishedPosition()));
+ progresses.put("ds.test_order#0", new InventoryTaskProgress(new FinishedPosition()));
+ JobItemInventoryTasksProgress progress = new JobItemInventoryTasksProgress(progresses);
+ Map<String, IngestPosition> orderPosition = progress.getInventoryPosition("order");
+ assertThat(orderPosition.size(), is(1));
+ assertTrue(orderPosition.containsKey("ds.order#0"));
+ Map<String, IngestPosition> testOrderPosition = progress.getInventoryPosition("test_order");
+ assertThat(testOrderPosition.size(), is(1));
+ assertTrue(testOrderPosition.containsKey("ds.test_order#0"));
+ Map<String, IngestPosition> orderItemPosition = progress.getInventoryPosition("order_item");
+ assertThat(orderItemPosition.size(), is(2));
Review Comment:
Could we assert position but not only size?
##########
kernel/data-pipeline/core/src/main/java/org/apache/shardingsphere/data/pipeline/common/job/progress/JobItemInventoryTasksProgress.java:
##########
@@ -43,7 +43,7 @@ public final class JobItemInventoryTasksProgress {
* @return inventory position
*/
public Map<String, IngestPosition> getInventoryPosition(final String tableName) {
- Pattern pattern = Pattern.compile(String.format("%s(#\\d+)?", tableName));
+ Pattern pattern = Pattern.compile(String.format("\\.%s#(\\d+)?", tableName));
Review Comment:
Is `#` always existing?
From old code:
```
private String generateTaskId(final InventoryDumperConfiguration inventoryDumperConfig) {
String result = String.format("%s.%s", inventoryDumperConfig.getDataSourceName(), inventoryDumperConfig.getActualTableName());
return null == inventoryDumperConfig.getShardingItem() ? result : result + "#" + inventoryDumperConfig.getShardingItem();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@shardingsphere.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org