You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2021/06/15 07:03:45 UTC
[GitHub] [skywalking] a1vin-tian opened a new issue #7120: Concurrent create PrepareRequest when persist Metrics
a1vin-tian opened a new issue #7120:
URL: https://github.com/apache/skywalking/issues/7120
Please answer these questions before submitting your issue.
- Why do you submit this issue?
- [ * ] Feature or performance improvement
Skywalking v8.4.0
___
### Requirement or improvement
I watched the Grafana and found persist to ES is too slow that costs 600s.
I found that there are 2 steps in the PersistenceTimer.extractDataAndSave
Step 1: Prepare requests which searching ES for merge Metrics. (Need Concurrent)
Step 2: Persist prepare request that we made in `Stop 1` to ES. (Already fixed in https://github.com/apache/skywalking/issues/6472)
I fix in my code. This is the result.
```
List<Future<List<PrepareRequest>>> futures = new ArrayList<>(persistenceWorkers.size());
persistenceWorkers.forEach(worker -> {
futures.add(prepareExecutorService.submit(() -> {
if (log.isDebugEnabled()) {
log.debug("extract {} worker data and save", worker.getClass().getName());
}
List<PrepareRequest> innerPrepareRequests = new ArrayList<>(5000);
worker.buildBatchRequests(innerPrepareRequests);
worker.endOfRound(System.currentTimeMillis() - lastTime);
return innerPrepareRequests;
}));
});
for (Future<List<PrepareRequest>> future : futures) {
List<PrepareRequest> requests = future.get();
if(CollectionUtils.isNotEmpty(requests)){
prepareRequests.addAll(requests);
}
}
```
Can we change it to concurrent when we prepare the request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] a1vin-tian commented on issue #7120: Concurrent create PrepareRequest when persist Metrics
Posted by GitBox <gi...@apache.org>.
a1vin-tian commented on issue #7120:
URL: https://github.com/apache/skywalking/issues/7120#issuecomment-861417501
@wu-sheng Yes, Step 2 waits until step 1 all subtasks finished at present. But we can do step 1 and step 2 concurrently, which will shorten the entire persistence time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #7120: Concurrent create PrepareRequest when persist Metrics
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #7120:
URL: https://github.com/apache/skywalking/issues/7120#issuecomment-861328774
@a1vin-tian Thanks for reporting this case clearly. We used to optimize execution in concurrency as we only have a report about we facing that issue.
Because you have proved the case existing, which should be optimized. One question for your codes, do you run preparing in batch mode(your Step 1) and wait until all subtasks finished, then run into the persistent stage(your Step 2)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #7120: Concurrent create PrepareRequest when persist Metrics
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #7120:
URL: https://github.com/apache/skywalking/issues/7120#issuecomment-861426307
Another way to fix is also merging the channels, which is going to be done once @mrproliu makes the Satellite ready for java agent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on issue #7120: Concurrent create PrepareRequest when persist Metrics
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on issue #7120:
URL: https://github.com/apache/skywalking/issues/7120#issuecomment-861420954
> Yes, Step 2 waits until step 1 all subtasks finished at present. But we can do step 1 and step 2 concurrently, which will shorten the entire persistence time.
I am thinking about this, as this could use the storage performance more efficiently due to part in the preparation stage and the other part in flushing. Do you have data to compare these 2 ways to verify the feeling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #7120: Concurrent create PrepareRequest when persist Metrics
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #7120:
URL: https://github.com/apache/skywalking/issues/7120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng closed issue #7120: Concurrent create PrepareRequest when persist Metrics
Posted by GitBox <gi...@apache.org>.
wu-sheng closed issue #7120:
URL: https://github.com/apache/skywalking/issues/7120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org