You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/11/17 09:17:28 UTC
[GitHub] [incubator-uniffle] zuston opened a new issue, #333: [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server
zuston opened a new issue, #333:
URL: https://github.com/apache/incubator-uniffle/issues/333
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
It's better to expose metrics of TOP10 app bytes in one shuffle server, especially when reaching the high watermark of disk.
### Motivation
_No response_
### Describe the solution
_No response_
### Additional context
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] amaliujia commented on issue #333: [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #333:
URL: https://github.com/apache/incubator-uniffle/issues/333#issuecomment-1336654319
@zuston could you give me some feedback on my WIP PR to see if I am on the right direction: https://github.com/apache/incubator-uniffle/pull/387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] amaliujia commented on issue #333: [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server
Posted by GitBox <gi...@apache.org>.
amaliujia commented on issue #333:
URL: https://github.com/apache/incubator-uniffle/issues/333#issuecomment-1328347340
Given this is a starter issue, do you mind sharing some key code pointers to help understand the workflow of the shuffle server that collects then observes the app bytes?
I assume it cannot be limited to only TOP 10 since the metrics will change and there always a need to sort?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #333: [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #333:
URL: https://github.com/apache/incubator-uniffle/issues/333#issuecomment-1320034811
> What can the metrics tell us? Which problems do you want to solve?
When reaching the high watermark, we could kill the max data size app to keep other apps stability
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #333: [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #333:
URL: https://github.com/apache/incubator-uniffle/issues/333#issuecomment-1328435807
> Given this is a starter issue, do you mind sharing some key code pointers to help understand the workflow of the shuffle server that collects then observes the app bytes?
>
> I assume it cannot be limited to only TOP 10 since the metrics will change and there always a need to sort?
Sorry, I must correct the above conversation that `The aggregated data of single app used bytes has been collected in current codebase` is wrong.
We have to collect all the app's write bytes in `ShuffleTaskManager.shuffleTaskInfos`. When method of `updateCachedBlockIds` is triggered, the size of written bytes should be updated in App's `ShuffleTaskInfo`.
After that, we could expose these top10 metrics by sorting these shuffleTaskInfos through extra api uri, like xxxx:19999/metrics/app.
Do you think so? Maybe my idea is not right, feel free to discuss more ~ @amaliujia
If you are interested in this issue, I will assign this to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #333: [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #333:
URL: https://github.com/apache/incubator-uniffle/issues/333#issuecomment-1320049582
> I think it's ok to include this metrics if it doen't introduce noticeable overhead.
I think it won’t. The aggregated data of single app used bytes has been collected in current codebase, we just need to expose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] kaijchen commented on issue #333: [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server
Posted by GitBox <gi...@apache.org>.
kaijchen commented on issue #333:
URL: https://github.com/apache/incubator-uniffle/issues/333#issuecomment-1320042966
I think it's ok to include this metrics if it doen't introduce too much overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server [incubator-uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston closed issue #333: [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server
URL: https://github.com/apache/incubator-uniffle/issues/333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
[GitHub] [incubator-uniffle] zuston commented on issue #333: [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server
Posted by GitBox <gi...@apache.org>.
zuston commented on issue #333:
URL: https://github.com/apache/incubator-uniffle/issues/333#issuecomment-1319791846
cc @jerqi @kaijchen Do you have some suggestion on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-uniffle] jerqi commented on issue #333: [FEATURE] Expose metrics of TOP10 app bytes in one shuffle server
Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #333:
URL: https://github.com/apache/incubator-uniffle/issues/333#issuecomment-1319864481
What can the metrics tell us? Which problems do you want to solve?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org