You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/05/08 02:03:41 UTC
[GitHub] [incubator-doris] chaoyli opened a new issue #3517: [Proposal] Get the hot spot of the doris cluster.
chaoyli opened a new issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517
Nowadays, We have no insight about the hotspot the cluster.
During the time, we don't know the scanner rows of one tablet during time.
Now we should collect the scanner rows of the tablet and scanner bytes of the tablet for query.
Using this information, we can got the insight about the hot spot of cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-625588846
The metric is like this
```
[
{
"tags":{
"tablet_id":10010,
"metric":"scanner_rows"
},
"value":1000,
"unit":"number"
},
{
"tags":{
"tablet_id":10010,
"metric":"scanner_bytes"
},
"value":1024,
"unit":"bytes"
}
]
```
Supposing there are 10000 tablets in one BE node.
Every time we will gather 10000 * 200 bytes = 2MB at 15 seconds.
I will use a map to store the information.
And at interval(one hour) will remove unused tablet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
imay commented on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-625760711
This is a good job.
However I think something maybe discussed clearly before writing code.
1. Who will update this metrics?
2. How will updater get the table id?
3. How will the final metric be summary?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-625588846
The metric is like this
```
[
{
"tags":{
"tablet_id":10010,
"metric":"scanner_rows"
},
"value":1000,
"unit":"number"
},
{
"tags":{
"tablet_id":10010,
"metric":"scanner_bytes"
},
"value":1024,
"unit":"bytes"
}
]
```
Supposing there are 10000 tablets in one BE node and collecting metrics at every 15 seconds.
Every time we will gather 10000 * 200 Bytes = 2MB, one hundred nodes will collect 200MB.
For metric in one BE node, I will use a map to store the information.
```
std::unordered_map<tablet_id, metric> tablet_metric.
```
Considering the timeliness, We should only need recent metric in one hour.
So I will add another map will change to
```
std::unordered_map<tablet_id, metric> tablet_metric.
std::map<access_time, tablet_id> tablets_access_time;
```
Tablets_access_time to update the tablet_id access_time, only tablets access in the last hour will be significantly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-625588846
Supposing there are 10000 tablets in one BE node, the metric is like this
```
[
{
"tags":{
"tablet_id":10010,
"metric":"scanner_rows"
},
"value":1000,
"unit":"number"
},
{
"tags":{
"tablet_id":10010,
"metric":"scanner_bytes"
},
"value":1024,
"unit":"bytes"
}
]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-625588846
The metric is like this
```
[
{
"tags":{
"tablet_id":10010,
"metric":"scanner_rows"
},
"value":1000,
"unit":"number"
},
{
"tags":{
"tablet_id":10010,
"metric":"scanner_bytes"
},
"value":1024,
"unit":"bytes"
}
]
```
Supposing there are 10000 tablets in one BE node.
Every time we will gather 10000 * 200 bytes = 2MB.
I will use a map to store the information.
And at interval(one hour) will remove unused tablet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-625588846
The metric is like this
```
[
{
"tags":{
"table_id":10010,
"metric":"scanner_rows"
},
"value":1000,
"unit":"number"
},
{
"tags":{
"table_id":10010,
"metric":"scanner_bytes"
},
"value":1024,
"unit":"bytes"
}
]
```
Supposing there are 10000 tablets in one BE node and collecting metrics at every 15 seconds.
Every time we will gather 10000 * 200 Bytes = 2MB, one hundred nodes will collect 200MB, this will be consume huge bandwidth. And also user will take care of table granularity than tablet. So we will collect table metrics instead of tablet metrics.
For metric in one BE node, I will use a map to store the information.
```
std::unordered_map<table_id, metric> table_metrics
```
The incremental metric value should be calculated at every collecting interval.
Like this
```
2020-05-08 14:00:00 10000 1024(scanner_rows)
2020-05-08 14:00:15 10000 2048(scanner_rows)
The incremental value at this cycle is 1024.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-625604902
There also exists another function using log.
If every query collecting the tablet scanner rows/bytes, and write it to log.
We can access the log to get the message, it needs log analysis which is not integrated in Doris.
So I will use the above method to solve this problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-625588846
The metric is like this
```
[
{
"tags":{
"tablet_id":10010,
"metric":"scanner_rows"
},
"value":1000,
"unit":"number"
},
{
"tags":{
"tablet_id":10010,
"metric":"scanner_bytes"
},
"value":1024,
"unit":"bytes"
}
]
```
Supposing there are 10000 tablets in one BE node.
Every time we will gather 10000 * 200 bytes = 2MB.
I will use a map to store the information.
And at interval will remove unused tablet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-626171173
I think we could refer to
`https://pingcap.com/blog-cn/observability-of-distributed-system/`
and
`https://cloud.google.com/bigtable/docs/keyvis-overview`
to learn something
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-626088176
> This is a good job.
> However I think something maybe discussed clearly before writing code.
>
> 1. Who will update this metrics?
> 2. How will updater get the table id?
> 3. How will the final metric be summary?
1. Scanner Thread will update the metrics, it is the same as the metric like the query profile.
2. Nowadays, tablet meta has save the table id in the metadata, so new table will know the table_id.
3. If metric has been collected by agent, you can store to another database, or can be summed by a central node. There are ok both.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli edited a comment on issue #3517: [Proposal] Get the hot spot of the doris cluster.
Posted by GitBox <gi...@apache.org>.
chaoyli edited a comment on issue #3517:
URL: https://github.com/apache/incubator-doris/issues/3517#issuecomment-625588846
The metric is like this
```
[
{
"tags":{
"tablet_id":10010,
"metric":"scanner_rows"
},
"value":1000,
"unit":"number"
},
{
"tags":{
"tablet_id":10010,
"metric":"scanner_bytes"
},
"value":1024,
"unit":"bytes"
}
]
```
Supposing there are 10000 tablets in one BE node and collecting metrics at every 15 seconds.
Every time we will gather 10000 * 200 Bytes = 2MB, one hundred nodes will collect 200MB.
For metric in one BE node, I will use a map to store the information.
```
std::unordered_map<tablet_id, metric> tablet_metric.
```
Considering the timeliness, We should only need recent metric in one hour.
So I will add another map will change to
```
std::unordered_map<tablet_id, metric> tablet_metric.
std::map<access_time, tablet_id> tablets_access_time;
```
Tablets_access_time to update the tablet_id access_time, only tablets access in the last hour will be significantly.
To be more concise, the 10000 can be created as a configuration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org