You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/11/16 07:38:59 UTC
[GitHub] [incubator-doris] tianhui5 opened a new issue #7129: [Feature] Limit cluster resource usage in user granularity
tianhui5 opened a new issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
### Description
In productive environment, the Doris cluster is often facing pressure from many aspects (mainly from stream load and query), cause many resource shortage problem like OOM, especially in shared cluster.
I think it's better to have a way to limit the usage of each user. Maybe limit the usage frequency is a proper way.
### Use case
_No response_
### Related issues
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-979058445
> I see. Doris already has `max_connection` limit which can be set for each user. But I think what you need is not just limit the number of connection, but to limit the `rate` of request.
>
> As far as I know, Guava's rate limiter may meet the requirement. But what more important is, how to define the `rate`? Simply put, it may be a limitation of QPS. But the essence is "control the consumption of cluster resources per unit time."
>
> So I think in the first version, we can implement this function through simple rules (such as QPS). But in the specific design, we must reflect the abstract design of "system resources" so that we can add more rules later.
>
> Looking forward your PR!
Where do you think the limiter should be put, BE or FE? As Guava is for Java, Do you think the limiter is better in FE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 edited a comment on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 edited a comment on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-1038570176
The limit of operation frequency is developed in #7474 , user can config the threshold through frontend config like below:
`ADMIN SET FRONTEND CONFIG ('key' = 'value')`
You can limit the query number(`max_running_query_num`) in certain period (`report_stats_period`), the default period is 10 second(10 * 1000). And you can also limit the load number through `max_running_txn_num`.
The design of this feature is clear:
Each FE keeps its query number locally, and reports the query number to Master every period, So every FE can get the query number in each FE through metadata synchronize. When there is a query arrive, if the total query number in last period exceeds threshold, the system reject the query. User can only query in next period.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] xinyiZzz commented on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-1000326390
> What if the request exceed the limit? return error or slow down? And is there any other system we can refer to?
Impala’s AdmissionController does a similar thing,
Introduction is here https://shimo.im/docs/6qxjctpyDHJgPwtw
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-969971912
I'm prefer to add syntax `LIMITER` on behalf of the frequency limiter. This limiter is an abstract concept, it can have many kinds of types like query or stream load.
So the SQL to monipulate `LIMITER` is like below:
`CREATE LIMITER name PORPERTIES("key1"="value1", "key2"="value2");`
`DROP LIMITER name;`
`SHOW LIMITER;`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-971323622
I think we can put `LIMITER` relative config in FE, and put metric data in BE. The procedure is like below:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-973726928
I see.
Doris already has `max_connection` limit which can be set for each user.
But I think what you need is not just limit the number of connection, but to limit the `rate` of request.
As far as I know, Guava's rate limiter may meet the requirement. But what more important is, how to define the `rate`?
Simply put, it may be a limitation of QPS. But the essence is "control the consumption of cluster resources per unit time."
So I think in the first version, we can implement this function through simple rules (such as QPS). But in the specific design, we must reflect the abstract design of "system resources" so that we can add more rules later.
Looking forward your PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 edited a comment on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 edited a comment on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-1038570176
The limit of operation frequency is developed in #7474 , user can config the threshold through frontend config like below:
`ADMIN SET FRONTEND CONFIG ('key' = 'value')`
You can limit the query number(`max_running_query_num`) or load number(`max_running_txn_num`) in certain period (`report_stats_period`), the default period is 10 second(10 * 1000).
The design of this feature is clear: Each FE keeps its query number locally, and broadcast the query number every period. So when there is a query arrive, if the total query number in last period exceeds threshold, the system reject the query. User can only query in next period.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 edited a comment on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 edited a comment on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-969987149
> What if the request exceed the limit? return error or slow down? And is there any other system we can refer to?
As far as I know, MySQL have variable `max_connection` to limit connection number. When connection exceed the limit, it returns error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 edited a comment on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 edited a comment on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-1038570176
The limit of operation frequency is developed in #7474 , user can config the threshold through frontend config like below:
`ADMIN SET FRONTEND CONFIG ('key' = 'value')`
You can limit the query number(`max_running_query_num`) or load number(`max_running_txn_num`) in certain period (`report_stats_period`), the default period is 10 second(10 * 1000).
The design of this feature is clear:
Each FE keeps its query number locally, and broadcast the query number every period. So when there is a query arrive, if the total query number in last period exceeds threshold, the system reject the query. User can only query in next period.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-1038570176
The limit of operation frequency is developed in #7474 , user can config the threshold through frontend config like below:
`ADMIN SET FRONTEND CONFIG ('key' = 'value')`
You can limit the query number(`max_running_query_num`) or load number(`max_running_txn_num`) in certain period (`report_stats_period`), the default period is 10 second(10 * 1000).
The design of this feature is clear: Each FE keep its query number locally, and broadcast the query number every period. So when there is a query arrive, if the total query number in last period is exceed threshold, it reject the query, user can only query in next period.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-969974125
What if the request exceed the limit? return error or slow down?
And is there any other system we can refer to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] tianhui5 commented on issue #7129: [Feature] Limit cluster resource usage in user granularity
Posted by GitBox <gi...@apache.org>.
tianhui5 commented on issue #7129:
URL: https://github.com/apache/incubator-doris/issues/7129#issuecomment-969987149
> What if the request exceed the limit? return error or slow down? And is there any other system we can refer to?
As far as I know, MySQL have variable `max_connection` to limit connection number. When connection exceed the limit, it returns error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org