You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/05/16 14:58:06 UTC
[GitHub] [hadoop-ozone] maobaolong opened a new pull request #929: HDDS-3593. Set a configurable maximum limit of available pipeline
maobaolong opened a new pull request #929:
URL: https://github.com/apache/hadoop-ozone/pull/929
## What changes were proposed in this pull request?
Set a configurable maximum limit of available pipeline,
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3593
## How was this patch tested?
A new cluster will not create pipeline more than confg autoCreateLimit,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #929: HDDS-3593. Set a configurable maximum limit of available pipeline
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #929:
URL: https://github.com/apache/hadoop-ozone/pull/929#issuecomment-629728685
@xiaoyuyao Thanks for review my PR.
`ozone.scm.ratis.pipeline.limit` is different with `ozone.scm.pipeline.creation.auto.limit`.
- `ozone.scm.ratis.pipeline.limit` is the total limit, if achieve this limit, no more pipeline would be created.
- `ozone.scm.pipeline.creation.auto.limit` is a limitation for `BackgroundPipelineCreator`, before `BackgroundPipelineCreator` want to create a pipeline periodically, it will check if the count of current available pipeline achieve the `ozone.scm.pipeline.creation.auto.limit`, if achieved already, `BackgroundPipelineCreator` will skip this loop.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on pull request #929: HDDS-3593. Set a configurable maximum limit of available pipeline
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on pull request #929:
URL: https://github.com/apache/hadoop-ozone/pull/929#issuecomment-633343824
> Thanks @xiaoyuyao for your suggestion, i if put the limit check to RatisPipelineProvider, all the ratis pipeline creation will be limit, my key point is limit the pipeline creation from BackgroundPipelineCreator, and a bigger limit check for other pipeline creation, for example, the putkey rpc call request a unknown pipeline, like 4 replication, if it cannot find the unknown type and replication factor pipeline, it will create pipeline sync, without the limit for BackgroundPipelineCreator, there must be no quota for the unknown type or replication factor pipeline.
@maobaolong You still have FACTOR to help you classify different behaviors. RatisPipelineProvider is the best place if you would complex logic for different Ratis pipelines. If you just want to skip creation, you can take a look at skipCreation function in BackgroundPipelineCreator. But I would you suggest to have a complete strategy to allocate different factor's pipelines.
At the end of the day, your purpose is to support multiple Ratis pipeline with different factors. This would include preallocation of pipelines, resource split and potential pipeline eviction. It may be more than just stop creating pipelines to leave quota to other pipelines with different factors. It's hard to rely on configs to know the best setup of pipeline quota. And it's almost impossible to maintain to balance of pipeline quota if you consider cluster expansion or shrinkage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong closed pull request #929: HDDS-3593. Set a configurable maximum limit of available pipeline
Posted by GitBox <gi...@apache.org>.
maobaolong closed pull request #929:
URL: https://github.com/apache/hadoop-ozone/pull/929
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #929: HDDS-3593. Set a configurable maximum limit of available pipeline
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #929:
URL: https://github.com/apache/hadoop-ozone/pull/929#issuecomment-648774117
/pending see @timmylicheng comments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on pull request #929: HDDS-3593. Set a configurable maximum limit of available pipeline
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on pull request #929:
URL: https://github.com/apache/hadoop-ozone/pull/929#issuecomment-632407759
Thanks @timmylicheng and @maobaolong for the clarification. If we really want to push a ratio/factor based on the number of live datanodes, can we move the logic down into RatisPipelineProvider with other limit check? This way, we don't need to introduce NodeManager to BackgroundPipelineCreator.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on pull request #929: HDDS-3593. Set a configurable maximum limit of available pipeline
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on pull request #929:
URL: https://github.com/apache/hadoop-ozone/pull/929#issuecomment-630657099
Pipeline limit is better to define along with datanode number. ozone.scm.ratis.pipeline.limit is mainly used in test and please be very careful to set it in distributed cluster since it's static and doesn't consider cluster expansion or shrinkage.
Also talk with baolong offline regarding this change.
@maobaolong @xiaoyuyao
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #929: HDDS-3593. Set a configurable maximum limit of available pipeline
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #929:
URL: https://github.com/apache/hadoop-ozone/pull/929#issuecomment-632487344
Thanks @xiaoyuyao for your suggestion, i if put the limit check to RatisPipelineProvider, all the ratis pipeline creation will be limit, my key point is limit the pipeline creation from BackgroundPipelineCreator, and a bigger limit check for other pipeline creation, for example, the putkey rpc call request a unknown pipeline, like 4 replication, if it cannot find the unknown type and replication factor pipeline, it will create pipeline sync, without the limit for BackgroundPipelineCreator, there must be no quota for the unknown type or replication factor pipeline.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] maobaolong commented on pull request #929: HDDS-3593. Set a configurable maximum limit of available pipeline
Posted by GitBox <gi...@apache.org>.
maobaolong commented on pull request #929:
URL: https://github.com/apache/hadoop-ozone/pull/929#issuecomment-632153048
@timmylicheng @xiaoyuyao Thanks for your review and the suggestion. I change the limit to a rate which consider healthy datanode number. PTAL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org