You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@yunikorn.apache.org by GitBox <gi...@apache.org> on 2020/10/13 12:45:29 UTC

[GitHub] [incubator-yunikorn-core] kingamarton opened a new pull request #215: [YUNIKORN-352] Validate queue caacity setup

kingamarton opened a new pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] kingamarton commented on a change in pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
kingamarton commented on a change in pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215#discussion_r507794230



##########
File path: pkg/common/configs/configvalidator.go
##########
@@ -72,37 +72,68 @@ func checkACL(acl string) error {
 	return nil
 }
 
-// Temporary convenience method: should use resource package to do this
-// currently no check for the type of resource as long as the value is OK all is OK
-func checkResource(res map[string]string) (int64, error) {
-	var totalres int64
-	for _, val := range res {
-		rescount, err := strconv.ParseInt(val, 10, 64)
-		if err != nil {
-			return 0, fmt.Errorf("resource parsing failed: %v", err)
+// Check the queue resource configuration settings.
+// - child or children cannot have higher maximum or guaranteed limits than parents
+// - children (added together) cannot have a higher guaranteed setting than a parent
+func checkResourceConfigurationsForQueue(cur *QueueConfig, parent *QueueConfig) error {
+	// If cur has children, make sure sum of children's guaranteed <= cur.guaranteed
+	curGuaranteedRes, err := resources.NewResourceFromConf(cur.Resources.Guaranteed)
+	if err != nil {
+		return err
+	}
+	if curGuaranteedRes.HasNegativeValue() {
+		return fmt.Errorf("invalid guaranteed resource for queue %s, cannot be negative", cur.Name)

Review comment:
       done
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] TravisBuddy commented on pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
TravisBuddy commented on pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215#issuecomment-712228026


   Hey @kingamarton,  
   Your changes look good to me!
   
   <a href="https:&#x2F;&#x2F;travis-ci.org&#x2F;apache&#x2F;incubator-yunikorn-core&#x2F;builds&#x2F;737089637">View build log</a>
   
   ###### TravisBuddy Request Identifier: 18b4d2e0-121c-11eb-af4f-e3a6e0ba4ce0
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] codecov[bot] commented on pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215#issuecomment-707744242


   # [Codecov](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=h1) Report
   > Merging [#215](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-yunikorn-core/commit/62489ba4d53f44f73e032784bb20d965dc9bbb4f?el=desc) will **increase** coverage by `0.06%`.
   > The diff coverage is `88.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/graphs/tree.svg?width=650&height=150&src=pr&token=SB9NrIi3Hy)](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master     #215      +/-   ##
   ==========================================
   + Coverage   60.84%   60.90%   +0.06%     
   ==========================================
     Files          67       67              
     Lines        5728     5778      +50     
   ==========================================
   + Hits         3485     3519      +34     
   - Misses       2101     2114      +13     
   - Partials      142      145       +3     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [pkg/cache/initializer.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL2NhY2hlL2luaXRpYWxpemVyLmdv) | `65.07% <ø> (-13.75%)` | :arrow_down: |
   | [pkg/common/configs/configvalidator.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL2NvbW1vbi9jb25maWdzL2NvbmZpZ3ZhbGlkYXRvci5nbw==) | `87.00% <86.04%> (+0.13%)` | :arrow_up: |
   | [pkg/common/resources/resources.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL2NvbW1vbi9yZXNvdXJjZXMvcmVzb3VyY2VzLmdv) | `96.19% <100.00%> (+0.12%)` | :arrow_up: |
   | [pkg/scheduler/scheduler.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL3NjaGVkdWxlci9zY2hlZHVsZXIuZ28=) | `0.00% <0.00%> (ø)` | |
   | [pkg/scheduler/scheduling\_application.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL3NjaGVkdWxlci9zY2hlZHVsaW5nX2FwcGxpY2F0aW9uLmdv) | `71.06% <0.00%> (+0.32%)` | :arrow_up: |
   | [pkg/webservice/handlers.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL3dlYnNlcnZpY2UvaGFuZGxlcnMuZ28=) | `53.89% <0.00%> (+3.89%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=footer). Last update [62489ba...1d6f139](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] yangwwei commented on a change in pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
yangwwei commented on a change in pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215#discussion_r504993591



##########
File path: pkg/common/configs/configvalidator.go
##########
@@ -72,37 +72,68 @@ func checkACL(acl string) error {
 	return nil
 }
 
-// Temporary convenience method: should use resource package to do this
-// currently no check for the type of resource as long as the value is OK all is OK
-func checkResource(res map[string]string) (int64, error) {
-	var totalres int64
-	for _, val := range res {
-		rescount, err := strconv.ParseInt(val, 10, 64)
-		if err != nil {
-			return 0, fmt.Errorf("resource parsing failed: %v", err)
+// Check the queue resource configuration settings.
+// - child or children cannot have higher maximum or guaranteed limits than parents
+// - children (added together) cannot have a higher guaranteed setting than a parent
+func checkResourceConfigurationsForQueue(cur *QueueConfig, parent *QueueConfig) error {
+	// If cur has children, make sure sum of children's guaranteed <= cur.guaranteed
+	curGuaranteedRes, err := resources.NewResourceFromConf(cur.Resources.Guaranteed)
+	if err != nil {
+		return err
+	}
+	if curGuaranteedRes.HasNegativeValue() {
+		return fmt.Errorf("invalid guaranteed resource for queue %s, cannot be negative", cur.Name)

Review comment:
       it makes sense to print the resource as well

##########
File path: pkg/common/configs/configvalidator.go
##########
@@ -72,37 +72,68 @@ func checkACL(acl string) error {
 	return nil
 }
 
-// Temporary convenience method: should use resource package to do this
-// currently no check for the type of resource as long as the value is OK all is OK
-func checkResource(res map[string]string) (int64, error) {
-	var totalres int64
-	for _, val := range res {
-		rescount, err := strconv.ParseInt(val, 10, 64)
-		if err != nil {
-			return 0, fmt.Errorf("resource parsing failed: %v", err)
+// Check the queue resource configuration settings.
+// - child or children cannot have higher maximum or guaranteed limits than parents
+// - children (added together) cannot have a higher guaranteed setting than a parent
+func checkResourceConfigurationsForQueue(cur *QueueConfig, parent *QueueConfig) error {
+	// If cur has children, make sure sum of children's guaranteed <= cur.guaranteed
+	curGuaranteedRes, err := resources.NewResourceFromConf(cur.Resources.Guaranteed)
+	if err != nil {
+		return err
+	}
+	if curGuaranteedRes.HasNegativeValue() {
+		return fmt.Errorf("invalid guaranteed resource for queue %s, cannot be negative", cur.Name)
+	}
+	curMaxRes, err := resources.NewResourceFromConf(cur.Resources.Max)
+	if err != nil {
+		return err
+	}
+	if curMaxRes.HasNegativeValue() {
+		return fmt.Errorf("invalid max resource for queue %s, cannot be negative", cur.Name)

Review comment:
       same as above

##########
File path: pkg/common/configs/configvalidator.go
##########
@@ -72,37 +72,68 @@ func checkACL(acl string) error {
 	return nil
 }
 
-// Temporary convenience method: should use resource package to do this
-// currently no check for the type of resource as long as the value is OK all is OK
-func checkResource(res map[string]string) (int64, error) {
-	var totalres int64
-	for _, val := range res {
-		rescount, err := strconv.ParseInt(val, 10, 64)
-		if err != nil {
-			return 0, fmt.Errorf("resource parsing failed: %v", err)
+// Check the queue resource configuration settings.
+// - child or children cannot have higher maximum or guaranteed limits than parents
+// - children (added together) cannot have a higher guaranteed setting than a parent
+func checkResourceConfigurationsForQueue(cur *QueueConfig, parent *QueueConfig) error {
+	// If cur has children, make sure sum of children's guaranteed <= cur.guaranteed
+	curGuaranteedRes, err := resources.NewResourceFromConf(cur.Resources.Guaranteed)
+	if err != nil {
+		return err
+	}
+	if curGuaranteedRes.HasNegativeValue() {
+		return fmt.Errorf("invalid guaranteed resource for queue %s, cannot be negative", cur.Name)
+	}
+	curMaxRes, err := resources.NewResourceFromConf(cur.Resources.Max)
+	if err != nil {
+		return err
+	}
+	if curMaxRes.HasNegativeValue() {
+		return fmt.Errorf("invalid max resource for queue %s, cannot be negative", cur.Name)
+	}
+
+	if len(cur.Queues) > 0 {
+		// Check children
+		for i := range cur.Queues {
+			if err := checkResourceConfigurationsForQueue(&cur.Queues[i], cur); err != nil {
+				return err
+			}
 		}
-		if rescount < 0 {
-			return 0, fmt.Errorf("resource value invalid, cannot be negative: %d", rescount)
+		sum := resources.NewResource()
+		for _, child := range cur.Queues {
+			childGuaranteed, err := resources.NewResourceFromConf(child.Resources.Guaranteed)
+			if err != nil {
+				return err
+			}
+			sum.AddTo(childGuaranteed)
 		}
-		totalres += rescount
-	}
-	return totalres, nil
-}
 
-// Check the resource configuration
-func checkResources(resource Resources) error {
-	// check guaranteed resources
-	if resource.Guaranteed != nil && len(resource.Guaranteed) != 0 {
-		_, err := checkResource(resource.Guaranteed)
-		if err != nil {
-			return err
+		if cur.Resources.Guaranteed != nil {
+			if !resources.FitIn(curGuaranteedRes, sum) {
+				return fmt.Errorf("queue %s has guaranteed-resources (%v) smaller than sum of children guaranteed resources (%v)", cur.Name, curGuaranteedRes, sum)
+			}
+		} else {
+			cur.Resources.Guaranteed = sum.ToConf()

Review comment:
       this seems to be modifying the value: `cur *QueueConfig`.
   Could you pls check if this is the case? I don't think we should modify the configs.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] kingamarton commented on a change in pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
kingamarton commented on a change in pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215#discussion_r507795485



##########
File path: pkg/common/configs/configvalidator.go
##########
@@ -72,37 +72,68 @@ func checkACL(acl string) error {
 	return nil
 }
 
-// Temporary convenience method: should use resource package to do this
-// currently no check for the type of resource as long as the value is OK all is OK
-func checkResource(res map[string]string) (int64, error) {
-	var totalres int64
-	for _, val := range res {
-		rescount, err := strconv.ParseInt(val, 10, 64)
-		if err != nil {
-			return 0, fmt.Errorf("resource parsing failed: %v", err)
+// Check the queue resource configuration settings.
+// - child or children cannot have higher maximum or guaranteed limits than parents
+// - children (added together) cannot have a higher guaranteed setting than a parent
+func checkResourceConfigurationsForQueue(cur *QueueConfig, parent *QueueConfig) error {
+	// If cur has children, make sure sum of children's guaranteed <= cur.guaranteed
+	curGuaranteedRes, err := resources.NewResourceFromConf(cur.Resources.Guaranteed)
+	if err != nil {
+		return err
+	}
+	if curGuaranteedRes.HasNegativeValue() {
+		return fmt.Errorf("invalid guaranteed resource for queue %s, cannot be negative", cur.Name)
+	}
+	curMaxRes, err := resources.NewResourceFromConf(cur.Resources.Max)
+	if err != nil {
+		return err
+	}
+	if curMaxRes.HasNegativeValue() {
+		return fmt.Errorf("invalid max resource for queue %s, cannot be negative", cur.Name)

Review comment:
       done
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] codecov[bot] edited a comment on pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215#issuecomment-707744242


   # [Codecov](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=h1) Report
   > Merging [#215](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-yunikorn-core/commit/62489ba4d53f44f73e032784bb20d965dc9bbb4f?el=desc) will **increase** coverage by `0.06%`.
   > The diff coverage is `88.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/graphs/tree.svg?width=650&height=150&src=pr&token=SB9NrIi3Hy)](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master     #215      +/-   ##
   ==========================================
   + Coverage   60.84%   60.90%   +0.06%     
   ==========================================
     Files          67       67              
     Lines        5728     5778      +50     
   ==========================================
   + Hits         3485     3519      +34     
   - Misses       2101     2114      +13     
   - Partials      142      145       +3     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [pkg/cache/initializer.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL2NhY2hlL2luaXRpYWxpemVyLmdv) | `65.07% <ø> (-13.75%)` | :arrow_down: |
   | [pkg/common/configs/configvalidator.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL2NvbW1vbi9jb25maWdzL2NvbmZpZ3ZhbGlkYXRvci5nbw==) | `87.00% <86.04%> (+0.13%)` | :arrow_up: |
   | [pkg/common/resources/resources.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL2NvbW1vbi9yZXNvdXJjZXMvcmVzb3VyY2VzLmdv) | `96.19% <100.00%> (+0.12%)` | :arrow_up: |
   | [pkg/scheduler/scheduler.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL3NjaGVkdWxlci9zY2hlZHVsZXIuZ28=) | `0.00% <0.00%> (ø)` | |
   | [pkg/scheduler/scheduling\_application.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL3NjaGVkdWxlci9zY2hlZHVsaW5nX2FwcGxpY2F0aW9uLmdv) | `71.06% <0.00%> (+0.32%)` | :arrow_up: |
   | [pkg/webservice/handlers.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL3dlYnNlcnZpY2UvaGFuZGxlcnMuZ28=) | `53.89% <0.00%> (+3.89%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=footer). Last update [62489ba...ebc9de6](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] yangwwei merged pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
yangwwei merged pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] kingamarton commented on a change in pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
kingamarton commented on a change in pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215#discussion_r507803112



##########
File path: pkg/common/configs/configvalidator.go
##########
@@ -72,37 +72,68 @@ func checkACL(acl string) error {
 	return nil
 }
 
-// Temporary convenience method: should use resource package to do this
-// currently no check for the type of resource as long as the value is OK all is OK
-func checkResource(res map[string]string) (int64, error) {
-	var totalres int64
-	for _, val := range res {
-		rescount, err := strconv.ParseInt(val, 10, 64)
-		if err != nil {
-			return 0, fmt.Errorf("resource parsing failed: %v", err)
+// Check the queue resource configuration settings.
+// - child or children cannot have higher maximum or guaranteed limits than parents
+// - children (added together) cannot have a higher guaranteed setting than a parent
+func checkResourceConfigurationsForQueue(cur *QueueConfig, parent *QueueConfig) error {
+	// If cur has children, make sure sum of children's guaranteed <= cur.guaranteed
+	curGuaranteedRes, err := resources.NewResourceFromConf(cur.Resources.Guaranteed)
+	if err != nil {
+		return err
+	}
+	if curGuaranteedRes.HasNegativeValue() {
+		return fmt.Errorf("invalid guaranteed resource for queue %s, cannot be negative", cur.Name)
+	}
+	curMaxRes, err := resources.NewResourceFromConf(cur.Resources.Max)
+	if err != nil {
+		return err
+	}
+	if curMaxRes.HasNegativeValue() {
+		return fmt.Errorf("invalid max resource for queue %s, cannot be negative", cur.Name)
+	}
+
+	if len(cur.Queues) > 0 {
+		// Check children
+		for i := range cur.Queues {
+			if err := checkResourceConfigurationsForQueue(&cur.Queues[i], cur); err != nil {
+				return err
+			}
 		}
-		if rescount < 0 {
-			return 0, fmt.Errorf("resource value invalid, cannot be negative: %d", rescount)
+		sum := resources.NewResource()
+		for _, child := range cur.Queues {
+			childGuaranteed, err := resources.NewResourceFromConf(child.Resources.Guaranteed)
+			if err != nil {
+				return err
+			}
+			sum.AddTo(childGuaranteed)
 		}
-		totalres += rescount
-	}
-	return totalres, nil
-}
 
-// Check the resource configuration
-func checkResources(resource Resources) error {
-	// check guaranteed resources
-	if resource.Guaranteed != nil && len(resource.Guaranteed) != 0 {
-		_, err := checkResource(resource.Guaranteed)
-		if err != nil {
-			return err
+		if cur.Resources.Guaranteed != nil {
+			if !resources.FitIn(curGuaranteedRes, sum) {
+				return fmt.Errorf("queue %s has guaranteed-resources (%v) smaller than sum of children guaranteed resources (%v)", cur.Name, curGuaranteedRes, sum)
+			}
+		} else {
+			cur.Resources.Guaranteed = sum.ToConf()

Review comment:
       Yes, it is the case. This is exactly the same check we had before in the `initializer.go` file. I just moved the check here, to have it in the validation phase. The guaranteed resource was modified there as well if it wasn't set. I haven't removed it because I didn't wanted to change the check's behaviour and I think it makes sense to set the guaranteed resource for a parent queue = to the sum of children's guaranteed resource it it is not set. @yangwwei please let me know if it makes sense for you.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] TravisBuddy commented on pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
TravisBuddy commented on pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215#issuecomment-707744212


   Hey @kingamarton,  
   Your changes look good to me!
   
   <a href="https:&#x2F;&#x2F;travis-ci.org&#x2F;apache&#x2F;incubator-yunikorn-core&#x2F;builds&#x2F;735375761">View build log</a>
   
   ###### TravisBuddy Request Identifier: 431c2320-0d59-11eb-9111-f326d0cb7f09
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] codecov[bot] edited a comment on pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
codecov[bot] edited a comment on pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215#issuecomment-707744242


   # [Codecov](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=h1) Report
   > Merging [#215](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=desc) into [master](https://codecov.io/gh/apache/incubator-yunikorn-core/commit/62489ba4d53f44f73e032784bb20d965dc9bbb4f?el=desc) will **increase** coverage by `0.06%`.
   > The diff coverage is `88.00%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/graphs/tree.svg?width=650&height=150&src=pr&token=SB9NrIi3Hy)](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master     #215      +/-   ##
   ==========================================
   + Coverage   60.84%   60.90%   +0.06%     
   ==========================================
     Files          67       67              
     Lines        5728     5778      +50     
   ==========================================
   + Hits         3485     3519      +34     
   - Misses       2101     2114      +13     
   - Partials      142      145       +3     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [pkg/cache/initializer.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL2NhY2hlL2luaXRpYWxpemVyLmdv) | `65.07% <ø> (-13.75%)` | :arrow_down: |
   | [pkg/common/configs/configvalidator.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL2NvbW1vbi9jb25maWdzL2NvbmZpZ3ZhbGlkYXRvci5nbw==) | `87.00% <86.04%> (+0.13%)` | :arrow_up: |
   | [pkg/common/resources/resources.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL2NvbW1vbi9yZXNvdXJjZXMvcmVzb3VyY2VzLmdv) | `96.19% <100.00%> (+0.12%)` | :arrow_up: |
   | [pkg/scheduler/scheduler.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL3NjaGVkdWxlci9zY2hlZHVsZXIuZ28=) | `0.00% <0.00%> (ø)` | |
   | [pkg/scheduler/scheduling\_application.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL3NjaGVkdWxlci9zY2hlZHVsaW5nX2FwcGxpY2F0aW9uLmdv) | `71.06% <0.00%> (+0.32%)` | :arrow_up: |
   | [pkg/webservice/handlers.go](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215/diff?src=pr&el=tree#diff-cGtnL3dlYnNlcnZpY2UvaGFuZGxlcnMuZ28=) | `53.89% <0.00%> (+3.89%)` | :arrow_up: |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=footer). Last update [62489ba...ebc9de6](https://codecov.io/gh/apache/incubator-yunikorn-core/pull/215?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-yunikorn-core] yangwwei commented on a change in pull request #215: [YUNIKORN-352] Validate queue capacity setup

Posted by GitBox <gi...@apache.org>.
yangwwei commented on a change in pull request #215:
URL: https://github.com/apache/incubator-yunikorn-core/pull/215#discussion_r508197216



##########
File path: pkg/common/configs/configvalidator.go
##########
@@ -72,37 +72,68 @@ func checkACL(acl string) error {
 	return nil
 }
 
-// Temporary convenience method: should use resource package to do this
-// currently no check for the type of resource as long as the value is OK all is OK
-func checkResource(res map[string]string) (int64, error) {
-	var totalres int64
-	for _, val := range res {
-		rescount, err := strconv.ParseInt(val, 10, 64)
-		if err != nil {
-			return 0, fmt.Errorf("resource parsing failed: %v", err)
+// Check the queue resource configuration settings.
+// - child or children cannot have higher maximum or guaranteed limits than parents
+// - children (added together) cannot have a higher guaranteed setting than a parent
+func checkResourceConfigurationsForQueue(cur *QueueConfig, parent *QueueConfig) error {
+	// If cur has children, make sure sum of children's guaranteed <= cur.guaranteed
+	curGuaranteedRes, err := resources.NewResourceFromConf(cur.Resources.Guaranteed)
+	if err != nil {
+		return err
+	}
+	if curGuaranteedRes.HasNegativeValue() {
+		return fmt.Errorf("invalid guaranteed resource for queue %s, cannot be negative", cur.Name)
+	}
+	curMaxRes, err := resources.NewResourceFromConf(cur.Resources.Max)
+	if err != nil {
+		return err
+	}
+	if curMaxRes.HasNegativeValue() {
+		return fmt.Errorf("invalid max resource for queue %s, cannot be negative", cur.Name)
+	}
+
+	if len(cur.Queues) > 0 {
+		// Check children
+		for i := range cur.Queues {
+			if err := checkResourceConfigurationsForQueue(&cur.Queues[i], cur); err != nil {
+				return err
+			}
 		}
-		if rescount < 0 {
-			return 0, fmt.Errorf("resource value invalid, cannot be negative: %d", rescount)
+		sum := resources.NewResource()
+		for _, child := range cur.Queues {
+			childGuaranteed, err := resources.NewResourceFromConf(child.Resources.Guaranteed)
+			if err != nil {
+				return err
+			}
+			sum.AddTo(childGuaranteed)
 		}
-		totalres += rescount
-	}
-	return totalres, nil
-}
 
-// Check the resource configuration
-func checkResources(resource Resources) error {
-	// check guaranteed resources
-	if resource.Guaranteed != nil && len(resource.Guaranteed) != 0 {
-		_, err := checkResource(resource.Guaranteed)
-		if err != nil {
-			return err
+		if cur.Resources.Guaranteed != nil {
+			if !resources.FitIn(curGuaranteedRes, sum) {
+				return fmt.Errorf("queue %s has guaranteed-resources (%v) smaller than sum of children guaranteed resources (%v)", cur.Name, curGuaranteedRes, sum)
+			}
+		} else {
+			cur.Resources.Guaranteed = sum.ToConf()

Review comment:
       If that's the case, I am OK with the change.
   The concern I had earlier was because this is a validation function, a validation function is not supposed to modify the original value of the object.
   By looking how we use the validation, it seems to be safe though. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org