You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@yunikorn.apache.org by GitBox <gi...@apache.org> on 2020/03/26 06:20:36 UTC
[GitHub] [incubator-yunikorn-k8shim] yangwwei opened a new pull request #88:
[YUNIKORN-54] admission-controller should keep generated ID tight
yangwwei opened a new pull request #88: [YUNIKORN-54] admission-controller should keep generated ID tight
URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/88
Simple change. We need a better format of generated appIDs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org
[GitHub] [incubator-yunikorn-k8shim] wilfred-s commented on a change in pull
request #88: [YUNIKORN-54] admission-controller should keep generated ID
tight
Posted by GitBox <gi...@apache.org>.
wilfred-s commented on a change in pull request #88: [YUNIKORN-54] admission-controller should keep generated ID tight
URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/88#discussion_r399224232
##########
File path: pkg/plugin/admissioncontrollers/webhook/admission_controller.go
##########
@@ -138,25 +138,7 @@ func updateLabels(pod *v1.Pod, patch []patchOperation) []patchOperation {
if _, ok := existingLabels[common.SparkLabelAppID]; !ok {
if _, ok := existingLabels[common.LabelApplicationID]; !ok {
- // if app id not exist, generate one
- // the generated ID is using [PodNamespace]_[PodName]_[Timestamp] naming convention.
- // some admission controllers have strict checks of the length/format of each labels,
- // this convention keeps the name tidy and short.
- podNamespace := "default"
- if pod.Namespace != "" {
- podNamespace = pod.Namespace
- }
-
- // pod's name can be generated, if name is not explicitly specified
- // look for generateName instead
- podName := "unknown"
- if pod.Name != "" {
- podName = pod.Name
- } else if pod.GenerateName != "" {
- podName = pod.GenerateName
- }
-
- generatedID := fmt.Sprintf("%s_%s_%d", podNamespace, podName, time.Now().Unix())
+ generatedID := fmt.Sprintf("__app_%d", time.Now().UnixNano())
Review comment:
Why throw out all the info we have and then try to substitute it with some other info?
If we have the detail we should use it. The change will limit the label to a maximum of 64 char.
If the build up `podName` turns out to be empty we should substitute at that point instead of the generic rule.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-yunikorn-k8shim] yangwwei commented on a change in pull
request #88: [YUNIKORN-54] admission-controller should keep generated ID
tight
Posted by GitBox <gi...@apache.org>.
yangwwei commented on a change in pull request #88: [YUNIKORN-54] admission-controller should keep generated ID tight
URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/88#discussion_r401314082
##########
File path: pkg/plugin/admissioncontrollers/webhook/admission_controller.go
##########
@@ -138,25 +138,7 @@ func updateLabels(pod *v1.Pod, patch []patchOperation) []patchOperation {
if _, ok := existingLabels[common.SparkLabelAppID]; !ok {
if _, ok := existingLabels[common.LabelApplicationID]; !ok {
- // if app id not exist, generate one
- // the generated ID is using [PodNamespace]_[PodName]_[Timestamp] naming convention.
- // some admission controllers have strict checks of the length/format of each labels,
- // this convention keeps the name tidy and short.
- podNamespace := "default"
- if pod.Namespace != "" {
- podNamespace = pod.Namespace
- }
-
- // pod's name can be generated, if name is not explicitly specified
- // look for generateName instead
- podName := "unknown"
- if pod.Name != "" {
- podName = pod.Name
- } else if pod.GenerateName != "" {
- podName = pod.GenerateName
- }
-
- generatedID := fmt.Sprintf("%s_%s_%d", podNamespace, podName, time.Now().Unix())
+ generatedID := fmt.Sprintf("__app_%d", time.Now().UnixNano())
Review comment:
Thanks for the review. I think that makes sense, we can do that.
In my latest commit, I use a very similar approach like @wilfred-s suggested, the only difference is to maintain a full timestamp to ensure the ID would be unique (pod name might be the same). UT added. Please help to review, thanks!
BTW, the max length is 63 chars instead of 64. See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-yunikorn-k8shim] adamantal commented on a change in pull
request #88: [YUNIKORN-54] admission-controller should keep generated ID
tight
Posted by GitBox <gi...@apache.org>.
adamantal commented on a change in pull request #88: [YUNIKORN-54] admission-controller should keep generated ID tight
URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/88#discussion_r400832078
##########
File path: pkg/plugin/admissioncontrollers/webhook/admission_controller.go
##########
@@ -138,25 +138,7 @@ func updateLabels(pod *v1.Pod, patch []patchOperation) []patchOperation {
if _, ok := existingLabels[common.SparkLabelAppID]; !ok {
if _, ok := existingLabels[common.LabelApplicationID]; !ok {
- // if app id not exist, generate one
- // the generated ID is using [PodNamespace]_[PodName]_[Timestamp] naming convention.
- // some admission controllers have strict checks of the length/format of each labels,
- // this convention keeps the name tidy and short.
- podNamespace := "default"
- if pod.Namespace != "" {
- podNamespace = pod.Namespace
- }
-
- // pod's name can be generated, if name is not explicitly specified
- // look for generateName instead
- podName := "unknown"
- if pod.Name != "" {
- podName = pod.Name
- } else if pod.GenerateName != "" {
- podName = pod.GenerateName
- }
-
- generatedID := fmt.Sprintf("%s_%s_%d", podNamespace, podName, time.Now().Unix())
+ generatedID := fmt.Sprintf("__app_%d", time.Now().UnixNano())
Review comment:
To me @wilfred-s suggestion make sense, assuming podName is not going to change.
It is much more informative from debugging perspective for instance to have IDs with more details in it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-yunikorn-k8shim] wilfred-s commented on a change in pull
request #88: [YUNIKORN-54] admission-controller should keep generated ID
tight
Posted by GitBox <gi...@apache.org>.
wilfred-s commented on a change in pull request #88: [YUNIKORN-54] admission-controller should keep generated ID tight
URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/88#discussion_r398573588
##########
File path: pkg/plugin/admissioncontrollers/webhook/admission_controller.go
##########
@@ -138,25 +138,7 @@ func updateLabels(pod *v1.Pod, patch []patchOperation) []patchOperation {
if _, ok := existingLabels[common.SparkLabelAppID]; !ok {
if _, ok := existingLabels[common.LabelApplicationID]; !ok {
- // if app id not exist, generate one
- // the generated ID is using [PodNamespace]_[PodName]_[Timestamp] naming convention.
- // some admission controllers have strict checks of the length/format of each labels,
- // this convention keeps the name tidy and short.
- podNamespace := "default"
- if pod.Namespace != "" {
- podNamespace = pod.Namespace
- }
-
- // pod's name can be generated, if name is not explicitly specified
- // look for generateName instead
- podName := "unknown"
- if pod.Name != "" {
- podName = pod.Name
- } else if pod.GenerateName != "" {
- podName = pod.GenerateName
- }
-
- generatedID := fmt.Sprintf("%s_%s_%d", podNamespace, podName, time.Now().Unix())
+ generatedID := fmt.Sprintf("__app_%d", time.Now().UnixNano())
Review comment:
I do not understand the change. We are going from using the pod name or generate name combined with a timestamp to just `app` and a timestamp. It would make tracking pods on the YuniKorn side really difficult.
Can we not use a length limited label setup still leveraging the name or generate name?
* remove the podNamespace (not making it more unique)
* limit the size of podName to 50 char in the printed text
* use a `_` as separator
* use the millis since 1970 (13 characters) if seconds is not good enough
```
timestamp := strconv.FormatInt(time.Now().UnixNano(), 10)
generatedID := fmt.Sprintf("%.50s_%.13s", podName, timestamp)
```
This will give a guaranteed maximum of 64 chars. We can even tweak that down if you think that is needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org
[GitHub] [incubator-yunikorn-k8shim] wilfred-s commented on a change in pull
request #88: [YUNIKORN-54] admission-controller should keep generated ID
tight
Posted by GitBox <gi...@apache.org>.
wilfred-s commented on a change in pull request #88: [YUNIKORN-54] admission-controller should keep generated ID tight
URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/88#discussion_r401320117
##########
File path: pkg/plugin/admissioncontrollers/webhook/admission_controller_test.go
##########
@@ -232,3 +258,26 @@ func TestValidateConfigMap(t *testing.T) {
assert.Equal(t, err.Error(), "required config 'queues.yaml' not found in this configmap")
// skip further validations which depends on the webservice of yunikorn-core
}
+
+func TestGenerateAppID(t *testing.T) {
+ appID := generateAppID("this-is-a-pod-name")
+ t.Logf("generated appID %s", appID)
Review comment:
These log messages show up as failures in the github runs. We don't need them to be logged for every test run, i'll remove on commit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-yunikorn-k8shim] yangwwei commented on issue #88:
[YUNIKORN-54] admission-controller should keep generated ID tight
Posted by GitBox <gi...@apache.org>.
yangwwei commented on issue #88: [YUNIKORN-54] admission-controller should keep generated ID tight
URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/88#issuecomment-607033684
Thanks @wilfred-s !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-yunikorn-k8shim] yangwwei commented on a change in pull
request #88: [YUNIKORN-54] admission-controller should keep generated ID
tight
Posted by GitBox <gi...@apache.org>.
yangwwei commented on a change in pull request #88: [YUNIKORN-54] admission-controller should keep generated ID tight
URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/88#discussion_r398630036
##########
File path: pkg/plugin/admissioncontrollers/webhook/admission_controller.go
##########
@@ -138,25 +138,7 @@ func updateLabels(pod *v1.Pod, patch []patchOperation) []patchOperation {
if _, ok := existingLabels[common.SparkLabelAppID]; !ok {
if _, ok := existingLabels[common.LabelApplicationID]; !ok {
- // if app id not exist, generate one
- // the generated ID is using [PodNamespace]_[PodName]_[Timestamp] naming convention.
- // some admission controllers have strict checks of the length/format of each labels,
- // this convention keeps the name tidy and short.
- podNamespace := "default"
- if pod.Namespace != "" {
- podNamespace = pod.Namespace
- }
-
- // pod's name can be generated, if name is not explicitly specified
- // look for generateName instead
- podName := "unknown"
- if pod.Name != "" {
- podName = pod.Name
- } else if pod.GenerateName != "" {
- podName = pod.GenerateName
- }
-
- generatedID := fmt.Sprintf("%s_%s_%d", podNamespace, podName, time.Now().Unix())
+ generatedID := fmt.Sprintf("__app_%d", time.Now().UnixNano())
Review comment:
One of the main reasons to have this change, not just to keep it < 64 char. Is the fact during the webhook phase, pod namespace, pod name could both be empty (in many cases). My suggestion is to use a general generated ID just with a timestamp, but get more info from attributes https://issues.apache.org/jira/browse/YUNIKORN-55.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: dev-help@yunikorn.apache.org
[GitHub] [incubator-yunikorn-k8shim] wilfred-s closed pull request #88:
[YUNIKORN-54] admission-controller should keep generated ID tight
Posted by GitBox <gi...@apache.org>.
wilfred-s closed pull request #88: [YUNIKORN-54] admission-controller should keep generated ID tight
URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/88
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-yunikorn-k8shim] yangwwei commented on a change in pull
request #88: [YUNIKORN-54] admission-controller should keep generated ID
tight
Posted by GitBox <gi...@apache.org>.
yangwwei commented on a change in pull request #88: [YUNIKORN-54] admission-controller should keep generated ID tight
URL: https://github.com/apache/incubator-yunikorn-k8shim/pull/88#discussion_r401314082
##########
File path: pkg/plugin/admissioncontrollers/webhook/admission_controller.go
##########
@@ -138,25 +138,7 @@ func updateLabels(pod *v1.Pod, patch []patchOperation) []patchOperation {
if _, ok := existingLabels[common.SparkLabelAppID]; !ok {
if _, ok := existingLabels[common.LabelApplicationID]; !ok {
- // if app id not exist, generate one
- // the generated ID is using [PodNamespace]_[PodName]_[Timestamp] naming convention.
- // some admission controllers have strict checks of the length/format of each labels,
- // this convention keeps the name tidy and short.
- podNamespace := "default"
- if pod.Namespace != "" {
- podNamespace = pod.Namespace
- }
-
- // pod's name can be generated, if name is not explicitly specified
- // look for generateName instead
- podName := "unknown"
- if pod.Name != "" {
- podName = pod.Name
- } else if pod.GenerateName != "" {
- podName = pod.GenerateName
- }
-
- generatedID := fmt.Sprintf("%s_%s_%d", podNamespace, podName, time.Now().Unix())
+ generatedID := fmt.Sprintf("__app_%d", time.Now().UnixNano())
Review comment:
Thanks for the review. I think that makes sense, we can do that.
In my latest commit, I use a very similar approach like @wilfred-s suggested, the only difference is to maintain a full timestamp to ensure the ID would be unique (pod name might be the same). UT added. Please help to review, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services