You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@superset.apache.org by GitBox <gi...@apache.org> on 2021/08/19 18:48:22 UTC
[GitHub] [superset] cccs-tom opened a new pull request #16361: Add extraVolumes and extraVolumeMounts to all main containers
cccs-tom opened a new pull request #16361:
URL: https://github.com/apache/superset/pull/16361
### SUMMARY
Adds `extraVolumes` and `extraVolumeMounts` objects to the Helm chart. Any volumes specified will be mounted to the main containers of all deployments as well as the init-job. Use cases include mounting additional Python modules (so they don't have to be written in-line as a YAML string) and passing in CA certs that are required by our stack.
### TESTING INSTRUCTIONS
- Add one or more `extraVolumes` and `extraVolumeMounts`.
- Run `helm install --dry-run` and confirm that the pod specs contain the new volumes
- Example:
```yaml
extraVolumes:
volume1:
configMap:
name: '{{ template "superset.fullname" . }}-custom-config'
volume2:
secret:
secretName: my-secret
extraVolumeMounts:
volume1:
mountPath: /mnt/volume1
volume2:
mountPath: /mnt/volume2
```
### ADDITIONAL INFORMATION
<!--- Check any relevant boxes with "x" -->
<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->
- [ x ] Has associated issue: #16359
- [ ] Required feature flags:
- [ ] Changes UI
- [ ] Includes DB Migration (follow approval process in [SIP-59](https://github.com/apache/superset/issues/13351))
- [ ] Migration is atomic, supports rollback & is backwards-compatible
- [ ] Confirm DB migration upgrade and downgrade tested
- [ ] Runtime estimates and downtime expectations provided
- [ ] Introduces new feature or API
- [ ] Removes existing feature or API
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] cccs-tom commented on a change in pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
cccs-tom commented on a change in pull request #16361:
URL: https://github.com/apache/superset/pull/16361#discussion_r693216536
##########
File path: helm/superset/templates/init-job.yaml
##########
@@ -76,5 +80,9 @@ spec:
configMap:
name: {{ template "superset.fullname" . }}-extra-config
{{- end }}
+ {{- range $volumeName, $volume := .Values.extraVolumes }}
Review comment:
I've just pushed this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] cccs-tom commented on a change in pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
cccs-tom commented on a change in pull request #16361:
URL: https://github.com/apache/superset/pull/16361#discussion_r692968489
##########
File path: helm/superset/templates/init-job.yaml
##########
@@ -76,5 +80,9 @@ spec:
configMap:
name: {{ template "superset.fullname" . }}-extra-config
{{- end }}
+ {{- range $volumeName, $volume := .Values.extraVolumes }}
Review comment:
Are you suggesting changing this to a `list` instead of a `map`? The reason I implemented it this way is that my team manages multiple clusters (with one deployment of Superset per cluster). It is useful to have one "base configuration" with common values, which can then be appended to in the more specific values. Making this a list would force us to repeat this part of our configuration in multiple locations, which I was trying to avoid.
Having said that, I'm open to changing it. I'm just trying to better understand your comment:
> That would allow users to customize extra volume options, etc.
With the way I implemented it, users are indeed free to specify whatever options they require. I've added a few extra options to the examples I committed in response to your previous comment. The only potential issue I can think of is that the volume names are slightly restricted by what Helm considers a valid key. Everything else will be used as-is by this line (85):
```yaml
{{- tpl (toYaml $volume) $ | nindent 10 -}}
```
Am I misunderstanding your comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] cccs-tom commented on a change in pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
cccs-tom commented on a change in pull request #16361:
URL: https://github.com/apache/superset/pull/16361#discussion_r693098848
##########
File path: helm/superset/templates/init-job.yaml
##########
@@ -76,5 +80,9 @@ spec:
configMap:
name: {{ template "superset.fullname" . }}-extra-config
{{- end }}
+ {{- range $volumeName, $volume := .Values.extraVolumes }}
Review comment:
I agree. So using the example you linked to, that would turn into:
```yaml
extraVolumes:
volumeName:
emptyDir: {}
extraVolumeMounts:
volumeName:
mountPath: "/opt/user/dir/mount"
```
The syntax is a little different, but I believe the flexibility is the same, is it not? With the added bonus that additional volumes can be appended without overriding the entire list (as per the use case I laid out in my previous reply).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a change in pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on a change in pull request #16361:
URL: https://github.com/apache/superset/pull/16361#discussion_r693103654
##########
File path: helm/superset/templates/init-job.yaml
##########
@@ -76,5 +80,9 @@ spec:
configMap:
name: {{ template "superset.fullname" . }}-extra-config
{{- end }}
+ {{- range $volumeName, $volume := .Values.extraVolumes }}
Review comment:
Actually, it would look like:
```yaml
extraVolumes:
- name: "cool-volume"
emptyDir: {}
extraVolumeMounts:
- name: "cool-volume"
mountPath: "/opt/user/dir/mount"
```
This allows us to add things like `defaultMode`, etc. to the volume definitions. We can also mount from secrets, or configMaps without changing this chart.
##########
File path: helm/superset/templates/init-job.yaml
##########
@@ -76,5 +80,9 @@ spec:
configMap:
name: {{ template "superset.fullname" . }}-extra-config
{{- end }}
+ {{- range $volumeName, $volume := .Values.extraVolumes }}
Review comment:
Actually, it would look like:
```yaml
extraVolumes:
- name: "cool-volume"
emptyDir: {}
extraVolumeMounts:
- name: "cool-volume"
mountPath: "/opt/user/dir/mount"
```
This allows us to add things like `defaultMode`, etc. to the volume definitions. We can also mount from secrets, or configMaps without changing this chart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] cccs-tom commented on a change in pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
cccs-tom commented on a change in pull request #16361:
URL: https://github.com/apache/superset/pull/16361#discussion_r692960285
##########
File path: helm/superset/values.yaml
##########
@@ -82,6 +82,10 @@ extraConfigs: {}
extraSecrets: {}
+extraVolumes: {}
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a change in pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on a change in pull request #16361:
URL: https://github.com/apache/superset/pull/16361#discussion_r692515654
##########
File path: helm/superset/templates/init-job.yaml
##########
@@ -76,5 +80,9 @@ spec:
configMap:
name: {{ template "superset.fullname" . }}-extra-config
{{- end }}
+ {{- range $volumeName, $volume := .Values.extraVolumes }}
Review comment:
Can we just range over all of these and just call toyaml on them directly? That would allow users to customize extra volume options, etc.
##########
File path: helm/superset/values.yaml
##########
@@ -82,6 +82,10 @@ extraConfigs: {}
extraSecrets: {}
+extraVolumes: {}
Review comment:
Please add an example of each so people know how to use these
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a change in pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on a change in pull request #16361:
URL: https://github.com/apache/superset/pull/16361#discussion_r693118570
##########
File path: helm/superset/templates/init-job.yaml
##########
@@ -76,5 +80,9 @@ spec:
configMap:
name: {{ template "superset.fullname" . }}-extra-config
{{- end }}
+ {{- range $volumeName, $volume := .Values.extraVolumes }}
Review comment:
I think we should use a list rather than a map here. Nested maps are a little tricky to follow in this context. The `volumes` and `volumeMounts` stanzas in Deployments, etc. use lists, so it makes the transition pretty straight forward for end users to just drop their list into this chart's values override.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] cccs-tom commented on a change in pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
cccs-tom commented on a change in pull request #16361:
URL: https://github.com/apache/superset/pull/16361#discussion_r693109767
##########
File path: helm/superset/templates/init-job.yaml
##########
@@ -76,5 +80,9 @@ spec:
configMap:
name: {{ template "superset.fullname" . }}-extra-config
{{- end }}
+ {{- range $volumeName, $volume := .Values.extraVolumes }}
Review comment:
What I meant was that, if you modify the example in the way I wrote it in my reply, you would get the exact same YAML output as if I made the changes you requested.
My intent was to allow maximum flexibility. As far as I know, this implementation allows 100% of the `volumes` and `volumeMounts` spec. Could you provide an example of something that isn't supported with this implementation to help me better understand, please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a change in pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on a change in pull request #16361:
URL: https://github.com/apache/superset/pull/16361#discussion_r693103654
##########
File path: helm/superset/templates/init-job.yaml
##########
@@ -76,5 +80,9 @@ spec:
configMap:
name: {{ template "superset.fullname" . }}-extra-config
{{- end }}
+ {{- range $volumeName, $volume := .Values.extraVolumes }}
Review comment:
Actually, it would look like:
```yaml
extraVolumes:
- name: "cool-volume"
emptyDir: {}
extraVolumeMounts:
- name: "cool-volume"
mountPath: "/opt/user/dir/mount"
```
This allows us to add things like `defaultMode`, etc. to the volume definitions. We can also mount from secrets, or configMaps without changing this chart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda commented on a change in pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
craig-rueda commented on a change in pull request #16361:
URL: https://github.com/apache/superset/pull/16361#discussion_r693049758
##########
File path: helm/superset/templates/init-job.yaml
##########
@@ -76,5 +80,9 @@ spec:
configMap:
name: {{ template "superset.fullname" . }}-extra-config
{{- end }}
+ {{- range $volumeName, $volume := .Values.extraVolumes }}
Review comment:
Makes total sense. I just think that most folks will run into a spot where they need an extra option for the volume, and will need to add a PR to get the additional config they need shoved in.
Something like this is what I was thinking:
https://github.com/Kong/charts/blob/main/charts/kong/values.yaml#L35
It really allows for the most flexibility possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org
[GitHub] [superset] craig-rueda merged pull request #16361: feat: Add extraVolumes and extraVolumeMounts to all main containers
Posted by GitBox <gi...@apache.org>.
craig-rueda merged pull request #16361:
URL: https://github.com/apache/superset/pull/16361
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@superset.apache.org
For additional commands, e-mail: notifications-help@superset.apache.org