You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "amoghrajesh (via GitHub)" <gi...@apache.org> on 2023/11/20 06:02:31 UTC
[PR] Connecting airflow containers with shareProcessNamespace [airflow]
amoghrajesh opened a new pull request, #35748:
URL: https://github.com/apache/airflow/pull/35748
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
As a part of the effort to enable kerberos to work e2e with airflow in a kubernetes deployment, some changes related to the pod template and worker deployment have gone in recently in PR: #35146 and #35548.
However, one problem remains. Which is the airflow sidecar container not exiting gracefully even after the "base" container completes or exits. To get this working, I was able to do a POC by making airflow kerberos container and airflow base container talk using shareProcessNamespace and then using "args" field of airflow kerberos sidecar to watch the base container periodically and kill the process of itself if airflow base container has exited. This doesn't add a lot onto the performance side as well because the command run by airflow kerberos will essentially be a finding a `pid` in `/proc/*/`.
This solution has been tested out on a secure environment like Redhat Openshift in conjunction to on prem kubernetes distribution.
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Connecting airflow containers with shareProcessNamespace [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1820451617
>So I'd say we should treat native side-containers as THE solution and the others are merely workarounds for versions of kubernwtes that do not have native side containers yet.
Right. This would be a good way forward & we should look forward at going this route. I am going to try this out by deploying a 1.28 k8s cluster, hopefully things will be good there 👍🏽
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Connecting airflow containers with shareProcessNamespace [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1818945067
Shared process namespace in this case kind of violates the whole assumption the separate kerberos container here is introduced for. The whole idea is that "airflow" containers do not see the kerberors / keytab used to refresh the short time token. Only kerberos container/refreshing process should ever be able to get access to keytab.
Keytab are long-living and provide full access to kerberos service. They use symmetric encryption and once you get hold of it, you are able to communicate with Kerberos server and obtain the short living ticket to do the job you are supposed to do - so it should be very strongly guarded property.
Generally Airflow components should never be able to see or obtain keytab files (only `airflow kerberos` refreshing container should have access to it) - instead all the components should only access the short-living ticket. This is the basic assumption that the whole separtion of the containers is based on - the two containers share only filesystem where the ticket is refreshed and nothing else.
See https://www.fortinet.com/resources/cyberglossary/kerberos-authentication for example.
If airflow components (specifically worker) will get access to keytab, someone could write a DAG to - for example - send the keytab to a remote system and once this happens and such keytab can be used by anyone to do anything. When you do the same with short living ticket, you are limited to only what the service allows the ticket to do + it's time limited so exposure of such breach is potentially much more limited.
Of course, ysing SharedProcessNamespace is not explicitly violating this assumption. It does not give the airflow component direct access to the keytab. So far so good. But it opens a way to other ways of obtaining the keytab by malicious actors. For example, you could connect to the kerberos process with gdb and dump the memory of it and retrieve the keytab from memory of that process. And this is only one of the ways you could retrieve that information, there are many others. Some of them you might protect better from, but generally speaking, it significantly decreases the isolation that container mechanism introduces (and the reason why the two are run in separate containers).
So while this solution is not directly giving access to the keytab, it does decrease isolation and introduces security risks. I am quite sure when we introduce it, this will be flagged as a security issue and we will have to fix it.
I think there are two options:
1) we should figure out a different way how to communicate with the kerberos refreshing process and stop it - one of the options is to let the running process be stoppable by sending a "shutdown" message via TCP connection - and expose the sidecar container's port to the main containers. Another option (probably simpler to implement) is to keep some kind of "shutdown" lock file in the same filesystem where the ticket is stored and signal the rerfreshing process that it should exit. It could be based on lock mechanism available natively in python.
2) If we limit the solution to only kubernetes 1.28+, we have the option of using Native Sidecar Containers and marking the kerberos sidecar as such https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/
Likely the best is combination of these 1) (for k8s < 1.28) and 2) (for k8s >= 1.28).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1422011784
##########
chart/templates/_helpers.yaml:
##########
@@ -997,3 +997,27 @@ capabilities:
app: keda-operator
{{- end }}
{{- end }}
+
+{{- define "kubernetesVersion" -}}
+{{ printf "%s" .Capabilities.KubeVersion.Version | trimPrefix "v" }}
+{{- end -}}
+
+{{- define "airflowKerberosContainerArgs" -}}
+{{- $kubectl_version := include "kubernetesVersion" . -}}
+{{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) (semverCompare "<1.28" $kubectl_version) }}
Review Comment:
> Was down with a bad flu whole of last week :/
Keep healty . This is very important. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Connecting airflow containers with shareProcessNamespace [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1818979974
Thank you for the detailed review @potiuk. This is extremely helpful to me.
> Another option (probably simpler to implement) is to keep some kind of "shutdown" lock file in the same filesystem where the ticket is stored and signal the rerfreshing process that it should exit. It could be based on lock mechanism available natively in python.
I like this idea. It is simple yet effective. I would like to explore more along these lines and get back in case of any doubts.
>If we limit the solution to only kubernetes 1.28+, we have the option of using Native Sidecar Containers and marking the kerberos sidecar as such https://kubernetes.io/blog/2023/08/25/native-sidecar-containers/
Came across this too. It is not recommended as of now since it is in its alpha stage. However, few questions here:
1. If 1) works for old and new k8s, no need to have 2) at all?
2. Do we have any utils to help us identify k8s version at the time of helm install itself?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1829266618
Pending:
1. Unit tests for the changes
2. Regressive testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1407285055
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -256,6 +257,10 @@ spec:
- name: worker-logs
containerPort: {{ .Values.ports.workerLogs }}
volumeMounts:
+ {{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) .Values.workers.kerberosSidecar.enabled (semverCompare "<1.28" $kubeVersion) }}
Review Comment:
Looks like 2.8 will change now, no? I see the RC out for 2.8 already.
cc @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1869943410
@potiuk made some changes to the PR. Can you take a look when you have some time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1432291298
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -232,7 +233,13 @@ spec:
{{- if .Values.workers.command }}
command: {{ tpl (toYaml .Values.workers.command) . | nindent 12 }}
{{- end }}
- {{- if .Values.workers.args }}
+ {{- if and .Values.workers.kerberosSidecar.enabled (semverCompare "<1.28" $kubeVersion) }}
+ args:
+ {{ tpl (toYaml .Values.workers.args) . | nindent 12 }} &&
Review Comment:
I tried something like this:
```
{{- if and .Values.workers.kerberosSidecar.enabled (semverCompare "<1.28" $kubeVersion) }}
args:
- {{ tpl (toYaml .Values.workers.args) . | nindent 12 }}
- "&&"
- "sh"
- "-c"
- >
echo "EXIT file content" > /opt/kerberos-exit/EXIT
{{- else if .Values.workers.args }}
args: {{ tpl (toYaml .Values.workers.args) . | nindent 12 }}
{{- end }}
```
This evaluates to:
```
args:
-
- bash
- -c
- |-
exec \
airflow celery worker
- "&&"
- "sh"
- "-c"
- >
echo "EXIT file content" > /opt/kerberos-exit/EXIT
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Connecting airflow containers with shareProcessNamespace [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1820450496
>I also think native sidecar is the long term way to go. It's been long time in the making - KEP for it has been discussed for 3 years at least - and it handles many edge cases and hides a lot of complexity under the hood of simple label.
Thanks for letting me know about this. Was not very aware of this. If this is the case, we should consider this as "THE ONLY" solution along with lock file as you mentioned for earlier versions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1863922172
@potiuk I made a change and I think it will work because:
In this example, the && is part of the command inside the shell script. This way, the entire script is executed, including the command after &&.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1430370771
##########
chart/templates/_helpers.yaml:
##########
@@ -997,3 +997,27 @@ capabilities:
app: keda-operator
{{- end }}
{{- end }}
+
+{{- define "kubernetesVersion" -}}
+{{ printf "%s" .Capabilities.KubeVersion.Version | trimPrefix "v" }}
+{{- end -}}
+
+{{- define "airflowKerberosContainerArgs" -}}
+{{- $kubectl_version := include "kubernetesVersion" . -}}
+{{- if (semverCompare "<1.28" $kubectl_version) }}
+- airflow
+- kerberos
+- "&"
+- while [ ! -f /opt/kerberos-exit/EXIT ]; do
Review Comment:
Maybe I am missing something - but when/where the EXIT file gets created
##########
chart/templates/_helpers.yaml:
##########
@@ -997,3 +997,27 @@ capabilities:
app: keda-operator
{{- end }}
{{- end }}
+
+{{- define "kubernetesVersion" -}}
+{{ printf "%s" .Capabilities.KubeVersion.Version | trimPrefix "v" }}
+{{- end -}}
+
+{{- define "airflowKerberosContainerArgs" -}}
+{{- $kubectl_version := include "kubernetesVersion" . -}}
+{{- if (semverCompare "<1.28" $kubectl_version) }}
+- airflow
+- kerberos
+- "&"
+- while [ ! -f /opt/kerberos-exit/EXIT ]; do
Review Comment:
Maybe I am missing something - but when/where the EXIT file gets created ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Connecting airflow containers with shareProcessNamespace [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1823829483
> I think we have ways to generate different resources for 1.28+ . I believe helm takes K8s version into account when generating resources. Look in our chart I think we have a few cases of that
@potiuk I did check for this. Unfortunately, we do not seem to have such a mechanism. We will eventually need it to get the 1.28 features and this is also useful in a long run.
I was thinking of introducing an env variable at the worker deployment level where the main container has the env that holds the kube version. We can conditionally set the args for containers and do a lot more things too.
For example:
```
containers:
- name: base
env:
- name: KUBE_VERSION
valueFrom:
configMapKeyRef:
name: your-scripts
key: kubeVersionScript.sh
volumeMounts:
- name: script-volume
mountPath: /scripts
args:
{{- if eq .Env.KUBE_VERSION "1.25" }}
- /bin/sh
- -c
- |
echo "Kube version is 1.25, do something"
{{- else if eq .Env.KUBE_VERSION "1.26" }}
- /bin/sh
- -c
- |
echo "Kube version is 1.26, do something else"
{{- else }}
- /bin/sh
- -c
- |
echo "Kube version is not 1.25 or 1.26, do default"
{{- end }}
initContainers:
- name: init-script
image: your-image
command: ["/bin/sh", "-c"]
args:
- |
export KUBE_VERSION=$(source /scripts/kubeVersionScript.sh)
volumeMounts:
- name: script-volume
mountPath: /scripts
volumes:
- name: script-volume
configMap:
name: your-scripts
```
Thoughts?
cc @jedcunningham looping you in. Pls correct me if I am misinformed here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Connecting airflow containers with shareProcessNamespace [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1818986629
>we should figure out a different way how to communicate with the kerberos refreshing process and stop it - one of the options is to let the running process be stoppable by sending a "shutdown" message via TCP connection - and expose the sidecar container's port to the main containers
This is not a bad option either. The main container could send across a kill signal through TCP as part of its postRun hook or so. That way, it would send the signal and exit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #35748: Handling exit of kerberos sidecar container for k8s < 1.28
URL: https://github.com/apache/airflow/pull/35748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1421932601
##########
chart/templates/_helpers.yaml:
##########
@@ -997,3 +997,27 @@ capabilities:
app: keda-operator
{{- end }}
{{- end }}
+
+{{- define "kubernetesVersion" -}}
+{{ printf "%s" .Capabilities.KubeVersion.Version | trimPrefix "v" }}
+{{- end -}}
+
+{{- define "airflowKerberosContainerArgs" -}}
+{{- $kubectl_version := include "kubernetesVersion" . -}}
+{{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) (semverCompare "<1.28" $kubectl_version) }}
Review Comment:
@potiuk Will work on adding the tests. Was down with a bad flu whole of last week :/
Hmm, I added 2.8.0 to avoid any regression i thought could occur due to airflow upgrades but now that I think of it, this is not a new feature, but an update to the existing behaviour. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1423386253
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -342,9 +347,13 @@ spec:
{{- if $containerLifecycleHooksKerberosSidecar }}
lifecycle: {{- tpl (toYaml $containerLifecycleHooksKerberosSidecar) . | nindent 12 }}
{{- end }}
- args: ["kerberos"]
+ args: {{ include "airflowKerberosContainerArgs" . | nindent 12 }}
resources: {{- toYaml .Values.workers.kerberosSidecar.resources | nindent 12 }}
volumeMounts:
+ {{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) (semverCompare "<1.28" $kubeVersion) }}
Review Comment:
@potiuk maybe we should actually remove the checks everywhere tbh.
##########
chart/templates/_helpers.yaml:
##########
@@ -997,3 +997,27 @@ capabilities:
app: keda-operator
{{- end }}
{{- end }}
+
+{{- define "kubernetesVersion" -}}
+{{ printf "%s" .Capabilities.KubeVersion.Version | trimPrefix "v" }}
+{{- end -}}
+
+{{- define "airflowKerberosContainerArgs" -}}
+{{- $kubectl_version := include "kubernetesVersion" . -}}
+{{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) (semverCompare "<1.28" $kubectl_version) }}
Review Comment:
Thank you. Taking care of myself :)
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -387,6 +396,10 @@ spec:
{{- if .Values.workers.extraVolumes }}
{{- tpl (toYaml .Values.workers.extraVolumes) . | nindent 8 }}
{{- end }}
+ {{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) .Values.workers.kerberosSidecar.enabled (semverCompare "<1.28" $kubeVersion) }}
Review Comment:
Not needed here as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1432286715
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -232,7 +233,13 @@ spec:
{{- if .Values.workers.command }}
command: {{ tpl (toYaml .Values.workers.command) . | nindent 12 }}
{{- end }}
- {{- if .Values.workers.args }}
+ {{- if and .Values.workers.kerberosSidecar.enabled (semverCompare "<1.28" $kubeVersion) }}
+ args:
+ {{ tpl (toYaml .Values.workers.args) . | nindent 12 }} &&
Review Comment:
Thanks for that @potiuk
Looking for alternatives, any ideas on this front?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1972190051
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1422011496
##########
chart/templates/_helpers.yaml:
##########
@@ -997,3 +997,27 @@ capabilities:
app: keda-operator
{{- end }}
{{- end }}
+
+{{- define "kubernetesVersion" -}}
+{{ printf "%s" .Capabilities.KubeVersion.Version | trimPrefix "v" }}
+{{- end -}}
+
+{{- define "airflowKerberosContainerArgs" -}}
+{{- $kubectl_version := include "kubernetesVersion" . -}}
+{{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) (semverCompare "<1.28" $kubectl_version) }}
Review Comment:
Yes. No need to Add 2.8.0 in this case as we are not using any new feature added in `kerberos` command as I understand ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1432294074
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -232,7 +233,13 @@ spec:
{{- if .Values.workers.command }}
command: {{ tpl (toYaml .Values.workers.command) . | nindent 12 }}
{{- end }}
- {{- if .Values.workers.args }}
+ {{- if and .Values.workers.kerberosSidecar.enabled (semverCompare "<1.28" $kubeVersion) }}
+ args:
+ {{ tpl (toYaml .Values.workers.args) . | nindent 12 }} &&
Review Comment:
And adding a trim in helm makes it like this:
```
args:
- - bash
- -c
- |-
exec \
airflow celery worker
- "&&"
- "sh"
- "-c"
- >
echo "EXIT file content" > /opt/kerberos-exit/EXIT
```
Thinking of some alternatives :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1438559079
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -232,7 +233,14 @@ spec:
{{- if .Values.workers.command }}
command: {{ tpl (toYaml .Values.workers.command) . | nindent 12 }}
{{- end }}
- {{- if .Values.workers.args }}
+ {{- if and .Values.workers.kerberosSidecar.enabled (semverCompare "<1.28" $kubeVersion) }}
+ args:
+ - bash
+ - -c
+ - |-
+ exec airflow celery worker &&
Review Comment:
I do not think it's going to work either. The exec will replace bash interpreter at the moment `airflow celery worker` executes and what is after `&&` will never get executed.
You can check it by just
```
bash -c "exec ls && echo TEST"
```
It will only list current directory but it will not echo TEST.
You just can't `exec` when you are running bash-interpreted command as at the moment of exec, the process that you run replaces `bash` interpreter.
What you **really** need to do here if you want to use shell, you need to run the worker without exec, but you also need to make sure signal propagation happens - so that the parent bash process wil propagate the signals. However maybe you do not need to do it when you are using our image - because we are using
`dumb_init` as entrypoint with `DUMB_INIT_SETSID` set to 1 which means that dumb-init will propagate signals to ALL the children of bash, not only to bash, in which case
```
bash -c "airflow celery worker && sh -c "echo 'EXIT file content' > /opt/kerberos-exit/EXIT"
````
as command might just work - but you need to test it. run it in kubernetes, find the process of worker, send a signal (say TERM) and see if it completes and if it correctly writes the EXIT file.
https://airflow.apache.org/docs/docker-stack/entrypoint.html#signal-propagation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1430370146
##########
chart/templates/_helpers.yaml:
##########
@@ -997,3 +997,27 @@ capabilities:
app: keda-operator
{{- end }}
{{- end }}
+
+{{- define "kubernetesVersion" -}}
+{{ printf "%s" .Capabilities.KubeVersion.Version | trimPrefix "v" }}
+{{- end -}}
+
+{{- define "airflowKerberosContainerArgs" -}}
+{{- $kubectl_version := include "kubernetesVersion" . -}}
+{{- if (semverCompare "<1.28" $kubectl_version) }}
+- airflow
+- kerberos
+- "&"
+- while [ ! -f /opt/kerberos-exit/EXIT ]; do
+ if [ -f /opt/kerberos-exit/EXIT ]; then
+ echo "Exit lockfile found. Exiting kerberos process."
+ break
+ else
+ echo "Exit lockfile not found. Sleeping for 1 second..."
+ sleep 5
+ fi
+ done
+{{- else }}
+- kerberos
Review Comment:
Should not it be:
```
- airflow
- kerberos
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1421827060
##########
chart/templates/_helpers.yaml:
##########
@@ -997,3 +997,27 @@ capabilities:
app: keda-operator
{{- end }}
{{- end }}
+
+{{- define "kubernetesVersion" -}}
+{{ printf "%s" .Capabilities.KubeVersion.Version | trimPrefix "v" }}
+{{- end -}}
+
+{{- define "airflowKerberosContainerArgs" -}}
+{{- $kubectl_version := include "kubernetesVersion" . -}}
+{{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) (semverCompare "<1.28" $kubectl_version) }}
Review Comment:
We need unit tests as well. But I wonder - why `Airflow >=2.8.0` ? We d o not seem to add any functionality for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1451962759
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -232,7 +233,14 @@ spec:
{{- if .Values.workers.command }}
command: {{ tpl (toYaml .Values.workers.command) . | nindent 12 }}
{{- end }}
- {{- if .Values.workers.args }}
+ {{- if and .Values.workers.kerberosSidecar.enabled (semverCompare "<1.28" $kubeVersion) }}
+ args:
+ - bash
+ - -c
+ - |-
+ exec airflow celery worker &&
Review Comment:
Thanks for the comment. Looking into what is the best way to do this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Connecting airflow containers with shareProcessNamespace [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1819032429
I think we have ways to generate different resources for 1.28+ . I believe helm takes K8s version into account when generating resources. Look in our chart I think we have a few cases of that
I also think native sidecar is the long term way to go. It's been long time in the making - KEP for it has been discussed for 3 years at least - and it handles many edge cases and hides a lot of complexity under the hood of simple label.
Also the 1) case is complex to develop and has limitations - need to open TCP/IP port or having write access from airflow components to the shared filesystem also impact security properties of this setup. Currently you just need read access from worker side to the filesystem abnd you do not need TCP/iP connection - so both proposals are also weakening the isolation (though not as much as shared process does). So I'd say we should treat native side-containers as THE solution and the others are merely workarounds for versions of kubernwtes that do not have native side containers yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1829277924
Ok seems like the issue in CI was intermittent. It is green now :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Connecting airflow containers with shareProcessNamespace [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1820964266
> Thanks for letting me know about this. Was not very aware of this. If this is the case, we should consider this as "THE ONLY" solution along with lock file as you mentioned for earlier versions
You can read about it for example here: https://buoyant.io/blog/kubernetes-1-28-revenge-of-the-sidecars it explains the history (started 2019 so I was not too far off with 3 years).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Connecting airflow containers with shareProcessNamespace [airflow]
Posted by "pankajkoti (via GitHub)" <gi...@apache.org>.
pankajkoti commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1823904072
@amoghrajesh just if you wonder if Jed is jot able to respond anytime soon, Jed is on parental leave 👶
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Connecting airflow containers with shareProcessNamespace [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1398696229
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -98,6 +98,9 @@ spec:
{{- toYaml $podAnnotations | nindent 8 }}
{{- end }}
spec:
+ {{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) .Values.workers.kerberosSidecar.enabled }}
Review Comment:
Since this is a new feature that works in conjunction to #35548, i have kept it to have a semver check
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -98,6 +98,9 @@ spec:
{{- toYaml $podAnnotations | nindent 8 }}
{{- end }}
spec:
+ {{- if and (semverCompare ">=2.8.0" .Values.airflowVersion) .Values.workers.kerberosSidecar.enabled }}
Review Comment:
For now, the `shareProcessNamespace` is kept under `kerberosSidecar` section to keep it tied to kerberos. It can be moved out to workers section in case it has another use case in the near future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1829196716
@potiuk re worked the PR to support it in a better way. Can you take a look when you have some time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1851276297
> Yes. No need to Add 2.8.0 in this case as we are not using any new feature added in kerberos command as I understand ?
You are right. No new feature for airflow kerberos. Removing the 2.8.0 check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1431994543
##########
chart/templates/workers/worker-deployment.yaml:
##########
@@ -232,7 +233,13 @@ spec:
{{- if .Values.workers.command }}
command: {{ tpl (toYaml .Values.workers.command) . | nindent 12 }}
{{- end }}
- {{- if .Values.workers.args }}
+ {{- if and .Values.workers.kerberosSidecar.enabled (semverCompare "<1.28" $kubeVersion) }}
+ args:
+ {{ tpl (toYaml .Values.workers.args) . | nindent 12 }} &&
Review Comment:
I think this will not work as expected for two reasons:
1) what you can see in the test - `exec airflow celery worker` will replace the bash interpreter with the python one (this is how exec works) - this means that whatever is after `&&` will not be executed because bash interpreter will not be there any more to execute it
2) Even if it did work and we had no exect, this would not work if the worker crashed and exited with error - because in this case what's after `&&` is not executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on code in PR #35748:
URL: https://github.com/apache/airflow/pull/35748#discussion_r1430916157
##########
chart/templates/_helpers.yaml:
##########
@@ -997,3 +997,27 @@ capabilities:
app: keda-operator
{{- end }}
{{- end }}
+
+{{- define "kubernetesVersion" -}}
+{{ printf "%s" .Capabilities.KubeVersion.Version | trimPrefix "v" }}
+{{- end -}}
+
+{{- define "airflowKerberosContainerArgs" -}}
+{{- $kubectl_version := include "kubernetesVersion" . -}}
+{{- if (semverCompare "<1.28" $kubectl_version) }}
+- airflow
+- kerberos
+- "&"
+- while [ ! -f /opt/kerberos-exit/EXIT ]; do
+ if [ -f /opt/kerberos-exit/EXIT ]; then
+ echo "Exit lockfile found. Exiting kerberos process."
+ break
+ else
+ echo "Exit lockfile not found. Sleeping for 1 second..."
+ sleep 5
+ fi
+ done
+{{- else }}
+- kerberos
Review Comment:
Right, good catch.
Thank you!
##########
chart/templates/_helpers.yaml:
##########
@@ -997,3 +997,27 @@ capabilities:
app: keda-operator
{{- end }}
{{- end }}
+
+{{- define "kubernetesVersion" -}}
+{{ printf "%s" .Capabilities.KubeVersion.Version | trimPrefix "v" }}
+{{- end -}}
+
+{{- define "airflowKerberosContainerArgs" -}}
+{{- $kubectl_version := include "kubernetesVersion" . -}}
+{{- if (semverCompare "<1.28" $kubectl_version) }}
+- airflow
+- kerberos
+- "&"
+- while [ ! -f /opt/kerberos-exit/EXIT ]; do
Review Comment:
Just realised that I didn't push the dockerfile. But i realised there is a better way of doing it just using YAML. Made the changes, pushing it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Handling exit of kerberos sidecar container for k8s < 1.28 [airflow]
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on PR #35748:
URL: https://github.com/apache/airflow/pull/35748#issuecomment-1859521685
@potiuk I think some tests fail in the CI. Working on fixing those, but the PR is up to date now. Could you take a look when you have some time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org