You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/10 09:55:24 UTC
[GitHub] [airflow] mikegit21 opened a new issue, #24369: Keytab file is not mounted in worker-kerberos container
mikegit21 opened a new issue, #24369:
URL: https://github.com/apache/airflow/issues/24369
### Official Helm Chart version
1.5.0
### Apache Airflow version
2.2.4
### Kubernetes Version
1.20+
### Helm Chart configuration
```
kerberos:
enabled: true
ccacheMountPath: /var/kerberos-ccache
ccacheFileName: cache
configPath: /etc/krb5.conf
keytabBase64Content: "<base64contentofkeytabfile>"
keytabPath: /etc/airflow.keytab
principal: <principal>
reinitFrequency: 3600
config: "|
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
[libdefaults]
default_realm = REALM1
dns_lookup_realm = false
dns_lookup_kdc = false
renew_lifetime = 7d
forwardable = true
udp_preference_limit = 1
kdc_timeout = 3000
[realms]
REALM1 = {
kdc = x
kdc = x
}
REALM2 = {
kdc = x
kdc = x
kdc = x
kdc = x
}
[domain_realm]
mapping = mapping"
```
### Docker Image customisations
```
FROM apache/airflow:2.2.4
# Switch user because otherwise installing dependencies will not work
USER root
# Update package index and upgrade packages to prevent security issues
RUN apt-get update && apt-get upgrade -y
# Needed to install custom package
RUN apt-get install build-essential unixodbc-dev libkrb5-dev -y
USER airflow
RUN pip install pipenv
COPY Pipfile Pipfile.lock /
RUN pipenv install --system
```
### What happened
When we are executing the Airflow pipeline which needs to connect to the kerberos database we are getting the following error:
```
File "/home/airflow/.local/lib/python3.7/site-packages/impala/dbapi.py", line 167, in connect
retries=retries)
File "/home/airflow/.local/lib/python3.7/site-packages/impala/hiveserver2.py", line 862, in connect
transport.open()
File "/home/airflow/.local/lib/python3.7/site-packages/thrift_sasl/__init__.py", line 82, in open
ret, chosen_mech, initial_response = self.sasl.start(self.mechanism)
File "/home/airflow/.local/lib/python3.7/site-packages/impala/sasl_compat.py", line 24, in start
return True, self.mechanism, self.process()
File "/home/airflow/.local/lib/python3.7/site-packages/puresasl/client.py", line 16, in wrapped
return f(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/puresasl/client.py", line 148, in process
return self._chosen_mech.process(challenge)
File "/home/airflow/.local/lib/python3.7/site-packages/puresasl/mechanisms.py", line 505, in process
kerberos.authGSSClientStep(self.context, '')
kerberos.GSSError: (('Unspecified GSS failure. Minor code may provide more information', 851968), ('Server not found in Kerberos database', -1765328377))
```
This is intersting because we use the exact same krb5.conf, keytab file and principal for accessing the database in another tool. I jumped into the `worker-kerberos` container and verified the `etc/krb5.conf` and the principal in the `opt/airflow/airflow.cfg` - they do look good. But I stumbled about the fact that I cannot find the `etc/airflow.keytab` file, which from what I understand should be mounted into the container based on the secret that the airflow helm chart creates. I suspect that the missing keytab file actually causes the problem as it contains the login information needed to connect to the kerberos database.
I also verified that the `airflow-kerberos-keytab` secret exists. It looks like this:
```
apiVersion: v1
kind: Secret
metadata:
name: airflow-kerberos-keytab
annotations:
meta.helm.sh/release-name: airflow
meta.helm.sh/release-namespace: namespace
type: Opaque
data:
kerberos.keytab: <base64-content>
```
The base64 content looks good to me.
### What you think should happen instead
The airflow.keytab file should be mounted into the `worker-kerberos` container, which in my opinion should solve the kerberos connection error.
### How to reproduce
You can reproduce this issue by using a similar kerberos configuration like the one I provided and then trying to access the database in a DAG.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1156341617
Also in `tests/charts/test_kerberos.py` you will find kerberos-related tests that do some generation of krb5.conf - maynbe those tests will give you some hints (they are actually running `helm template` under the hood and you can even grab generated template this way and analyse it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mikegit21 commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by GitBox <gi...@apache.org>.
mikegit21 commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1153814172
I was accidentally looking in the `worker` container, I can see the keytab file in the `worker-kerberos` container and was able to check it using `ktutil`- looks good to me. The only thing I could think of would be a problem with the `krb5.conf`. I specified the log files like this:
```
[logging]
default = "FILE:{{ template "airflow_logs_no_quote" . }}/kerberos_libs.log"
kdc = "FILE:{{ template "airflow_logs_no_quote" . }}/kerberos_kdc.log"
```
But couldn't find anything in `opt/airflow/` so I am a little bit lost what could be the problem. The config is correct as we are using it in another application (but as a seperate `.conf` file in this case). So it can only be some sort of formatting issue that I am not aware of.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] npsables commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by "npsables (via GitHub)" <gi...@apache.org>.
npsables commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1429052945
I want to reuse kerberos.keytab file in workers but can't because **the helm didn't mount it** to worker container. The sidecar (airflow kerberos command) is also broken in my build. But I resolve all of this by reduce to 1 worker, mount /etc/kerberos to this, disable kerberos sidecar, and schedule an airflow task to refresh the kerberos ticket.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1153290849
You seem to know exactly what the problem is (and have a way of testing it) - would you maybe like to add a PR fixing it ? It does not seem like complex thing to do - testing and having a good environment to do so, seem like the most complex part.
Would you lile to pay back and contribute back to Airflow and become one of the > 2K contributors to Apache Airflow @mikegit21 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1669256893
Indeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1152187635
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mikegit21 commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by GitBox <gi...@apache.org>.
mikegit21 commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1153549881
I can try to fix it in the HELM chart locally and if it solves the problem I could create a PR.
Otherwise I will come back to this thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1153554649
> I can try to fix it in the HELM chart locally and if it solves the problem I could create a PR.
>
> Otherwise I will come back to this thread
Perfect! Thanks. Happy to help with your PR - just Ping me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] amoghrajesh commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by "amoghrajesh (via GitHub)" <gi...@apache.org>.
amoghrajesh commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1668888023
Looks to me that this issue has been fixed. Can I close it? @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] npsables commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by "npsables (via GitHub)" <gi...@apache.org>.
npsables commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1427727748
> Author
Hi [mikegit21](https://github.com/mikegit21), did you resolve this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1156336456
Hard to say without deep looking, I know that kerberos was notoriously difficult to configure and even slightest typos made it stop working. I am pretty sure though that kerberos was working at some point in time, and there are tests covering that. I think you'd need to deep e bit deeper and try to see maybe some typo is there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24369:
URL: https://github.com/apache/airflow/issues/24369#issuecomment-1156338467
You can see "test_kerberos_auth.py" for example - we actually start kerberos integration there and run testing with a test krb.conf - maybe trying to reproduce it there will help (you need to start breeze with `--integration kerberos` enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #24369: Keytab file is not mounted in worker-kerberos container
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #24369: Keytab file is not mounted in worker-kerberos container
URL: https://github.com/apache/airflow/issues/24369
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org