You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/17 09:40:03 UTC
[GitHub] [airflow] Scope0910 opened a new issue #21638: Spark Connection with k8s in URL not mapped correctly
Scope0910 opened a new issue #21638:
URL: https://github.com/apache/airflow/issues/21638
### Official Helm Chart version
1.2.0
### Apache Airflow version
2.1.4
### Kubernetes Version
v1.21.6+bb8d50a
### Helm Chart configuration
I defined a new Connection String for AIRFLOW_CONN_SPARK_DEFAULT in values.yaml like the following section (base64 encoded or with correct string (spark://k8s://100.68.0.1:443?deploy-mode=cluster) :
extraSecrets:
'{{ .Release.Name }}-airflow-connections':
data: |
AIRFLOW_CONN_SPARK_DEFAULT: 'c3Bhcms6Ly9rOHM6Ly8xMDAuNjguMC4xOjQ0Mz9kZXBsb3ktbW9kZT1jbHVzdGVy'
in Section extraEnvFrom i defined the following:
extraEnvFrom: |
- secretRef:
name: '{{ .Release.Name }}-airflow-connections'
### Docker Image customisations
added apache-airflow-providers-apache-spark to base Image
### What happened
Airflow Connection mapped wrong because of the k8s:// within the url. if i ask for the connection with cmd "airflow connections get spark_default" then host=k8s and schema=/100.60.0.1:443 which is wrong
### What you expected to happen
the spark Connection based on k8s (spark://k8s://100.68.0.1:443?deploy-mode=cluster) should be parsed correctly
### How to reproduce
define in values.yaml
extraSecrets:
'{{ .Release.Name }}-airflow-connections':
data: |
AIRFLOW_CONN_SPARK_DEFAULT: 'c3Bhcms6Ly9rOHM6Ly8xMDAuNjguMC4xOjQ0Mz9kZXBsb3ktbW9kZT1jbHVzdGVy'
extraEnvFrom: |
- secretRef:
name: '{{ .Release.Name }}-airflow-connections'
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21638: Spark Connection with k8s in URL not mapped correctly
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21638:
URL: https://github.com/apache/airflow/issues/21638#issuecomment-1042976133
Would you lke to make a PR for that @Scope0910 - seems that it should not be difficult and you have an environment that you could test it in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elyselam commented on issue #21638: Spark Connection with k8s in URL not mapped correctly
Posted by GitBox <gi...@apache.org>.
elyselam commented on issue #21638:
URL: https://github.com/apache/airflow/issues/21638#issuecomment-1046542271
Can I get this assigned to me please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21638: Spark Connection with k8s in URL not mapped correctly
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21638:
URL: https://github.com/apache/airflow/issues/21638#issuecomment-1043459849
Yeah. Doesn't really matter who's bug it is - Maybe we can simply workaround it in airflow ? Maybe we can fix some world around us rather than garbage-in -> garbage-out :)
I do not say we have to - but if possible - why not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #21638: Spark Connection with k8s in URL not mapped correctly
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #21638:
URL: https://github.com/apache/airflow/issues/21638#issuecomment-1043274219
@potiuk This is a bug in the URL parser in the Python built-in library
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Scope0910 commented on issue #21638: Spark Connection with k8s in URL not mapped correctly
Posted by GitBox <gi...@apache.org>.
Scope0910 commented on issue #21638:
URL: https://github.com/apache/airflow/issues/21638#issuecomment-1043346570
i agree with @mik-laj it isn't quite a bug of the helm chart himself. urllib parser in python is quite pain. Maybe it should handled in models/connection.py ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #21638: Spark Connection with k8s in URL not mapped correctly
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21638:
URL: https://github.com/apache/airflow/issues/21638#issuecomment-1042752322
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] elyselam removed a comment on issue #21638: Spark Connection with k8s in URL not mapped correctly
Posted by GitBox <gi...@apache.org>.
elyselam removed a comment on issue #21638:
URL: https://github.com/apache/airflow/issues/21638#issuecomment-1046542271
Can I get this assigned to me please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org