You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/23 09:49:50 UTC
[GitHub] [airflow] andormarkus opened a new issue #16010: Helm chart: How to enable HTTPS connection
andormarkus opened a new issue #16010:
URL: https://github.com/apache/airflow/issues/16010
**Apache Airflow version**: 2.0.2
**Helm chart version**: 1.0.0
**Kubernetes version**: 1.20
**What happened**:
I have tried to enable HTTPS connection, however when I enable it liveness and readiness probes are failing
```bash
▶ kubectl -n airflow describe pod/airflow-webserver-564fcf9c5c-jqtt4
Name: airflow-webserver-564fcf9c5c-jqtt4
Namespace: airflow
Priority: 0
Node: ip-10-10-13-31.eu-central-1.compute.internal/10.10.13.31
Start Time: Sun, 23 May 2021 11:37:08 +0200
Labels: component=webserver
pod-template-hash=564fcf9c5c
release=airflow
tier=airflow
Annotations: checksum/airflow-config: 4e5f26f48f157218dcaf8eddac95e912aa7a47a4eb75b04697bb3701fd550340
checksum/extra-configmaps: 2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
checksum/extra-secrets: bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
checksum/metadata-secret: 765446a9def21895960ebc0df295399c35d3467cdd84be6ef8bc65c18ca0f7e5
checksum/pgbouncer-config-secret: da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
checksum/webserver-config: 4a2281a4e3ed0cc5e89f07aba3c1bb314ea51c17cb5d2b41e9b045054a6b5c72
kubernetes.io/psp: eks.privileged
Status: Running
IP: 10.10.13.56
IPs:
IP: 10.10.13.56
Controlled By: ReplicaSet/airflow-webserver-564fcf9c5c
Init Containers:
wait-for-airflow-migrations:
Container ID: docker://da6e1a36ac496982bc6d9058e1d94791cf99bf7d70dbbe10be2f2bbe0446eae8
Image: apache/airflow:2.0.2
Image ID: docker-pullable://apache/airflow@sha256:1c3dbd1c3e964e98fffcb58efb77b41179dc1322a9d3919e6f4289f2d2d84625
Port: <none>
Host Port: <none>
Args:
python
-c
import airflow
import logging
import os
import time
from alembic.config import Config
from alembic.runtime.migration import MigrationContext
from alembic.script import ScriptDirectory
from airflow import settings
package_dir = os.path.abspath(os.path.dirname(airflow.__file__))
directory = os.path.join(package_dir, 'migrations')
config = Config(os.path.join(package_dir, 'alembic.ini'))
config.set_main_option('script_location', directory)
config.set_main_option('sqlalchemy.url', settings.SQL_ALCHEMY_CONN.replace('%', '%%'))
script_ = ScriptDirectory.from_config(config)
timeout=60
with settings.engine.connect() as connection:
context = MigrationContext.configure(connection)
ticker = 0
while True:
source_heads = set(script_.get_heads())
db_heads = set(context.get_current_heads())
if source_heads == db_heads:
break
if ticker >= timeout:
raise TimeoutError("There are still unapplied migrations after {} seconds.".format(ticker))
ticker += 1
time.sleep(1)
logging.info('Waiting for migrations... %s second(s)', ticker)
State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 23 May 2021 11:37:11 +0200
Finished: Sun, 23 May 2021 11:37:24 +0200
Ready: True
Restart Count: 0
Environment:
AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in secret 'airflow-fernet-key'> Optional: false
AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in secret 'airflow-postgres-password'> Optional: false
AIRFLOW_CONN_AIRFLOW_DB: <set to the key 'connection' in secret 'airflow-postgres-password'> Optional: false
AWS_DEFAULT_REGION: eu-central-1
AWS_REGION: eu-central-1
AWS_ROLE_ARN: arn:aws:iam::589348632636:role/dataeng-iamrole-eksairflow-dev
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from airflow-token-9cqzb (ro)
Containers:
webserver:
Container ID: docker://2b8ee8a99ebd088b545685b3d1bc290adb0a7675e4935322a9549a152339a249
Image: apache/airflow:2.0.2
Image ID: docker-pullable://apache/airflow@sha256:1c3dbd1c3e964e98fffcb58efb77b41179dc1322a9d3919e6f4289f2d2d84625
Port: 8080/TCP
Host Port: 0/TCP
Args:
bash
-c
exec airflow webserver
State: Running
Started: Sun, 23 May 2021 11:39:48 +0200
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Sun, 23 May 2021 11:37:26 +0200
Finished: Sun, 23 May 2021 11:39:48 +0200
Ready: False
Restart Count: 1
Liveness: http-get http://:8080/airflow/health delay=15s timeout=30s period=5s #success=1 #failure=20
Readiness: http-get http://:8080/airflow/health delay=15s timeout=30s period=5s #success=1 #failure=20
Environment:
AIRFLOW__CORE__FERNET_KEY: <set to the key 'fernet-key' in secret 'airflow-fernet-key'> Optional: false
AIRFLOW__CORE__SQL_ALCHEMY_CONN: <set to the key 'connection' in secret 'airflow-postgres-password'> Optional: false
AIRFLOW_CONN_AIRFLOW_DB: <set to the key 'connection' in secret 'airflow-postgres-password'> Optional: false
AWS_DEFAULT_REGION: eu-central-1
AWS_REGION: eu-central-1
AWS_ROLE_ARN: arn:aws:iam::589348632636:role/dataeng-iamrole-eksairflow-dev
AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
Mounts:
/opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
/opt/airflow/dags from dags (rw)
/opt/airflow/logs from logs (rw)
/opt/airflow/secrets/ssl from airflow-ssl (ro)
/var/run/secrets/eks.amazonaws.com/serviceaccount from aws-iam-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from airflow-token-9cqzb (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
aws-iam-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 86400
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: airflow-airflow-config
Optional: false
dags:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: airflow-dags
ReadOnly: false
logs:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: airflow-logs
ReadOnly: false
airflow-ssl:
Type: Secret (a volume populated by a Secret)
SecretName: airflow-ssl
Optional: false
airflow-token-9cqzb:
Type: Secret (a volume populated by a Secret)
SecretName: airflow-token-9cqzb
Optional: false
QoS Class: BestEffort
Node-Selectors: namespace=airflow
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m14s default-scheduler Successfully assigned airflow/airflow-webserver-564fcf9c5c-jqtt4 to ip-10-10-13-31.eu-central-1.compute.internal
Normal Pulled 4m11s kubelet Container image "apache/airflow:2.0.2" already present on machine
Normal Created 4m11s kubelet Created container wait-for-airflow-migrations
Normal Started 4m11s kubelet Started container wait-for-airflow-migrations
Normal Pulled 3m57s kubelet Container image "apache/airflow:2.0.2" already present on machine
Normal Created 3m57s kubelet Created container webserver
Normal Started 3m56s kubelet Started container webserver
Warning Unhealthy 3m37s kubelet Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": dial tcp 10.10.13.56:8080: connect: connection refused
Warning Unhealthy 3m36s (x2 over 3m41s) kubelet Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": dial tcp 10.10.13.56:8080: connect: connection refused
Warning Unhealthy 3m2s kubelet Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 3m1s kubelet Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:37846->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 3m1s kubelet Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38016->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 3m1s kubelet Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38046->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 3m1s kubelet Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38048->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 2m57s kubelet Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38056->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 2m56s kubelet Readiness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38060->10.10.13.56:8080: read: connection reset by peer
Warning Unhealthy 2m32s (x9 over 2m52s) kubelet (combined from similar events): Liveness probe failed: Get "http://10.10.13.56:8080/airflow/health": read tcp 10.10.13.31:38192->10.10.13.56:8080: read: connection reset by peer
```
**How to reproduce it**:
Helm chart myconf.yaml looks like this
```yaml
executor: KubernetesExecutor
fernetKey: "XXXXXXXXXX"
webserver:
base_url: "https://k8s.foobaa.com/airflow"
web_server_ssl_cert: "/opt/airflow/secrets/ssl/tls.crt"
web_server_ssl_key: "/opt/airflow/secrets/ssl/tls.key"
webserver:
service:
type: NodePort
extraVolumeMounts:
- name: airflow-ssl
mountPath: /opt/airflow/secrets/ssl
readOnly: true
extraVolumes:
- name: airflow-ssl
secret:
secretName: airflow-ssl
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-846557324
Hi @potiuk and @mik-laj
Thank you for the idea to shifting the SSL into the Ingress. We are using AWS Load Balancer Controller which supports ingress side certificates. I think more people might have the same issue who are new into Kubernetes and Airflow. Can we extended [Helm chart documentation](http://airflow.apache.org/docs/helm-chart/stable/index.html) with mik-laj's recommendation?
I have followed Marc Lamberti's 'Apache Airflow on AWS EKS: The Hands-On Guide' which clearly won't work with the newly released official chart.
Helm chart myconf.yaml looks like this
```yaml
ingress:
enabled: true
web:
precedingPaths:
- path: "/*"
serviceName: "ssl-redirect"
servicePort: "use-annotation"
path: "/airflow/*"
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:eu-central-1:XXXXXXXXXX:certificate/XXXXXXXXXXX
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847240302
Thanks @andormarkus We will take a look at those issues you have created
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847041776
> I'm more than happy to submit a pr, however I'm busy with migration to the official helm chart. Airflow version `2.0.2+` are causing so much headache. See #16013 #16020
I see. Did you (by any chance) try 2.1 or manually upgrading `cncf-kubernetes` provider to 1.2.0? t
I think there are couple of issues fixed in `cncf.kubernetes` 1.2.0 version of the provider and at least one looks like might be fixing the labelling problem you explained (https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/index.html#id1) - you can see detailed list of commits for 1.2.0 `cncf.kubernetes` provider here: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/commits.html#id1 and the one I am thinking about is https://github.com/apache/airflow/commit/be421a6b07c2ae9167150b77dc1185a94812b358
The easiest (if you want to try 1.0.2 with new provider) is to build your own image and extend it by installing the new provider with something like:
```
FROM apache/airflow:2.0.2
pip install --upgrade apache-airflow-providers-cncf-kubernetes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-1004944901
Hi
We are using this config with helm 1.3.0
```yaml
config:
webserver:
base_url: "http://dataengineering.acme.com/airflow"
ingress:
enabled: true
web:
precedingPaths:
- path: "/*"
pathType: "ImplementationSpecific"
serviceName: "ssl-redirect"
servicePort: "use-annotation"
path: "/airflow/*"
annotations:
external-dns.alpha.kubernetes.io/hostname: ${web_url}
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/group.name: dataengineering
alb.ingress.kubernetes.io/group.order: '10'
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
alb.ingress.kubernetes.io/certificate-arn: ${aws_acm_certificate_arn}
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
```
I think your `config.webserver.base_url` is incorrect, it needs to be a valid private/public URL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-846554952
You just don't need to configure Nginx, but you can configure SSL in Ingress. This will also allow certmanager to automatically manage the certificates.
Example:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# add an annotation indicating the issuer to use.
cert-manager.io/cluster-issuer: nameOfClusterIssuer
name: myIngress
namespace: myIngress
spec:
rules:
- host: example.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: myservice
port:
number: 80
tls: # < placing a host in the TLS config will indicate a certificate should be created
- hosts:
- example.com
secretName: myingress-cert # < cert-manager will store the created certificate in this secret.
```
See: https://cert-manager.io/docs/usage/ingress/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847041776
> I'm more than happy to submit a pr, however I'm busy with migration to the official helm chart. Airflow version `2.0.2+` are causing so much headache. See #16013 #16020
I see. Did you (by any chance) try 2.1.0 or manually upgrading `cncf-kubernetes` provider to 1.2.0?
I think there are couple of issues fixed in `cncf.kubernetes` 1.2.0 version of the provider and at least one looks like might be fixing the labelling problem you explained (https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/index.html#id1) - you can see detailed list of commits for 1.2.0 `cncf.kubernetes` provider here: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/commits.html#id1 and the one I am thinking about is https://github.com/apache/airflow/commit/be421a6b07c2ae9167150b77dc1185a94812b358
The easiest (if you want to try 1.0.2 with new provider) is to build your own image and extend it by installing the new provider with something like:
```
FROM apache/airflow:2.0.2-pythonX.Y
RUN pip install --upgrade apache-airflow-providers-cncf-kubernetes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-846554952
You just don't need to configure Nginx, but you can configure SSL in Ingress. This will also allow certmanager to automatically manage the certificates.
Example:
```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# add an annotation indicating the issuer to use.
cert-manager.io/cluster-issuer: nameOfClusterIssuer
name: myIngress
namespace: myIngress
spec:
rules:
- host: example.com
http:
paths:
- pathType: Prefix
path: /
backend:
service:
name: myservice
port:
number: 80
tls: # < placing a host in the TLS config will indicate a certificate should be created
- hosts:
- example.com
secretName: myingress-cert # < cert-manager will store the created certificate in this secret.
```
See: https://cert-manager.io/docs/usage/ingress/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
kaxil closed issue #16010:
URL: https://github.com/apache/airflow/issues/16010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rdeteix commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
rdeteix commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-1007210968
The alb controller has changed since v2.3 : https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/guide/tasks/ssl_redirect/
I'll try to make it work/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-846559874
> Can we extended Helm chart documentation with mik-laj's recommendation?
Good idea. Are you willing to submit a PR? I'm happy to help. with review.
> I have followed Marc Lamberti's 'Apache Airflow on AWS EKS: The Hands-On Guide' which clearly won't work with the newly released official chart.
CC: @marclamberti
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rdeteix commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
rdeteix commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-1004754174
Hello
Is it still working ?
I'am on k8s v1.21, helm 1.3.0, airflow 2.2.3.
After getting an error about a missing pathType I tried :
```
values:
config:
webserver:
base_url: "http://localhost:8080/dev"
ingress:
enabled: true
web:
precedingPaths:
- path: "/*"
pathType: "ImplementationSpecific"
serviceName: "ssl-redirect"
servicePort: "use-annotation"
path: /dev/*
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
alb.ingress.kubernetes.io/certificate-arn: mycertificate
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
```
I let the service to be a cluster ip.
My ingress give me:
```
Name: airflow-dev-airflow-ingress
Namespace: dev
Address: my-adress
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
*
/* ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
/dev/* airflow-dev-webserver:airflow-ui ()
Annotations: alb.ingress.kubernetes.io/actions.ssl-redirect:
{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
alb.ingress.kubernetes.io/certificate-arn: mycertificate
alb.ingress.kubernetes.io/listen-ports: [{"HTTP": 80}, {"HTTPS":443}]
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb
meta.helm.sh/release-name: airflow-dev
meta.helm.sh/release-namespace: dev
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfullyReconciled 112s ingress Successfully reconciled
```
The ingress manifest that it created is:
```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
{ "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/certificate-arn: *****
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb
meta.helm.sh/release-name: airflow-dev
meta.helm.sh/release-namespace: dev
spec:
rules:
- http:
paths:
- backend:
service:
name: ssl-redirect
port:
name: use-annotation
path: /*
pathType: ImplementationSpecific
- backend:
service:
name: airflow-dev-webserver
port:
name: airflow-ui
path: /dev/*
pathType: ImplementationSpecific
status:
loadBalancer:
ingress:
- hostname: *****
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847212751
So we need to take a closer look then! Thanks for verifying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Anirudhaagrawal edited a comment on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
Anirudhaagrawal edited a comment on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847816045
My issue is somewhat related. Airflow UI keeps redirecting me to HTTP instead of https.
I was following this.
https://airflow.apache.org/docs/apache-airflow/stable/howto/run-behind-proxy.html
I tried adding something like this
```
webserver:
base_url: "https://k8s.foobaa.com/airflow"
```
But it gave me
```
airflow:
- webserver: Additional property base_url is not allowed
```
I am using aws elb with Kong for proxy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Anirudhaagrawal commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
Anirudhaagrawal commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847816045
My issue is somewhat related. Airflow UI keeps redirecting me to HTTP instead of https.
I was following this.
https://airflow.apache.org/docs/apache-airflow/stable/howto/run-behind-proxy.html
I tried adding something like this
```
webserver:
base_url: "https://k8s.foobaa.com/airflow"
```
But it gave me
```
airflow:
- webserver: Additional property base_url is not allowed
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-846542829
I believe all the deployments in K8S should be done using http (or if istio is enabled they can be additionally protected via mTLS).
I think the usual pattern of providing SSL deployment is via external proxy/gateways, especially that they can then provide additional authentication mechanism (Oauth/client certificates etc.) which can be configured outside of Airflow and even provide single authentication/SSL endpoints for multiple services.
I think this is common best practice to decouple SSL termination and the service itself.
Here is a example of simple nginx-based SSL proxy that you can follow: https://kubernetes.io/blog/2015/07/strong-simple-ssl-for-kubernetes/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marclamberti commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
marclamberti commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-846574118
Thanks for noticing me :)
I will update the course
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847929472
@Anirudhaagrawal During my initial comment I was'n fully familiar with the helm chart values file. potiuk and mik-laj recommended: we should not use HTTPS inside kubernetes, it will fail on the liveness and readiness probes.
Recommended traffic flow looks like this:
internet --> HTTPS --> K8s ingress controller --> HTTP --> Airflow webserver
my configuration looks like this (AWS EKS with `aws-load-balancer-controller`):
```yaml
config:
webserver:
base_url: https://k8s.foobaa.com/airflow
ingress:
enabled: true
web:
precedingPaths:
- path: "/*"
serviceName: "ssl-redirect"
servicePort: "use-annotation"
path: "/airflow/*"
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
alb.ingress.kubernetes.io/certificate-arn: ${aws_acm_certificate_arn}
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rdeteix edited a comment on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
rdeteix edited a comment on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-1004754174
Hello
Is it still working ?
I'am on k8s v1.21, helm 1.3.0, airflow 2.2.3.
After getting an error about a missing pathType I tried :
```
values:
config:
webserver:
base_url: "http://localhost:8080/dev"
ingress:
enabled: true
web:
precedingPaths:
- path: "/*"
pathType: "ImplementationSpecific"
serviceName: "ssl-redirect"
servicePort: "use-annotation"
path: /dev/*
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
alb.ingress.kubernetes.io/certificate-arn: mycertificate
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
```
I let the service to be a cluster ip.
My ingress gives me:
```
Name: airflow-dev-airflow-ingress
Namespace: dev
Address: my-adress
Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
Host Path Backends
---- ---- --------
*
/* ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
/dev/* airflow-dev-webserver:airflow-ui ()
Annotations: alb.ingress.kubernetes.io/actions.ssl-redirect:
{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
alb.ingress.kubernetes.io/certificate-arn: mycertificate
alb.ingress.kubernetes.io/listen-ports: [{"HTTP": 80}, {"HTTPS":443}]
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb
meta.helm.sh/release-name: airflow-dev
meta.helm.sh/release-namespace: dev
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfullyReconciled 112s ingress Successfully reconciled
```
The ingress manifest that it created is:
```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
{ "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/certificate-arn: *****
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb
meta.helm.sh/release-name: airflow-dev
meta.helm.sh/release-namespace: dev
spec:
rules:
- http:
paths:
- backend:
service:
name: ssl-redirect
port:
name: use-annotation
path: /*
pathType: ImplementationSpecific
- backend:
service:
name: airflow-dev-webserver
port:
name: airflow-ui
path: /dev/*
pathType: ImplementationSpecific
status:
loadBalancer:
ingress:
- hostname: *****
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rdeteix commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
rdeteix commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-1005086285
Thx for the response.
I've already tried it.
Which k8s version are you working with ?
I also tried it without the reverse proxy :
```
ingress:
enabled: true
web:
precedingPaths:
- path: "/*"
pathType: "ImplementationSpecific"
serviceName: "ssl-redirect"
servicePort: "use-annotation"
path: "/*"
annotations:
external-dns.alpha.kubernetes.io/hostname: ****
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/group.name: airflow
alb.ingress.kubernetes.io/group.order: '10'
alb.ingress.kubernetes.io/scheme: internal
alb.ingress.kubernetes.io/target-type: ip
alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,stickiness.lb_cookie.duration_seconds=3600
alb.ingress.kubernetes.io/certificate-arn: ****
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
```
I have:
```
ts=2022-01-04T18:52:13.592931819Z caller=release.go:397 component=release release=airflow-dev targetNamespace=dev resource=dev:helmrelease/airflow helmVersion=v3 warning="failed to annotate release resources: error: arguments in resource/name form must have a single resource and name\n" phase=annotate
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847041776
> I'm more than happy to submit a pr, however I'm busy with migration to the official helm chart. Airflow version `2.0.2+` are causing so much headache. See #16013 #16020
I see. Did you (by any chance) try 2.1.0 or manually upgrading `cncf-kubernetes` provider to 1.2.0?
I think there are couple of issues fixed in `cncf.kubernetes` 1.2.0 version of the provider and at least one looks like might be fixing the labelling problem you explained (https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/index.html#id1) - you can see detailed list of commits for 1.2.0 `cncf.kubernetes` provider here: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/commits.html#id1 and the one I am thinking about is https://github.com/apache/airflow/commit/be421a6b07c2ae9167150b77dc1185a94812b358
The easiest (if you want to try 1.0.2 with new provider) is to build your own image and extend it by installing the new provider with something like:
```
FROM apache/airflow:2.0.2
pip install --upgrade apache-airflow-providers-cncf-kubernetes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847981937
I am going to close this ticket for now with https://github.com/apache/airflow/issues/16010#issuecomment-847929472 as the correct solution. For other issues #16013 #16020 -- We will take a look and comment on those specific tickets
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rdeteix commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
rdeteix commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-1007210968
The alb controller has changed since v2.3 : https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.3/guide/tasks/ssl_redirect/
I'll try to make it work/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847196112
Hi @potiuk, I have tried the official helm chart 1.0.0 with Airflow 2.0.1, 2.0.2, 2.1.0 as well. Only 2.0.1 is working, 2.0.2 and 2.1.0 does not work mentioned in the related issues. As I checked Airflow 2.1.0 comes with `apache-airflow-providers-cncf-kubernetes==1.2.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847041776
> I'm more than happy to submit a pr, however I'm busy with migration to the official helm chart. Airflow version `2.0.2+` are causing so much headache. See #16013 #16020
I see. Did you (by any chance) try 2.1 or manually upgrading `cncf-kubernetes` provider to 1.2.0?
I think there are couple of issues fixed in `cncf.kubernetes` 1.2.0 version of the provider and at least one looks like might be fixing the labelling problem you explained (https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/index.html#id1) - you can see detailed list of commits for 1.2.0 `cncf.kubernetes` provider here: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/commits.html#id1 and the one I am thinking about is https://github.com/apache/airflow/commit/be421a6b07c2ae9167150b77dc1185a94812b358
The easiest (if you want to try 1.0.2 with new provider) is to build your own image and extend it by installing the new provider with something like:
```
FROM apache/airflow:2.0.2
pip install --upgrade apache-airflow-providers-cncf-kubernetes
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-846600296
Yep. It would be great to submit a pr @andormarkus . And yeah. Ingress is even better than having your own proxy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] andormarkus commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
andormarkus commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-847035222
I'm more than happy to submit a pr, however I'm busy with migration to the official helm chart. Airflow version `2.0.2+` are causing so much headache. See #16013 #16020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #16010: Helm chart: How to enable HTTPS connection
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #16010:
URL: https://github.com/apache/airflow/issues/16010#issuecomment-846542829
I believe all the deployments in K8S should be done using http (or if istio is enabled they can be additionally protected via mTLS. I think the usual pattern of providing SSL deployment is via external proxy/gateways, especially that they can then provide additional authentication mechanism (Oauth/client certificates etc.) which can be configured outside of Airflow and even provide single authentication/SSL endpoints for multiple services.
I think this is common best practice to decouple SSL termination and the service itself.
Here is a example of simple nginx-based SSL proxy that you can follow: https://kubernetes.io/blog/2015/07/strong-simple-ssl-for-kubernetes/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org