You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/13 06:13:45 UTC
[GitHub] [airflow] m3t41 opened a new issue, #26359: HELM CHART over ARGOCD on OpenShift -> Scheudular disconnections
m3t41 opened a new issue, #26359:
URL: https://github.com/apache/airflow/issues/26359
### Apache Airflow version
2.3.4
### What happened
I am an SRE, and i have a Problem to get the Helm Chart up und Running on an OpenShift Cluster.
changes i made to the chart
Executor = KubernetesExecutor
SecurityContext disable in all Stages
Set Limits in all Stages
Redis = disable
that's it
and then i get these errors
[34m2022-09-13 05:52:43,424[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efc70>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True[0m
[[34m2022-09-13 05:52:43,424[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0a60>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue[0m
[[34m2022-09-13 05:54:54,496[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0f40>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue[0m
[[34m2022-09-13 05:54:54,496[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efe80>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True[0m
[[34m2022-09-13 05:57:05,568[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c47808040>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True[0m
[[34m2022-09-13 05:57:05,568[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0df0>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue[0m
### What you think should happen instead
`[38;5;6mairflow-scheduler [38;5;5m05:50:17.69 [0m
[38;5;6mairflow-scheduler [38;5;5m05:50:17.69 [0m[1mWelcome to the Bitnami airflow-scheduler container[0m
[38;5;6mairflow-scheduler [38;5;5m05:50:17.69 [0mSubscribe to project updates by watching [1mhttps://github.com/bitnami/containers[0m
[38;5;6mairflow-scheduler [38;5;5m05:50:17.70 [0mSubmit issues and feature requests at [1mhttps://github.com/bitnami/containers/issues[0m
[38;5;6mairflow-scheduler [38;5;5m05:50:17.70 [0m
[38;5;6mairflow-scheduler [38;5;5m05:50:17.75 [0m[38;5;2mINFO [0m ==> Enabling non-root system user with nss_wrapper
[38;5;6mairflow-scheduler [38;5;5m05:50:17.76 [0m[38;5;2mINFO [0m ==> ** Starting Airflow setup **
[38;5;6mairflow-scheduler [38;5;5m05:50:17.87 [0m[38;5;2mINFO [0m ==> No injected configuration file found. Creating default config file
[38;5;6mairflow-scheduler [38;5;5m05:50:19.65 [0m[38;5;2mINFO [0m ==> Configuring Airflow database
[38;5;6mairflow-scheduler [38;5;5m05:50:19.67 [0m[38;5;2mINFO [0m ==> Trying to connect to the database server
[38;5;6mairflow-scheduler [38;5;5m05:50:23.37 [0m[38;5;2mINFO [0m ==> Waiting for Airflow Webserser to be up
[38;5;6mairflow-scheduler [38;5;5m05:50:23.45 [0m[38;5;2mINFO [0m ==> ** Airflow setup finished! **
[38;5;6mairflow-scheduler [38;5;5m05:50:23.47 [0m[38;5;2mINFO [0m ==> ** Starting Airflow **
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[[34m2022-09-13 05:50:31,573[0m] {[34mscheduler_job.py:[0m708} INFO[0m - Starting the scheduler[0m
[[34m2022-09-13 05:50:31,573[0m] {[34mscheduler_job.py:[0m713} INFO[0m - Processing each file at most -1 times[0m
[[34m2022-09-13 05:50:31,578[0m] {[34mexecutor_loader.py:[0m105} INFO[0m - Loaded executor: KubernetesExecutor[0m
[[34m2022-09-13 05:50:31,675[0m] {[34mkubernetes_executor.py:[0m520} INFO[0m - Start Kubernetes executor[0m
[[34m2022-09-13 05:50:31,856[0m] {[34mkubernetes_executor.py:[0m128} INFO[0m - Event: and now my watch begins starting at resource_version: 0[0m
[[34m2022-09-13 05:50:31,875[0m] {[34mkubernetes_executor.py:[0m469} INFO[0m - Found 0 queued task instances[0m
[[34m2022-09-13 05:50:31,957[0m] {[34mmanager.py:[0m160} INFO[0m - Launched DagFileProcessorManager with pid: 124[0m
[[34m2022-09-13 05:50:31,959[0m] {[34mscheduler_job.py:[0m1233} INFO[0m - Resetting orphaned tasks for active dag runs[0m
[[34m2022-09-13 05:50:31,965[0m] {[34msettings.py:[0m55} INFO[0m - Configured default timezone Timezone('UTC')[0m
[[34m2022-09-13 05:50:31,969[0m] {[34mscheduler_job.py:[0m1256} INFO[0m - Marked 1 SchedulerJob instances as failed[0m
[[34m2022-09-13 05:52:43,424[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efc70>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True[0m
[[34m2022-09-13 05:52:43,424[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0a60>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue[0m
[[34m2022-09-13 05:54:54,496[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0f40>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue[0m
[[34m2022-09-13 05:54:54,496[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c477efe80>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True[0m
[[34m2022-09-13 05:57:05,568[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c47808040>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True[0m
[[34m2022-09-13 05:57:05,568[0m] {[34mconnectionpool.py:[0m810} WARNING[0m - Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839f0df0>: Failed to establish a new connection: [Errno 110] Connection timed out')': /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue[0m
[[34m2022-09-13 05:59:16,640[0m] {[34mkubernetes_executor.py:[0m111} ERROR[0m - Unknown error in KubernetesJobWatcher. Failing[0m
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9c47808190>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 102, in run
self.resource_version = self._run(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 147, in _run
for event in list_worker_pods():
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/watch/watch.py", line 163, in stream
resp = func(*args, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 213, in request
r = self.pool_manager.request(method, url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
return self.request_encode_url(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.26.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c47808190>: Failed to establish a new connection: [Errno 110] Connection timed out'))
[[34m2022-09-13 05:59:16,642[0m] {[34mscheduler_job.py:[0m768} ERROR[0m - Exception when executing SchedulerJob._run_scheduler_loop[0m
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9c839d7430>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 751, in _execute
self._run_scheduler_loop()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 806, in _run_scheduler_loop
self.adopt_or_reset_orphaned_tasks()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1236, in adopt_or_reset_orphaned_tasks
for attempt in run_with_db_retries(logger=self.log):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/tenacity/__init__.py", line 382, in __iter__
do = self.iter(retry_state=retry_state)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/tenacity/__init__.py", line 349, in iter
return fut.result()
File "/opt/bitnami/python/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/opt/bitnami/python/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1281, in adopt_or_reset_orphaned_tasks
to_reset = self.executor.try_adopt_task_instances(tis_to_reset_or_adopt)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 713, in try_adopt_task_instances
self._adopt_completed_pods(kube_client)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 760, in _adopt_completed_pods
pod_list = kube_client.list_namespaced_pod(namespace=self.kube_config.kube_namespace, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 213, in request
r = self.pool_manager.request(method, url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
return self.request_encode_url(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.26.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839d7430>: Failed to establish a new connection: [Errno 110] Connection timed out'))
[[34m2022-09-13 05:59:16,650[0m] {[34mkubernetes_executor.py:[0m821} INFO[0m - Shutting down Kubernetes executor[0m
Process KubernetesJobWatcher-3:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9c47808190>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/bitnami/python/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 102, in run
self.resource_version = self._run(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 147, in _run
for event in list_worker_pods():
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/watch/watch.py", line 163, in stream
resp = func(*args, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 213, in request
r = self.pool_manager.request(method, url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
return self.request_encode_url(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.26.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/xxx/pods?labelSelector=airflow-worker%3D373&resourceVersion=0&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c47808190>: Failed to establish a new connection: [Errno 110] Connection timed out'))
[[34m2022-09-13 05:59:17,755[0m] {[34mprocess_utils.py:[0m125} INFO[0m - Sending Signals.SIGTERM to group 124. PIDs of all processes in the group: [124][0m
[[34m2022-09-13 05:59:17,755[0m] {[34mprocess_utils.py:[0m80} INFO[0m - Sending the signal Signals.SIGTERM to group 124[0m
[[34m2022-09-13 05:59:18,393[0m] {[34mprocess_utils.py:[0m75} INFO[0m - Process psutil.Process(pid=124, status='terminated', exitcode=0, started='05:50:31') (124) terminated with exit code 0[0m
[[34m2022-09-13 05:59:18,393[0m] {[34mscheduler_job.py:[0m780} INFO[0m - Exited execute loop[0m
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
raise err
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 358, in connect
self.sock = conn = self._new_conn()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f9c839d7430>: Failed to establish a new connection: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/bitnami/airflow/venv/bin/airflow", line 8, in <module>
sys.exit(main())
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/__main__.py", line 38, in main
args.func(args)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/cli/commands/scheduler_command.py", line 75, in scheduler
_run_scheduler_job(args=args)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/cli/commands/scheduler_command.py", line 46, in _run_scheduler_job
job.run()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 244, in run
self._execute()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 751, in _execute
self._run_scheduler_loop()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 806, in _run_scheduler_loop
self.adopt_or_reset_orphaned_tasks()
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1236, in adopt_or_reset_orphaned_tasks
for attempt in run_with_db_retries(logger=self.log):
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/tenacity/__init__.py", line 382, in __iter__
do = self.iter(retry_state=retry_state)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/tenacity/__init__.py", line 349, in iter
return fut.result()
File "/opt/bitnami/python/lib/python3.8/concurrent/futures/_base.py", line 437, in result
return self.__get_result()
File "/opt/bitnami/python/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/jobs/scheduler_job.py", line 1281, in adopt_or_reset_orphaned_tasks
to_reset = self.executor.try_adopt_task_instances(tis_to_reset_or_adopt)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 713, in try_adopt_task_instances
self._adopt_completed_pods(kube_client)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 760, in _adopt_completed_pods
pod_list = kube_client.list_namespaced_pod(namespace=self.kube_config.kube_namespace, **kwargs)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15697, in list_namespaced_pod
return self.list_namespaced_pod_with_http_info(namespace, **kwargs) # noqa: E501
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 15812, in list_namespaced_pod_with_http_info
return self.api_client.call_api(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api
return self.__call_api(resource_path, method,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
response_data = self.request(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 373, in request
return self.rest_client.GET(url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 240, in GET
return self.request("GET", url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/kubernetes/client/rest.py", line 213, in request
r = self.pool_manager.request(method, url,
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
return self.request_encode_url(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/opt/bitnami/airflow/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='172.26.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/xxx/pods?fieldSelector=status.phase%3DSucceeded&labelSelector=kubernetes_executor%3DTrue (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9c839d7430>: Failed to establish a new connection: [Errno 110] Connection timed out'))`
### How to reproduce
i am Using ArgoCD to deploy the chart to OpenShift.
### Operating System
OpenShift
### Versions of Apache Airflow Providers
_No response_
### Deployment
Other 3rd-party Helm chart
### Deployment details
bitnami helm chart 13.1.3
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #26359: HELM CHART over ARGOCD on OpenShift -> Scheudular disconnections
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26359:
URL: https://github.com/apache/airflow/issues/26359#issuecomment-1245114336
I think you should either raise it to bitnami (this community has nothing to do with it - they should support it), or switch to the official helm chart which is managed by the community https://airflow.apache.org/docs/helm-chart/stable/index.html
Converting it to a discussion in case more information is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #26359: HELM CHART over ARGOCD on OpenShift -> Scheudular disconnections
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26359:
URL: https://github.com/apache/airflow/issues/26359#issuecomment-1244946954
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #26359: HELM CHART over ARGOCD on OpenShift -> Scheudular disconnections
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #26359: HELM CHART over ARGOCD on OpenShift -> Scheudular disconnections
URL: https://github.com/apache/airflow/issues/26359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org