You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/07/03 15:07:03 UTC

[GitHub] [airflow] bhavaniravi opened a new issue #9643: SSL connection has been closed unexpectedly

bhavaniravi opened a new issue #9643:
URL: https://github.com/apache/airflow/issues/9643


   **Apache Airflow version**:
   1.10.11
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
   1.17.2
   
   **Environment**: 
   
   - **Cloud provider or hardware configuration**:  Self hosted kube cluster
   - **OS** (e.g. from /etc/os-release): Debian GNU/Linux
   - **Kernel** (e.g. `uname -a`): Linux
   - **Install tools**:
   - **Others**:
   
   **What happened**:
   
   Postgres SSL connection closed unexpectedly + Cannot write S3 logs
   
   Both the errors happen together or separately
   
   **What you expected to happen**:
   
   1. Operational Errors should be graciously handled
   2. S3 logs failure should retry on failure
   
   **How to reproduce it**:
   
   1. Create a task that connects to same postgres as airflow DB
   2. Let airflow update it's metastore, update the DB with queries from `PythonOperator`
   3. Write logs to S3 
   
   **Anything else we need to know**:
   
   3/5 - Pods has this error
   1/5 - Pods has this error + S3 failure 
   
   Any relevant logs to include? Put them here inside a detail tag:
   
   <details><summary>
   Log from worker pod
   </summary> 
   
   ```
   ERROR: airflow.jobs.local_task_job.LocalTaskJob LocalTaskJob heartbeat got an exception
   Traceback (most recent call last):
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2345, in _wrap_pool_connect
   return fn()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
   return _ConnectionFairy._checkout(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
   fairy = _ConnectionRecord.checkout(pool)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
   rec = pool._do_get()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
   return self._create_connection()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
   return _ConnectionRecord(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
   self.__connect(first_connect_check=True)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
   pool.logger.debug("Error on connect(): %s", e)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
   exc_value, with_traceback=exc_tb,
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
   raise exception
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
   connection = pool._invoke_creator(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
   return dialect.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
   return self.dbapi.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
   conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
   psycopg2.OperationalError: SSL connection has been closed unexpectedly
   The above exception was the direct cause of the following exception:
   Traceback (most recent call last):
   File "/usr/local/lib/python3.6/site-packages/airflow/jobs/base_job.py", line 199, in heartbeat
   self.heartbeat_callback(session=session)
   File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 70, in wrapper
   return func(*args, **kwargs)
   File "/usr/local/lib/python3.6/site-packages/airflow/jobs/local_task_job.py", line 142, in heartbeat_callback
   self.task_instance.refresh_from_db()
   File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
   return func(*args, **kwargs)
   File "/usr/local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 474, in refresh_from_db
   ti = qry.first()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3375, in first
   ret = list(self[0:1])
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3149, in __getitem__
   return list(res)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
   return self._execute_and_instances(context)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3503, in _execute_and_instances
   querycontext, self._connection_from_session, close_with_result=True
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3518, in _get_bind_args
   mapper=self._bind_mapper(), clause=querycontext.statement, **kw
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3496, in _connection_from_session
   conn = self.session.connection(**kw)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1141, in connection
   execution_options=execution_options,
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1147, in _connection_for_bind
   engine, execution_options
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
   conn = bind._contextual_connect()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2311, in _contextual_connect
   self._wrap_pool_connect(self.pool.connect, None),
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2349, in _wrap_pool_connect
   e, dialect, self
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1591, in _handle_dbapi_exception_noconnection
   sqlalchemy_exception, with_traceback=exc_info[2], from_=e
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
   raise exception
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2345, in _wrap_pool_connect
   return fn()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
   return _ConnectionFairy._checkout(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
   fairy = _ConnectionRecord.checkout(pool)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
   rec = pool._do_get()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
   return self._create_connection()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
   return _ConnectionRecord(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
   self.__connect(first_connect_check=True)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
   pool.logger.debug("Error on connect(): %s", e)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
   exc_value, with_traceback=exc_tb,
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
   raise exception
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
   connection = pool._invoke_creator(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
   return dialect.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
   return self.dbapi.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
   conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
   sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL connection has been closed unexpectedly
   (Background on this error at: http://sqlalche.me/e/e3q8)
   INFO: airflow.jobs.local_task_job.LocalTaskJob Task exited with return code 0
   [2020-07-03 14:41:55,153] {s3_task_handler.py:180} ERROR - Could not write logs to s3://sdqdevamrasp97540/sdq_airflowlogs_dev/dataload_dag/load_pfizer_study_0_B7931023/2020-07-03T14:37:11.188201+00:00/1.log
   Traceback (most recent call last):
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2345, in _wrap_pool_connect
   return fn()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
   return _ConnectionFairy._checkout(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
   fairy = _ConnectionRecord.checkout(pool)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
   rec = pool._do_get()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
   return self._create_connection()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
   return _ConnectionRecord(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
   self.__connect(first_connect_check=True)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
   pool.logger.debug("Error on connect(): %s", e)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
   exc_value, with_traceback=exc_tb,
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
   raise exception
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
   connection = pool._invoke_creator(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
   return dialect.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
   return self.dbapi.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
   conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
   psycopg2.OperationalError: SSL connection has been closed unexpectedly
   The above exception was the direct cause of the following exception:
   Traceback (most recent call last):
   File "/usr/local/lib/python3.6/site-packages/airflow/utils/log/s3_task_handler.py", line 177, in s3_write
   encrypt=conf.getboolean('core', 'ENCRYPT_S3_LOGS'),
   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line 409, in load_string
   self._upload_file_obj(file_obj, key, bucket_name, replace, encrypt, acl_policy)
   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line 490, in _upload_file_obj
   client = self.get_conn()
   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line 44, in get_conn
   return self.get_client_type('s3')
   File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py", line 176, in get_client_type
   session, endpoint_url = self._get_credentials(region_name)
   File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py", line 102, in _get_credentials
   connection_object = self.get_connection(self.aws_conn_id)
   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 84, in get_connection
   conn = random.choice(list(cls.get_connections(conn_id)))
   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 80, in get_connections
   return secrets.get_connections(conn_id)
   File "/usr/local/lib/python3.6/site-packages/airflow/secrets/__init__.py", line 52, in get_connections
   conn_list = secrets_backend.get_connections(conn_id=conn_id)
   File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
   return func(*args, **kwargs)
   File "/usr/local/lib/python3.6/site-packages/airflow/secrets/metastore.py", line 38, in get_connections
   conn_list = session.query(Connection).filter(Connection.conn_id == conn_id).all()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3319, in all
   return list(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
   return self._execute_and_instances(context)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3503, in _execute_and_instances
   querycontext, self._connection_from_session, close_with_result=True
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3518, in _get_bind_args
   mapper=self._bind_mapper(), clause=querycontext.statement, **kw
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3496, in _connection_from_session
   conn = self.session.connection(**kw)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1141, in connection
   execution_options=execution_options,
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1147, in _connection_for_bind
   engine, execution_options
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
   conn = bind._contextual_connect()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2311, in _contextual_connect
   self._wrap_pool_connect(self.pool.connect, None),
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2349, in _wrap_pool_connect
   e, dialect, self
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1591, in _handle_dbapi_exception_noconnection
   sqlalchemy_exception, with_traceback=exc_info[2], from_=e
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
   raise exception
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2345, in _wrap_pool_connect
   return fn()
   ERROR: airflow.utils.log.s3_task_handler.S3TaskHandler Could not write logs to <S3_PATH>
   Traceback (most recent call last):
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2345, in _wrap_pool_connect
   return fn()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
   return _ConnectionFairy._checkout(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
   fairy = _ConnectionRecord.checkout(pool)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
   rec = pool._do_get()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
   return self._create_connection()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
   return _ConnectionRecord(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
   self.__connect(first_connect_check=True)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
   pool.logger.debug("Error on connect(): %s", e)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
   exc_value, with_traceback=exc_tb,
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
   raise exception
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
   connection = pool._invoke_creator(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
   return dialect.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
   return self.dbapi.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
   conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
   psycopg2.OperationalError: SSL connection has been closed unexpectedly
   The above exception was the direct cause of the following exception:
   Traceback (most recent call last):
   File "/usr/local/lib/python3.6/site-packages/airflow/utils/log/s3_task_handler.py", line 177, in s3_write
   encrypt=conf.getboolean('core', 'ENCRYPT_S3_LOGS'),
   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line 409, in load_string
   self._upload_file_obj(file_obj, key, bucket_name, replace, encrypt, acl_policy)
   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line 490, in _upload_file_obj
   client = self.get_conn()
   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/S3_hook.py", line 44, in get_conn
   return self.get_client_type('s3')
   File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py", line 176, in get_client_type
   session, endpoint_url = self._get_credentials(region_name)
   File "/usr/local/lib/python3.6/site-packages/airflow/contrib/hooks/aws_hook.py", line 102, in _get_credentials
   connection_object = self.get_connection(self.aws_conn_id)
   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 84, in get_connection
   conn = random.choice(list(cls.get_connections(conn_id)))
   File "/usr/local/lib/python3.6/site-packages/airflow/hooks/base_hook.py", line 80, in get_connections
   return secrets.get_connections(conn_id)
   File "/usr/local/lib/python3.6/site-packages/airflow/secrets/__init__.py", line 52, in get_connections
   conn_list = secrets_backend.get_connections(conn_id=conn_id)
   File "/usr/local/lib/python3.6/site-packages/airflow/utils/db.py", line 74, in wrapper
   return func(*args, **kwargs)
   File "/usr/local/lib/python3.6/site-packages/airflow/secrets/metastore.py", line 38, in get_connections
   conn_list = session.query(Connection).filter(Connection.conn_id == conn_id).all()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3319, in all
   return list(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3481, in __iter__
   return self._execute_and_instances(context)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3503, in _execute_and_instances
   querycontext, self._connection_from_session, close_with_result=True
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3518, in _get_bind_args
   mapper=self._bind_mapper(), clause=querycontext.statement, **kw
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 3496, in _connection_from_session
   conn = self.session.connection(**kw)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1141, in connection
   execution_options=execution_options,
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1147, in _connection_for_bind
   engine, execution_options
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 433, in _connection_for_bind
   conn = bind._contextual_connect()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2311, in _contextual_connect
   self._wrap_pool_connect(self.pool.connect, None),
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2349, in _wrap_pool_connect
   e, dialect, self
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1591, in _handle_dbapi_exception_noconnection
   sqlalchemy_exception, with_traceback=exc_info[2], from_=e
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
   raise exception
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 2345, in _wrap_pool_connect
   return fn()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
   return _ConnectionFairy._checkout(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
   fairy = _ConnectionRecord.checkout(pool)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
   rec = pool._do_get()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
   return self._create_connection()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
   return _ConnectionRecord(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
   self.__connect(first_connect_check=True)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
   pool.logger.debug("Error on connect(): %s", e)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
   exc_value, with_traceback=exc_tb,
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
   raise exception
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
   connection = pool._invoke_creator(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
   return dialect.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
   return self.dbapi.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
   conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
   sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL connection has been closed unexpectedly
   (Background on this error at: http://sqlalche.me/e/e3q8)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 364, in connect
   return _ConnectionFairy._checkout(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 778, in _checkout
   fairy = _ConnectionRecord.checkout(pool)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 495, in checkout
   rec = pool._do_get()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/impl.py", line 239, in _do_get
   return self._create_connection()
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 309, in _create_connection
   return _ConnectionRecord(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 440, in __init__
   self.__connect(first_connect_check=True)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 661, in __connect
   pool.logger.debug("Error on connect(): %s", e)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
   exc_value, with_traceback=exc_tb,
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
   raise exception
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/pool/base.py", line 656, in __connect
   connection = pool._invoke_creator(self)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py", line 114, in connect
   return dialect.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 490, in connect
   return self.dbapi.connect(*cargs, **cparams)
   File "/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
   conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
   sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL connection has been closed unexpectedly
   (Background on this error at: http://sqlalche.me/e/e3q8)
   ```
   
   </details>
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk closed issue #9643: SSL connection has been closed unexpectedly

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #9643:
URL: https://github.com/apache/airflow/issues/9643


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #9643: SSL connection has been closed unexpectedly

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #9643:
URL: https://github.com/apache/airflow/issues/9643#issuecomment-653590257


   Unfortunately this is not a generic problem - it looks like this is not a  generic airlfow problem but rather your configuration problem so it falls under "support request/query" which should be discussed and raised in Slack, not as github issue. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] bhavaniravi commented on issue #9643: SSL connection has been closed unexpectedly

Posted by GitBox <gi...@apache.org>.
bhavaniravi commented on issue #9643:
URL: https://github.com/apache/airflow/issues/9643#issuecomment-723395297


   This issue has popped up again. Clearly, it is an infra issue, but my question is would it be better to handle `OperationalError` in airflow. 
   
   We had the same issue in the task code. We were able to close the connection in case of an error and retry something like this.
   
   ```
   BEGIN
   
   if that fails -> close connection -> open new -> try to start transaction again
   ```
   
   The problem is we cannot do the same for airflow, thereby leading to failures when
   
   1. Updating the task status
   2. Logging
   
   @potiuk @mik-laj 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org