[jira] [Created] (AIRFLOW-387) Dropped postgres connections

["Andreas Merkel (JIRA)" <ji...@apache.org>]

Andreas Merkel created AIRFLOW-387:
--------------------------------------

             Summary: Dropped postgres connections
                 Key: AIRFLOW-387
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-387
             Project: Apache Airflow
          Issue Type: Bug
          Components: core
    Affects Versions: Airflow 1.7.1.3
         Environment: Running aiflow with {{postgresql+psycopg2}}, Postgres 9.1.19 backend.

{{airflow.cfg}}:

{code}
[core]
airflow_home = $PWD
dags_folder = $PWD/dags
base_log_folder = $PWD/airflow_logs
plugins_folder = $PWD/plugins
executor = LocalExecutor
sql_alchemy_conn = postgresql+psycopg2://postgres:postgres@10.128.5.58/airflow
parallelism = 32
dag_concurrency = 16
max_active_runs_per_dag = 16
load_examples = False
donot_pickle = False
fernet_key =  ; provided via environment
dags_are_paused_at_creation = False

[webserver]
expose_config = true
authenticate = False
filter_by_owner = False
workers = 4
worker_class = sync

[scheduler]
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5

[celery]
celeryd_concurrency = 1  ; Oddly enough, this is needed though we don't use celery.
{code}

Installed packages:

{code}
airflow==1.7.1.3
alembic==0.8.6
Babel==1.3
cee-syslog-handler==0.3.3
cffi==1.6.0
chartkick==0.4.2
croniter==0.3.12
cryptography==1.3.2
dill==0.2.5
docutils==0.12
enum34==1.1.6
Flask==0.10.1
Flask-Admin==1.4.0
Flask-Cache==0.13.1
Flask-Login==0.2.11
Flask-WTF==0.12
funcsigs==0.4
future==0.15.2
gunicorn==19.3.0
idna==2.1
ipaddress==1.0.16
itsdangerous==0.24
Jinja2==2.8
lockfile==0.12.2
Mako==1.0.4
Markdown==2.6.6
MarkupSafe==0.23
named-cee-logger==0.2.0
numpy==1.11.0
pandas==0.18.1
psycopg2==2.6.1
pyasn1==0.1.9
pycparser==2.14
Pygments==2.1.3
python-daemon==2.1.1
python-dateutil==2.5.3
python-editor==1.0.1
pytz==2016.4
requests==2.10.0
setproctitle==1.1.10
six==1.10.0
smoketest-dag==0.7
SQLAlchemy==1.0.13
thrift==0.9.3
Werkzeug==0.11.10
WTForms==2.1
{code}


            Reporter: Andreas Merkel


With the {{postgresql+psycopg2}} driver, we observe dropped connections in the database logs on the postgres server:

{code}
2016-08-02 14:49:26 UTC LOG:  could not receive data from client: Connection reset by peer
2016-08-02 14:49:26 UTC LOG:  unexpected EOF on client connection
{code}

Three of these messages appear even if Airflow is started without any DAGs. If there are DAGs, the message appears whenever the DAG is executed (even for simple DAGs like a Python operator that does nothing).

On the Airflow side, everything works fine; it's just the messages in the log indicating that Airflow is somehow dropping the connections to Postgres without closing them properly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)