[jira] [Commented] (AIRFLOW-3979) airflow scheduler can't connect to my postgres database

["Dan Ryan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880021#comment-16880021 ] 

Dan Ryan commented on AIRFLOW-3979:
-----------------------------------

I'd like to second this issue – I've seen a number of these issues closed as duplicates and the provided resolution is that the warning is not being displayed, but I'd encourage the maintainers to pause for a second and consider why we are looking at logs and reporting the issue – speaking for myself, at least, it's because my scheduler is no longer processing work, none of my tasks are progressing, and the only thing my logs are telling me is that the connection has been invalidated (sometimes) or that DNS resolution doesn't work anymore (sometimes) or that SSL is broken (sometimes).

 

This is only broken on 1.10.3 for me, I have an instance of 1.10.1 running in production but 1.10.3 is completely unusable for me, not just because it logs this message but because the message seems associated with an actual problem. I'd be happy to have someone look at this to gather more info as I'm not too familiar with the codebase or where to begin troubleshooting.

> airflow scheduler can't connect to my postgres database
> -------------------------------------------------------
>
>                 Key: AIRFLOW-3979
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3979
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: scheduler
>            Reporter: Patrick Whitlock
>            Priority: Major
>
> [2019-02-28 23:11:52,031] \{jobs.py:1477} INFO - Starting the scheduler
> [2019-02-28 23:11:52,031] \{jobs.py:1485} INFO - Running execute loop for -1 seconds
> [2019-02-28 23:11:52,031] \{jobs.py:1486} INFO - Processing each file at most -1 times
> [2019-02-28 23:11:52,032] \{jobs.py:1489} INFO - Searching for files in /root/airflow/dags
> [2019-02-28 23:11:52,035] \{jobs.py:1491} INFO - There are 20 files in /root/airflow/dags
> [2019-02-28 23:11:52,081] \{jobs.py:1534} INFO - Resetting orphaned tasks for active dag runs
> [2019-02-28 23:11:52,096] \{dag_processing.py:453} INFO - Launched DagFileProcessorManager with pid: 7567
> [2019-02-28 23:11:52,097] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:11:52,102] \{settings.py:51} INFO - Configured default timezone <Timezone [UTC]>
> [2019-02-28 23:11:52,111] \{settings.py:174} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=7567
> [2019-02-28 23:11:54,099] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:11:56,101] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:11:58,104] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:00,106] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:02,108] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:04,110] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:06,113] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:08,115] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:10,117] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:12,120] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:13,146] \{sqlalchemy.py:79} WARNING - DB connection invalidated. Reconnecting...
> [2019-02-28 23:12:14,122] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:16,124] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:18,126] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:20,129] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:22,131] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:24,133] \{jobs.py:1559} INFO - Harvesting DAG parsing results
> [2019-02-28 23:12:24,155] \{sqlalchemy.py:79} WARNING - DB connection invalidated. Reconnecting...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)