You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/07/25 03:20:35 UTC
[GitHub] [airflow] zikun opened a new pull request #9993: Avoid sharing session with RenderedTaskInstanceFields write and delete
zikun opened a new pull request #9993:
URL: https://github.com/apache/airflow/pull/9993
Sharing session with RTIF leads to idle-in-transaction timeout error when DAG serialization is enabled and task running duration exceeds the idle-in-transaction timeout setting of the database.
The change was introduced in #6788.
In many production databases, the idle-in-transaction timeout is not unlimited. For example, if the timeout is set to 12 hours, any task that runs for more than 12 hours will raise an exception even if the actual task succeeds.
Procedure to reproduce the bug and isolate the issue:
Use this docker-compose:
```
version: "3"
services:
webserver:
image: apache/airflow:1.10.11
volumes:
- logs:/opt/airflow/logs
environment:
- AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://postgres:airflow@postgres/airflow
- AIRFLOW__CORE__EXECUTOR=LocalExecutor
- AIRFLOW__CORE__STORE_SERIALIZED_DAGS=True
- AIRFLOW__CORE__STORE_DAG_CODE=True
- AIRFLOW__CORE__LOGGING_LEVEL=DEBUG
depends_on:
- postgres
command: webserver
ports:
- "8080:8080"
scheduler:
image: apache/airflow:1.10.11
volumes:
- logs:/opt/airflow/logs
- ./dags/:/opt/airflow/dags/:ro
environment:
- AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://postgres:airflow@postgres/airflow
- AIRFLOW__CORE__EXECUTOR=LocalExecutor
- AIRFLOW__CORE__STORE_SERIALIZED_DAGS=True
- AIRFLOW__CORE__STORE_DAG_CODE=True
- AIRFLOW__CORE__LOGGING_LEVEL=DEBUG
depends_on:
- postgres
command: scheduler
postgres:
image: library/postgres:10.7
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
volumes:
- /dev/urandom:/dev/random # Required to get non-blocking entropy source
ports:
- "5432:5432"
command: postgres -c 'idle_in_transaction_session_timeout=60000'
volumes:
logs:
```
and put this DAG in dags folder:
```
from datetime import datetime
import time
import logging
from airflow.models.dag import DAG
from airflow.operators.python_operator import PythonOperator
logging.getLogger('sqlalchemy.engine').setLevel(logging.DEBUG)
def sleep():
logging.info("sleep")
time.sleep(61)
with DAG(
dag_id="sleep",
schedule_interval="0 0 * * *",
catchup=False,
default_args={
"start_date": datetime(2020, 1, 1)
},
) as dag:
sensor = PythonOperator(task_id="sleep_py", python_callable=sleep)
```
Run the following:
```
docker-compose up -d postgres
docker-compose run --rm webserver initdb
docker-compose up -d
```
Enable the DAG in webserver http://localhost:8080/ and wait for 1 minute to see the log.
{python_operator.py:114} INFO - Done. Returned value was: None
{taskinstance.py:1150} ERROR - (psycopg2.errors.IdleInTransactionSessionTimeout) terminating connection due to idle-in-transaction timeout
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
Search for `BEGIN (implicit)` in the log. The `BEGIN (implicit)` for `rendered_task_instance_fields` does not have a corresponding `COMMIT`
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #9993: Avoid sharing session with RenderedTaskInstanceFields write and delete
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #9993:
URL: https://github.com/apache/airflow/pull/9993
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #9993: Avoid sharing session with RenderedTaskInstanceFields write and delete
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #9993:
URL: https://github.com/apache/airflow/pull/9993#issuecomment-663883132
Thanks @zikun
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org