You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/20 19:40:05 UTC
[GitHub] [airflow] vinny-santaiti opened a new issue #13797: Sentry celery dag task run error
vinny-santaiti opened a new issue #13797:
URL: https://github.com/apache/airflow/issues/13797
**Apache Airflow version**: 2.0.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): N/A
**Environment**:
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release): Centos7
- **Kernel** (e.g. `uname -a`): Linux 3.10.0-693.5.2.el7.x86_64
- **Install tools**: celery==4.4.0, sentry-sdk==0.19.5
- **Others**: python 3.6.8
**What happened**:
We see this in the sentry error logs randomly for all dag tasks:
`TypeError in airflow.executors.celery_executor.execute_command`
```
TypeError: _run_mini_scheduler_on_child_tasks() got multiple values for argument 'session'
File "airflow/sentry.py", line 159, in wrapper
return func(task_instance, *args, session=session, **kwargs)
```
**What you expected to happen**:
No error in sentry.
**How to reproduce it**:
Schedule or manually run a dag task such as PythonOperator.
The error msg will appear when airflow runs dag task.
The error will not appear in the airflow web server logs but only on Sentry server.
**Anything else we need to know**:
N/A
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-765485709
@tiopi @AbhiPrasad @1untitaker Can I ask for help? Do you have time to look at these integrations and fix them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-763885913
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Junnplus commented on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
Junnplus commented on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-765201194
full context
```
_run_mini_scheduler_on_child_tasks() got multiple values for argument 'session'
airflow/sentry.py in wrapper at line 159
def wrapper(task_instance, *args, session=None, **kwargs):
# Wrapping the _run_raw_task function with push_scope to contain
# tags and breadcrumbs to a specific Task Instance
with sentry_sdk.push_scope():
try:
return func(task_instance, *args, session=session, **kwargs)
except Exception as e:
self.add_tagging(task_instance)
self.add_breadcrumbs(task_instance, session=session)
sentry_sdk.capture_exception(e)
raise
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gedOHub commented on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
gedOHub commented on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-767078208
Hey guys,
I'm seeing the same issue on my instance too
Instance info:
- AirFlow version 2.0.0
- Python version 3.8.7
- Pip version 20.2.4
- Sentry-sdk version 0.19.5
```
# airflow/sentry.py in wrapper at line 159
...
def wrapper(task_instance, *args, session=None, **kwargs):
# Wrapping the _run_raw_task function with push_scope to contain
# tags and breadcrumbs to a specific Task Instance
with sentry_sdk.push_scope():
try:
>>>>> return func(task_instance, *args, session=session, **kwargs)
except Exception as e:
self.add_tagging(task_instance)
self.add_breadcrumbs(task_instance, session=session)
sentry_sdk.capture_exception(e)
raise
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gedOHub edited a comment on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
gedOHub edited a comment on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-767078208
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jporter-dev commented on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
jporter-dev commented on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-767778342
Having the same issue using the LocalExecutor with Airflow 2.0 running on Ubuntu 20.04.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gedOHub commented on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
gedOHub commented on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-767078208
Hey guys,
I'm seeing the same issue on my instance too
Instance info:
- AirFlow version 2.0.0
- Python version 3.8.7
- Pip version 20.2.4
- Sentry-sdk version 0.19.5
```
# airflow/sentry.py in wrapper at line 159
...
def wrapper(task_instance, *args, session=None, **kwargs):
# Wrapping the _run_raw_task function with push_scope to contain
# tags and breadcrumbs to a specific Task Instance
with sentry_sdk.push_scope():
try:
>>>>> return func(task_instance, *args, session=session, **kwargs)
except Exception as e:
self.add_tagging(task_instance)
self.add_breadcrumbs(task_instance, session=session)
sentry_sdk.capture_exception(e)
raise
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gedOHub edited a comment on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
gedOHub edited a comment on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-767078208
Hey guys,
I'm seeing the same issue on my instance too
Instance info:
- AirFlow version 2.0.0
- Python version 3.8.7
- Pip version 20.2.4
- Sentry-sdk version 0.19.5
Running on docker container
```
# airflow/sentry.py in wrapper at line 159
...
def wrapper(task_instance, *args, session=None, **kwargs):
# Wrapping the _run_raw_task function with push_scope to contain
# tags and breadcrumbs to a specific Task Instance
with sentry_sdk.push_scope():
try:
>>>>> return func(task_instance, *args, session=session, **kwargs)
except Exception as e:
self.add_tagging(task_instance)
self.add_breadcrumbs(task_instance, session=session)
sentry_sdk.capture_exception(e)
raise
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gedOHub edited a comment on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
gedOHub edited a comment on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-767078208
Hey guys,
I'm seeing the same issue on my instance too
Instance info:
- AirFlow version 2.0.0
- Python version 3.8.7
- Pip version 20.2.4
- Sentry-sdk version 0.19.5
Running on docker container
```
# airflow/sentry.py in wrapper at line 159
...
def wrapper(task_instance, *args, session=None, **kwargs):
# Wrapping the _run_raw_task function with push_scope to contain
# tags and breadcrumbs to a specific Task Instance
with sentry_sdk.push_scope():
try:
>>>>> return func(task_instance, *args, session=session, **kwargs)
except Exception as e:
self.add_tagging(task_instance)
self.add_breadcrumbs(task_instance, session=session)
sentry_sdk.capture_exception(e)
raise
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb closed issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
ashb closed issue #13797:
URL: https://github.com/apache/airflow/issues/13797
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] maxcountryman commented on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
maxcountryman commented on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-788584540
Also seeing this issue.
It's particularly noisy and somewhat disruptive as it seems to also immediately trigger: `Failed to execute task Task received SIGTERM signal.` and then `AirflowException: Celery command failed on host: {redacted}` quite often as of Airflow 2.0.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Junnplus commented on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
Junnplus commented on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-765200094
related code: https://github.com/apache/airflow/blob/master/airflow/models/taskinstance.py#L1176
`_run_mini_scheduler_on_child_tasks` use positional arguments
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jporter-dev edited a comment on issue #13797: Sentry celery dag task run error
Posted by GitBox <gi...@apache.org>.
jporter-dev edited a comment on issue #13797:
URL: https://github.com/apache/airflow/issues/13797#issuecomment-767778342
Having the same issue using the LocalExecutor.
- Airflow 2.0.0
- Python 3.8.5
- pip 20.2.4
- Ubuntu 20.04
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org