You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/08 08:43:18 UTC
[GitHub] [airflow] rodrigo-morais opened a new issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
rodrigo-morais opened a new issue #16881:
URL: https://github.com/apache/airflow/issues/16881
**Apache Airflow version**: 2.1.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): v1.18.17-gke.1901
**Environment**:
- **Cloud provider or hardware configuration**: Google Cloud
- **OS** (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
- **Kernel** (e.g. `uname -a`): Linux airflow-scheduler-7697b66974-m6mct 5.4.89+ #1 SMP Sat Feb 13 19:45:14 PST 2021 x86_64 GNU/Linux
- **Install tools**:
- **Others**:
**What happened**:
When the `scheduler` is restarted the currently running tasks are facing SIGTERM error.
Every time the `scheduler` is restarted or re-deployed then the current `scheduler` is terminated and a new `scheduler` is created. If during this process exist tasks running the new `scheduler` will terminate these tasks with `complete` status and new tasks will be created to continue the work of the terminated ones. After few seconds the new tasks are terminated with `error` status and SIGTERM error.
<details><summary>Error log</summary> [2021-07-07 14:59:49,024] {cursor.py:661} INFO - query execution done
[2021-07-07 14:59:49,025] {arrow_result.pyx:0} INFO - fetching data done
[2021-07-07 15:00:07,361] {local_task_job.py:196} WARNING - State of this instance has been externally set to failed. Terminating instance.
[2021-07-07 15:00:07,363] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 150
[2021-07-07 15:00:12,845] {taskinstance.py:1264} ERROR - Received SIGTERM. Terminating subprocesses.
[2021-07-07 15:00:12,907] {process_utils.py:66} INFO - Process psutil.Process(pid=150, status='terminated', exitcode=0, started='14:59:46') (150) terminated with exit code 0 </details>
**What you expected to happen**:
The tasks currently running should be allowed to finish their process or the substitute tasks should execute their process with success.
The new `scheduler` should interfere with the running tasks.
**How to reproduce it**:
To reproduce is necessary to start a DAG that has some task(s) that take some minutes to be completed. During this task(s) processing a new deploy for `scheduler` should be executed. During the re-deploy, the current `scheduler` will be terminated and a new one will be created. The current task(s) will be completed (without finish their processing) and substituted for new ones that will fail in seconds.
**Anything else we need to know**:
The problem was not happening with Airflow 1.10.15 and it started to happens after the upgrade to Airflow 2.1.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-876297877
The error log from the task doesn't show us anything useful -- it only shows that it's been set to failed in the DB.
Can you look for something relevant to this task in new and old scheduler logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rodrigo-morais commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
rodrigo-morais commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-876296394
Kubernetes executor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] DreadfullyDistinct edited a comment on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
DreadfullyDistinct edited a comment on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-971712889
Hello,
So far the git .release:2.2.2+9ac742885ffb83c15f7e3dc910b0cf9df073407a for Airflow did not resolve the issue.
We get same behaviour after Scheduler restarts (running tasks getting terminated) Will try the resolution @dberzano suggested and update. Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rodrigo-morais edited a comment on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
rodrigo-morais edited a comment on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-876388007
Pods before to deploy a new scheduler
```
NAME READY STATUS RESTARTS AGE
airflow-scheduler-5c989d8478-kvllt 1/1 Running 0 3m57s
airflow-web-74f56b9df-hvpzz 2/2 Running 0 3m57s
ascheckcountssfvspgbusinessdb.89bf993a26f3431fa07667de82954239 1/1 Running 0 40s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 1/1 Running 0 40s
dbt-docs-647db48b7d-vpj4p 1/1 Running 0 3m57s
```
Pods after to deploy the scheduler
```
NAME READY STATUS RESTARTS AGE
airflow-scheduler-5c989d8478-kvllt 1/1 Running 0 4m6s
airflow-scheduler-776f94956b-s4dfv 0/1 ContainerCreating 0 5s
airflow-web-5c667d85db-kfcx6 0/2 ContainerCreating 0 2s
airflow-web-74f56b9df-hvpzz 2/2 Running 0 4m6s
ascheckcountssfvspgbusinessdb.89bf993a26f3431fa07667de82954239 1/1 Running 0 49s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 1/1 Running 0 49s
dbt-docs-647db48b7d-vpj4p 1/1 Running 0 4m6s
```
Pods after the current scheduler have been terminated and the new one has been created
```
NAME READY STATUS RESTARTS AGE
airflow-scheduler-776f94956b-s4dfv 1/1 Running 0 3m6s
airflow-web-5c667d85db-kfcx6 2/2 Running 0 3m3s
ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2 0/1 Error 0 2m16s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 0/1 Completed 0 3m50s
dbt-docs-647db48b7d-vpj4p 1/1 Running 0 7m7s
```
<details><summary>Pod with error</summary>
```log
Running <TaskInstance: as_check_counts_sf_vs_pg.contentdb 2021-07-08T11:42:09.208095+00:00 [queued]> on host ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 237, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 64, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 120, in _run_task_by_local_task_job
run_job.run()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 237, in run
self._execute()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 133, in _execute
self.heartbeat()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 218, in heartbeat
self.heartbeat_callback(session=session)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 207, in heartbeat_callback
ti._run_finished_callback(error=error) # pylint: disable=protected-access
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1367, in _run_finished_callback
task.on_failure_callback(context)
```
</details>
<details><summary>Pod with scheduler</summary>
```log
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:850: DeprecationWarning: Specifying both AIRFLOW_HOME environment variable and airflow_home in the config file is deprecated. Please use only the AIRFLOW_HOME environment variable and remove the config file entry.
warnings.warn(msg, category=DeprecationWarning)
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-07-08 11:44:16,768] {dagrun.py:429} ERROR - Marking run <DagRun as_check_counts_sf_vs_pg @ 2021-07-08 11:42:09.208095+00:00: manual__2021-07-08T11:42:09.208095+00:00, externally triggered: True> failed
[2021-07-08 11:44:26,873] {kubernetes_executor.py:202} ERROR - Event: ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2 Failed
```
</details>
<details><summary>Log of the task</summary>
```log
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1137, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/usr/local/airflow/plugins/operators/jw_pg_vs_sf_count_check.py", line 65, in execute
cur_pg.execute(sql)
File "/usr/local/lib/python3.8/encodings/utf_8.py", line 15, in decode
def decode(input, errors='strict'):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1266, in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
[2021-07-08 13:16:42,544] {taskinstance.py:1524} INFO - Marking task as FAILED. dag_id=as_check_counts_sf_vs_pg, task_id=contentdb, execution_date=20210708T114209, start_date=20210708T131622, end_date=20210708T131642
[2021-07-08 13:16:42,601] {process_utils.py:66} INFO - Process psutil.Process(pid=149, status='terminated', exitcode=1, started='13:16:21') (149) terminated with exit code 1
```
The UI:
The previous and these logs don't have any error in the DB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] DreadfullyDistinct commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
DreadfullyDistinct commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-971712889
Hello,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] GaoJiaChengPaul edited a comment on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
GaoJiaChengPaul edited a comment on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-928504787
Hi All,
We are having the same problem as below discussion. In our case we are running 2 schedulers in two different clusters with Kubernetes Executor.
#18455
@ashb
Hi Ashb,
Any suggestions about this?
Do we need to reset a running task?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-876252227
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] DreadfullyDistinct commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
DreadfullyDistinct commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-972671589
Hi,
so our case was also related to service account permissions. We have now following permissions configured for service account, which works now fine for scheduler detecting already running pods before restart.
```
resources:
- pods
verbs:
- get
- create
- list
- delete
- patch
```
Thanks @dberzano for suggestion. if anyone sees the issue again, I guess you need to check if you get some "Resource: Forbidden" Errors on scheduler logs to see if you have a permission missing, to adopt a pod that was running before scheduler start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] GaoJiaChengPaul commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
GaoJiaChengPaul commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-928504787
Hi All,
We are having the same problem as below discussion. In our case we are running 2 schedulers in two different clusters with Kubernetes Executor.
#18455
@ashb
Hi Ashb,
Any suggestions about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-876295172
Which executor are you using @rodrigo-morais ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] GaoJiaChengPaul commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
GaoJiaChengPaul commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-928504787
Hi All,
We are having the same problem as below discussion. In our case we are running 2 schedulers in two different clusters with Kubernetes Executor.
#18455
@ashb
Hi Ashb,
Any suggestions about this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy closed issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #16881:
URL: https://github.com/apache/airflow/issues/16881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy closed issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #16881:
URL: https://github.com/apache/airflow/issues/16881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dberzano commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
dberzano commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-912603049
OK, let me retract - our problem was, seemingly, different. The `serviceaccount` provided to the scheduler was not bound to roles containing the verb `patch` for the `pod` kind... so the issue is solved for us.
@rodrigo-morais maybe it was the same issue for you, not sure. Apologies in advance for the noise, but let me share in case it helps you - In our case, the scheduler's error message contained the following failed attempt to adopt pods on the new scheduler's startup:
```
2021-09-03 13:31:57,269] {kubernetes_executor.py:663} INFO - attempting to adopt pod verylongpodname.37909dcabdcfe4598967b725b12ef92c
[2021-09-03 13:31:57,278] {kubernetes_executor.py:681} INFO - Failed to adopt pod verylongpodname.37909dcabdcfe4598967b725b12ef92c. Reason: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'aaaaaaaa-aaaa-aaaa-1111-bbbbbbbbbbbb', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Fri, 03 Sep 2021 13:31:57 GMT', 'Content-Length': '501'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"verylongpodname.37909dcabdcfe4598967b725b12ef92c\" is forbidden: User \"system:serviceaccount:ournamespace:ourserviceaccount\" cannot patch resource \"pods\" in API group \"\" in the namespace \"ournamespace\"","reason":"Forbidden","details":{"name":"verylongpodname.37909dcabdcfe4598967b725b12ef92c","kind":"pods"},"code":403}
```
We've edited the `role` resource (check with `kubectl get role`) by adding the `patch` verb to the `pods` kind, which was missing as the error message said quite clearly indeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dberzano commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
dberzano commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-912539618
For the record, we're having the same problem. This is perfectly reproducible when:
* we have any task running using the `KubernetesExecutor`
* the scheduler gets restarted for some reason
We always end up with:
* the currently-running pod being terminated,
* a new one being created in place,
* the task terminating with the following message:
```
[2021-09-03 13:23:40,216] {local_task_job.py:209} WARNING - State of this instance has been externally set to queued. Terminating instance.
```
Note that this is happening with **Airflow v2.1.3**.
(cc'ing @tomaszZdunek too)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rodrigo-morais edited a comment on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
rodrigo-morais edited a comment on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-876388007
Pods before to deploy a new scheduler
```
NAME READY STATUS RESTARTS AGE
airflow-scheduler-5c989d8478-kvllt 1/1 Running 0 3m57s
airflow-web-74f56b9df-hvpzz 2/2 Running 0 3m57s
ascheckcountssfvspgbusinessdb.89bf993a26f3431fa07667de82954239 1/1 Running 0 40s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 1/1 Running 0 40s
dbt-docs-647db48b7d-vpj4p 1/1 Running 0 3m57s
```
Pods after to deploy the scheduler
```
NAME READY STATUS RESTARTS AGE
airflow-scheduler-5c989d8478-kvllt 1/1 Running 0 4m6s
airflow-scheduler-776f94956b-s4dfv 0/1 ContainerCreating 0 5s
airflow-web-5c667d85db-kfcx6 0/2 ContainerCreating 0 2s
airflow-web-74f56b9df-hvpzz 2/2 Running 0 4m6s
ascheckcountssfvspgbusinessdb.89bf993a26f3431fa07667de82954239 1/1 Running 0 49s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 1/1 Running 0 49s
dbt-docs-647db48b7d-vpj4p 1/1 Running 0 4m6s
```
Pods after the current scheduler have been terminated and the new one has been created
```
NAME READY STATUS RESTARTS AGE
airflow-scheduler-776f94956b-s4dfv 1/1 Running 0 3m6s
airflow-web-5c667d85db-kfcx6 2/2 Running 0 3m3s
ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2 0/1 Error 0 2m16s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 0/1 Completed 0 3m50s
dbt-docs-647db48b7d-vpj4p 1/1 Running 0 7m7s
```
<details><summary>Pod with error</summary>Running <TaskInstance: as_check_counts_sf_vs_pg.contentdb 2021-07-08T11:42:09.208095+00:00 [queued]> on host ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 237, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 64, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 120, in _run_task_by_local_task_job
run_job.run()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 237, in run
self._execute()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 133, in _execute
self.heartbeat()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 218, in heartbeat
self.heartbeat_callback(session=session)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 207, in heartbeat_callback
ti._run_finished_callback(error=error) # pylint: disable=protected-access
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1367, in _run_finished_callback
task.on_failure_callback(context)</details>
<details><summary>Pod with scheduler</summary>
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:850: DeprecationWarning: Specifying both AIRFLOW_HOME environment variable and airflow_home in the config file is deprecated. Please use only the AIRFLOW_HOME environment variable and remove the config file entry.
warnings.warn(msg, category=DeprecationWarning)
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-07-08 11:44:16,768] {dagrun.py:429} ERROR - Marking run <DagRun as_check_counts_sf_vs_pg @ 2021-07-08 11:42:09.208095+00:00: manual__2021-07-08T11:42:09.208095+00:00, externally triggered: True> failed
[2021-07-08 11:44:26,873] {kubernetes_executor.py:202} ERROR - Event: ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2 Failed</details>
<details><summary>Log of the task</summary>Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1137, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1341, in _execute_task
result = task_copy.execute(context=context)
File "/usr/local/airflow/plugins/operators/jw_pg_vs_sf_count_check.py", line 65, in execute
cur_pg.execute(sql)
File "/usr/local/lib/python3.8/encodings/utf_8.py", line 15, in decode
def decode(input, errors='strict'):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1266, in signal_handler
raise AirflowException("Task received SIGTERM signal")
airflow.exceptions.AirflowException: Task received SIGTERM signal
[2021-07-08 13:16:42,544] {taskinstance.py:1524} INFO - Marking task as FAILED. dag_id=as_check_counts_sf_vs_pg, task_id=contentdb, execution_date=20210708T114209, start_date=20210708T131622, end_date=20210708T131642
[2021-07-08 13:16:42,601] {process_utils.py:66} INFO - Process psutil.Process(pid=149, status='terminated', exitcode=1, started='13:16:21') (149) terminated with exit code 1
The UI:
The previous and these logs don't have any error in the DB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] rodrigo-morais commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
rodrigo-morais commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-876388007
Pods before to deploy a new scheduler
```
NAME READY STATUS RESTARTS AGE
airflow-scheduler-5c989d8478-kvllt 1/1 Running 0 3m57s
airflow-web-74f56b9df-hvpzz 2/2 Running 0 3m57s
ascheckcountssfvspgbusinessdb.89bf993a26f3431fa07667de82954239 1/1 Running 0 40s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 1/1 Running 0 40s
dbt-docs-647db48b7d-vpj4p 1/1 Running 0 3m57s
```
Pods after to deploy the scheduler
```
NAME READY STATUS RESTARTS AGE
airflow-scheduler-5c989d8478-kvllt 1/1 Running 0 4m6s
airflow-scheduler-776f94956b-s4dfv 0/1 ContainerCreating 0 5s
airflow-web-5c667d85db-kfcx6 0/2 ContainerCreating 0 2s
airflow-web-74f56b9df-hvpzz 2/2 Running 0 4m6s
ascheckcountssfvspgbusinessdb.89bf993a26f3431fa07667de82954239 1/1 Running 0 49s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 1/1 Running 0 49s
dbt-docs-647db48b7d-vpj4p 1/1 Running 0 4m6s
```
Pods after the current scheduler have been terminated and the new one has been created
```
NAME READY STATUS RESTARTS AGE
airflow-scheduler-776f94956b-s4dfv 1/1 Running 0 3m6s
airflow-web-5c667d85db-kfcx6 2/2 Running 0 3m3s
ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2 0/1 Error 0 2m16s
ascheckcountssfvspgcontentdb.8cf4f8e0ae794b3dbc9b471c9aef4532 0/1 Completed 0 3m50s
dbt-docs-647db48b7d-vpj4p 1/1 Running 0 7m7s
```
<details><summary>Pod with error</summary>Running <TaskInstance: as_check_counts_sf_vs_pg.contentdb 2021-07-08T11:42:09.208095+00:00 [queued]> on host ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 237, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 64, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 120, in _run_task_by_local_task_job
run_job.run()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 237, in run
self._execute()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 133, in _execute
self.heartbeat()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/base_job.py", line 218, in heartbeat
self.heartbeat_callback(session=session)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/jobs/local_task_job.py", line 207, in heartbeat_callback
ti._run_finished_callback(error=error) # pylint: disable=protected-access
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1367, in _run_finished_callback
task.on_failure_callback(context)</details>
<details><summary>Pod with scheduler</summary>
/home/airflow/.local/lib/python3.8/site-packages/airflow/configuration.py:850: DeprecationWarning: Specifying both AIRFLOW_HOME environment variable and airflow_home in the config file is deprecated. Please use only the AIRFLOW_HOME environment variable and remove the config file entry.
warnings.warn(msg, category=DeprecationWarning)
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2021-07-08 11:44:16,768] {dagrun.py:429} ERROR - Marking run <DagRun as_check_counts_sf_vs_pg @ 2021-07-08 11:42:09.208095+00:00: manual__2021-07-08T11:42:09.208095+00:00, externally triggered: True> failed
[2021-07-08 11:44:26,873] {kubernetes_executor.py:202} ERROR - Event: ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2 Failed</details>
<details><summary>Log of the task</summary>[2021-07-08 11:43:57,254] {taskinstance.py:876} INFO - Dependencies all met for <TaskInstance: as_check_counts_sf_vs_pg.contentdb 2021-07-08T11:42:09.208095+00:00 [queued]>
[2021-07-08 11:43:57,268] {taskinstance.py:876} INFO - Dependencies all met for <TaskInstance: as_check_counts_sf_vs_pg.contentdb 2021-07-08T11:42:09.208095+00:00 [queued]>
[2021-07-08 11:43:57,269] {taskinstance.py:1067} INFO -
--------------------------------------------------------------------------------
[2021-07-08 11:43:57,269] {taskinstance.py:1068} INFO - Starting attempt 2 of 1
[2021-07-08 11:43:57,269] {taskinstance.py:1069} INFO -
--------------------------------------------------------------------------------
[2021-07-08 11:43:57,293] {taskinstance.py:1087} INFO - Executing <Task(jw_PgVsSfCountCheck): contentdb> on 2021-07-08T11:42:09.208095+00:00
[2021-07-08 11:43:57,298] {standard_task_runner.py:52} INFO - Started process 149 to run task
[2021-07-08 11:43:57,305] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'as_check_counts_sf_vs_pg', 'contentdb', '2021-07-08T11:42:09.208095+00:00', '--job-id', '583317', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/gen_dags.py', '--cfg-path', '/tmp/tmpz_sm9qlb', '--error-file', '/tmp/tmpmsipgv0p']
[2021-07-08 11:43:57,307] {standard_task_runner.py:77} INFO - Job 583317: Subtask contentdb
[2021-07-08 11:43:57,415] {logging_mixin.py:104} INFO - Running <TaskInstance: as_check_counts_sf_vs_pg.contentdb 2021-07-08T11:42:09.208095+00:00 [running]> on host ascheckcountssfvspgcontentdb.44eee838b6754b0aa0c8e6288a6195a2
[2021-07-08 11:43:57,550] {taskinstance.py:1280} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=Rodrigo
AIRFLOW_CTX_DAG_ID=as_check_counts_sf_vs_pg
AIRFLOW_CTX_EXECUTION_DATE=2021-07-08T11:42:09.208095+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-07-08T11:42:09.208095+00:00
[2021-07-08 11:43:57,611] {connection.py:257} INFO - Snowflake Connector for Python Version: 2.4.6, Python Version: 3.8.10, Platform: Linux-5.4.89+-x86_64-with-glibc2.2.5
[2021-07-08 11:43:57,612] {connection.py:868} INFO - This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity.
[2021-07-08 11:43:57,612] {connection.py:886} INFO - Setting use_openssl_only mode to False
[2021-07-08 11:43:58,015] {cursor.py:637} INFO - query: [SELECT T.TABLE_NAME, IFF(C.COLUMN_NAME IS NULL, 0, 1) AS HAS_CREATED_AT FROM SHA...]
[2021-07-08 11:43:59,060] {cursor.py:661} INFO - query execution done
[2021-07-08 11:43:59,061] {arrow_result.pyx:0} INFO - fetching data done
[2021-07-08 11:43:59,077] {cursor.py:637} INFO - query: [SELECT 'AGE_CERTIFICATION' AS TABLE_NAME, COUNT(1) AS COUNT, NULL AS CREATED_AT ...]
[2021-07-08 11:43:59,575] {cursor.py:661} INFO - query execution done
[2021-07-08 11:43:59,576] {arrow_result.pyx:0} INFO - fetching data done
[2021-07-08 11:44:17,664] {local_task_job.py:196} WARNING - State of this instance has been externally set to failed. Terminating instance.
[2021-07-08 11:44:17,666] {process_utils.py:100} INFO - Sending Signals.SIGTERM to GPID 149
[2021-07-08 11:44:25,218] {taskinstance.py:1264} ERROR - Received SIGTERM. Terminating subprocesses.
[2021-07-08 11:44:25,276] {process_utils.py:66} INFO - Process psutil.Process(pid=149, status='terminated', exitcode=0, started='11:43:57') (149) terminated with exit code 0</details>
The UI:
The previous and these logs don't have any error in the DB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] saisujithkp commented on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
saisujithkp commented on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-878412513
Could be a Duplicate of #16298, I'm having the same issue when migrated from 1.10.14 to 2.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] GaoJiaChengPaul edited a comment on issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
GaoJiaChengPaul edited a comment on issue #16881:
URL: https://github.com/apache/airflow/issues/16881#issuecomment-928504787
Hi All,
We are having the same problem as below discussion. In our case we are running 2 schedulers in two different clusters with Kubernetes Executor.
#18455
@ashb
Hi Ashb,
Any suggestions about this?
Do we need to reset a running task?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy closed issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #16881:
URL: https://github.com/apache/airflow/issues/16881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy closed issue #16881: Re-deploy scheduler tasks failing with SIGTERM on K8s executor
Posted by GitBox <gi...@apache.org>.
ephraimbuddy closed issue #16881:
URL: https://github.com/apache/airflow/issues/16881
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org