You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/03 14:08:39 UTC
[GitHub] [airflow] terryfugit opened a new issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
terryfugit opened a new issue #17394:
URL: https://github.com/apache/airflow/issues/17394
**Apache Airflow version**: 2.1.2
**What happened**:
Receive warning message "WARNING - Recorded pid 4969 does not match the current pid 4969", then the "INFO - Sending Signals.SIGTERM to GPID 23307" message terminating our task.
We see other reported issues where the PID numbers in the message are different, but ours are always the same, not always the same number but recorded=current.
Works fine 90% or the time. This seems intermittent since upgrade to 2.1.2
**What you expected to happen**: expected to finish as normal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nmehraein commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
nmehraein commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892505803
Can also be same problem as https://github.com/apache/airflow/pull/17333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-891878172
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] terryfugit edited a comment on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
terryfugit edited a comment on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-895567088
The reported code fix above did not resolve the issue. However, back to the note from @jedcunningham , we removed the run_as_user= from the DAGs and have not seen the issue since. Our DAGS are all executing as a particular user by default, but we had run_as_user coded from earlier test environments. Removing the superfluous run_as_user rectified all of our errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] terryfugit commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
terryfugit commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892702487
@ephraimbuddy We have run_as_user set on all our dags. It does seem, as indicated, that the task is terminated at the five second heartbeat interval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] terryfugit closed issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
terryfugit closed issue #17394:
URL: https://github.com/apache/airflow/issues/17394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892065842
Likely related to `run_as_user`:
https://github.com/apache/airflow/blob/4dae4ec9a4148db6c32f15ccc122d42c93a2c58a/airflow/jobs/local_task_job.py#L198-L199
Do you have that set in your DAG or [default_impersonation](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#default-impersonation) set?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] terryfugit commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
terryfugit commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-895567088
The reported code fis above did not resolve the issue. However, back to the note from @jedcunningham , we removed the run_as_user= from the DAGs and have not seen the issue since.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892695780
> But our PID is the same. Always the same. We don't understand why we would get it in this instance.
>
> #17411 seems like a candidate. Looking through logs it seems as though we get it after the heartbeat time has lapsed. That doesn't, however, provide us an ideas on why we would get it in that case anyway since our PID is the same, nor an options for addressing the issue.
Are you using impersonation as @jedcunningham asked above? You can also explain to us how we could reproduce this, e.g how long the dag takes to run, and also provide a simple dag if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] terryfugit commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
terryfugit commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892689988
But our PID is the same. Always the same. We don't understand why we would get it in this instance.
#17411 seems like a candidate. Looking through logs it seems as though we get it after the heartbeat time has lapsed. That doesn't, however, provide us an ideas on why we would get it in that case anyway since our PID is the same, nor an options for addressing the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] terryfugit edited a comment on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
terryfugit edited a comment on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-895567088
The reported code fix above did not resolve the issue. However, back to the note from @jedcunningham , we removed the run_as_user= from the DAGs and have not seen the issue since.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] terryfugit commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
terryfugit commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892689988
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892695780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] terryfugit commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
terryfugit commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892753520
@ephraimbuddy We are going to apply this fix to see if it addresses the issue.
https://github.com/apache/airflow/pull/17333/commits/428cc76cc589efb73751bea6e925e8fa28bfe812
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] nmehraein removed a comment on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
nmehraein removed a comment on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892505803
Can also be same problem as https://github.com/apache/airflow/pull/17333
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892748830
> @ephraimbuddy We have run_as_user set on all our dags. It does seem, as indicated, that the task is terminated at the five second heartbeat interval.
Ok. Wondering if you can try the dag on the `main` branch? I suspect it has been fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #17394: WARNING - Recorded pid 4969 does not match the current pid 4969
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #17394:
URL: https://github.com/apache/airflow/issues/17394#issuecomment-892771590
> @ephraimbuddy We are going to apply this fix to see if it addresses the issue.
> [428cc76](https://github.com/apache/airflow/commit/428cc76cc589efb73751bea6e925e8fa28bfe812)
Ok. Let us know if it resolved it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org