You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/10/22 12:57:42 UTC
[GitHub] [airflow] artur-v-zaripov opened a new pull request #19157: fix "mismatch process ids" issue for the "run as user" case
artur-v-zaripov opened a new pull request #19157:
URL: https://github.com/apache/airflow/pull/19157
**ISSUE**
During the upgrade from Airflow 1.10.15 to Airflow 2.1.4 we've faced the following error:
```python
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/cli.py", line 92, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 292, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 105, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 163, in _run_task_by_local_task_job
run_job.run()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 245, in run
self._execute()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/local_task_job.py", line 134, in _execute
self.heartbeat()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 226, in heartbeat
self.heartbeat_callback(session=session)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/jobs/local_task_job.py", line 209, in heartbeat_callback
raise AirflowException("PID of job runner does not match")
airflow.exceptions.AirflowException: PID of job runner does not match
```
Also, we tried to reproduce it on clean install using Helm charts from the repository and Airflow version 2.2 - the same issue occurs.
**SAMPLE DAG**
We are using `KubernetesExecutor`, DAGs with `KubernetesPodOperator`, and `run_as_user` setting.
To reproduce the bug, we've created a sample dag:
```python
from airflow import DAG
from datetime import datetime, timedelta
from airflow.providers.cncf.kubernetes.operators import kubernetes_pod
from airflow.operators.dummy_operator import DummyOperator
from kubernetes.client import models as k8s
YESTERDAY = datetime.now() - timedelta(days=1)
default_args = {
'depends_on_past': False,
'retries': 0,
'start_date': datetime(2019, 1, 1),
'namespace': "airflow-new",
'image_pull_policy': "Always",
'is_delete_operator_pod': False,
'in_cluster': True,
'startup_timeout_seconds': 300,
'hostnetwork': True,
'run_as_user': "airflow"
}
with DAG(
dag_id='chart_test_dag',
schedule_interval="30 0 * * *",
max_active_runs=1,
catchup=False,
start_date=YESTERDAY,
default_args=default_args,
) as dag:
start = DummyOperator(task_id="start", dag=dag)
pro = kubernetes_pod.KubernetesPodOperator(
task_id="pro",
dag=dag,
name="pro",
cmds=["bash", "-cx"],
arguments=["pwd"],
env_vars=[k8s.V1EnvVar(name="EMR_CONFIG_ID", value="DEV")],
image="ubuntu:latest",
namespace="airflow-new",
)
start >> pro
```
**ENVIRONMENT**
Python 3.8
Kubernetes 1.19
**RELATED TO**
Probably, related issues:
https://github.com/apache/airflow/issues/17394
https://github.com/apache/airflow/issues/18041
**PR**
This PR contains the fix that helped to avoid this issue. Any other solutions are much appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #19157: fix "mismatch process ids" issue for the "run as user" case
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #19157:
URL: https://github.com/apache/airflow/pull/19157
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #19157: fix "mismatch process ids" issue for the "run as user" case
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #19157:
URL: https://github.com/apache/airflow/pull/19157#issuecomment-988360953
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #19157: fix "mismatch process ids" issue for the "run as user" case
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #19157:
URL: https://github.com/apache/airflow/pull/19157#issuecomment-949607089
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] derkuci commented on pull request #19157: fix "mismatch process ids" issue for the "run as user" case
Posted by GitBox <gi...@apache.org>.
derkuci commented on pull request #19157:
URL: https://github.com/apache/airflow/pull/19157#issuecomment-1001737855
I have the same problem when upgrading from 1.9 to 2.2. I couldn't figure out a fix and have to give up all the use of "run_as_user" (had to rewrite/rearrange the tasks). That's a shame.
I tried to go through the airflow code, but with very limited knowledge about its architect assumptions, I didn't progress much. All I can guess is that there's inconsistency between how a task identify itself and how it communicates (heartbeat) with the scheduler/db/whatever. Your proposed code change seems a good start, but don't fully resolve the inconsistency.
For example, with Celery, the process hierarchy with run_as_user is
```
celery worker process
\-- (forked) task process
\-- sudo process
```
I've seen in `LocalTaskJob.heartbeat_callback()`, `ti.pid=<task process pid>` and `current_pid=<sudo process pid>`. The comparison is actually between `ti.pid's ppid` i.e. `<celery worker process pid>` versus `<sudo process pid>` which I couldn't understand at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org