You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/26 09:23:15 UTC
[GitHub] [airflow] sudarshan2906 opened a new issue #17223: run_as_user shows none even when default_impersonation set in the config
sudarshan2906 opened a new issue #17223:
URL: https://github.com/apache/airflow/issues/17223
**Apache Airflow version**:
2.1.2
**Environment**:
- **Cloud provider or hardware configuration**: AWS ECS
- **OS** (e.g. from /etc/os-release): debian
- **Kernel** (e.g. `uname -a`): Linux
- **Install tools**:
- **Others**:
**What happened**:
When setting default_impersonation in airflow.cfg as `testuser` and not passing run_as_user in the default_parm or operator parameters, it is showing run_as_user in the UI as None (It is using testuser to execute the dag though):
And also throwing error in logs and some of the dags are failing
```
[2021-07-26 09:53:41,929: ERROR/ForkPoolWorker-7] Failed to execute task PID of job runner does not match.
--
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/executors/celery_executor.py", line 117, in _execute_in_fork
args.func(args)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 238, in task_run
_run_task_by_selected_method(args, dag, ti)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 64, in _run_task_by_selected_method
_run_task_by_local_task_job(args, ti)
File "/usr/local/lib/python3.7/site-packages/airflow/cli/commands/task_command.py", line 121, in _run_task_by_local_task_job
run_job.run()
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 245, in run
self._execute()
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/local_task_job.py", line 131, in _execute
self.heartbeat()
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/base_job.py", line 226, in heartbeat
self.heartbeat_callback(session=session)
File "/usr/local/lib/python3.7/site-packages/airflow/utils/session.py", line 67, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/airflow/jobs/local_task_job.py", line 195, in heartbeat_callback
raise AirflowException("PID of job runner does not match")
airflow.exceptions.AirflowException: PID of job runner does not match
```
**What you expected to happen**:
Dags not to fail and logs not to show this error messages.
**How to reproduce it**:
Set default_impersonation in airflow.cfg and don't pass run_as_user in task
Run a dag with multiple task (more than 1) which runs for more than 10sec
**Anything else we need to know**:
I suspect that the if-else statement [here](https://github.com/apache/airflow/blob/main/airflow/jobs/local_task_job.py#L196) is causing the issue
Also I am trying to set run_as_user for all dags to be `testuser` using cluster policy, doing that also its giving the same error
```
def task_policy(task):
task.run_as_user = 'testuser'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #17223: run_as_user shows none even when default_impersonation set in the config
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #17223:
URL: https://github.com/apache/airflow/issues/17223#issuecomment-886656096
I have added a PR for the part that tasks are failing to run https://github.com/apache/airflow/pull/17229
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #17223: run_as_user shows none even when default_impersonation set in the config
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #17223:
URL: https://github.com/apache/airflow/issues/17223#issuecomment-886804471
> @ephraimbuddy This should also fix the task cluster policy which I am using?
>
> ```
> def task_policy(task):
> task.run_as_user = 'testuser'
> ```
I will check ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on issue #17223: run_as_user shows none even when default_impersonation set in the config
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on issue #17223:
URL: https://github.com/apache/airflow/issues/17223#issuecomment-886547214
Did this work in 2.1.1 @sudarshan2906? I mean the part that tasks are failing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudarshan2906 commented on issue #17223: run_as_user shows none even when default_impersonation set in the config
Posted by GitBox <gi...@apache.org>.
sudarshan2906 commented on issue #17223:
URL: https://github.com/apache/airflow/issues/17223#issuecomment-886562640
Not sure. It started giving error when I enabled impersonationation in 2.1.2 as mentioned here -> http://airflow.apache.org/docs/apache-airflow/stable/security/workload.html
I haven't tried it in 2.1.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sudarshan2906 commented on issue #17223: run_as_user shows none even when default_impersonation set in the config
Posted by GitBox <gi...@apache.org>.
sudarshan2906 commented on issue #17223:
URL: https://github.com/apache/airflow/issues/17223#issuecomment-886753579
@ephraimbuddy This should also fix the task cluster policy which I am using?
```
def task_policy(task):
task.run_as_user = 'testuser'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org