You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/10 10:40:26 UTC

[GitHub] [airflow] mluscon opened a new issue #21487: Requiring sudo privilages

mluscon opened a new issue #21487:
URL: https://github.com/apache/airflow/issues/21487


   ### Apache Airflow version
   
   2.2.3 (latest released)
   
   ### What happened
   
   If airflow worker process is run under non-root user and a dag contains run_as statement, airflow calls "sudo chmod" during the creation of tmp files. The following code snippet comes from /[airflow](https://github.com/apache/airflow/tree/main/airflow)/[task](https://github.com/apache/airflow/tree/main/airflow/task)/[task_runner](https://github.com/apache/airflow/tree/main/airflow/task/task_runner)/base_task_runner.py .
   ```
           if self.run_as_user and (self.run_as_user != getuser()):
               # We want to include any environment variables now, as we won't
               # want to have to specify them in the sudo call - they would show
               # up in `ps` that way! And run commands now, as the other user
               # might not be able to run the cmds to get credentials
               cfg_path = tmp_configuration_copy(chmod=0o600, include_env=True, include_cmds=True)
   
               # Give ownership of file to user; only they can read and write
               subprocess.check_call(
                   ['sudo', 'chown', self.run_as_user, cfg_path, self._error_file.name], close_fds=True
               )
   
               # propagate PYTHONPATH environment variable
               pythonpath_value = os.environ.get(PYTHONPATH_VAR, '')
               popen_prepend = ['sudo', '-E', '-H', '-u', self.run_as_user]
   ```
   Calling `sudo chown` in subprocess actually requires airflow process user to be able to sudo to root which averts the main point of running airflow worker process as a non-root user. I would therefore propose to allow run_as_user read and write by setting facl attributes of tmp file accordingly. 
   
   Will you be willing to accept a PR addressing this issue via setfacl?
   
   ### What you expected to happen
   
   _No response_
   
   ### How to reproduce
   
   _No response_
   
   ### Operating System
   
   Linux
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034916837


   BTW. Is there a chance that setfacl command will be unavailable or configured not to work? I guess we can conditionally run the old sudo way in case it fails just in case?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034925023


   But it is a nice feature indeed.
   
   BTW. Just to correct your statement - the current option with sudo does NOT avert the main purpose. Maybe there is a misunderstanding about the purpose of impersonation.
   
   The purpose of impersonation is not to prevent the "airflow" user to run as root, but to make sure that the "task" coming from DAG wirtten by DAG author is executed only using the "target user" (which does not have sudo). So it worked fine before as expected. 
   
   The FACL solution simply changes the requirements that the airflow user must fullfill to be able to use. Instead of having `sudo as root` the "${TMP_DIR}"  folder must be mounted on acl-enabled volume and the airlfow user should be able to `sudo as run_for_user`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034925023


   But it is a nice feature indeed.
   
   BTW. Just to correct your statement - the current option with sudo does NOT avert the main purpose. Maybe there is a misunderstanding about the purpose of impersonation.
   
   The purpose of impersonation is not to prevent the "airflow" user to run as root, but to make sure that the "task" coming from DAG wirtten by DAG author is executed only using the "target user" (which does not have to use sudo). So it worked fine before as expected. 
   
   The FACL solution simply changes the requirements that the airflow user must fullfill to be able to use. Instead of having `sudo to root` the "${TMP_DIR}"  folder must be mounted on acl-enabled volume and the airlfow user should be able to `sudo as run_for_user`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] mluscon commented on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
mluscon commented on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1035480216


   > Looked a bit closer - I assume you only want to add the "airflow" user a capability to only sudo to "run_as_user" but not to root? Seems doable.
   
   Exactly, my "airflow" user is capable of impersonating all of run_as_users that I am using but not root.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034919265


   Ah yeah - `acl` needs to be enabled when you mount partition, so while it does make sense, you need to add it to impersonation prerequisites and provide fallback to sudo in this case. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034925023


   But it is a nice feature indeed.
   
   BTW. Just to correct your statement - the current option with sudo does NOT avert the main purpose. Maybe there is a misunderstanding about the purpose of impersonation.
   
   The purpose of impersonation is not to prevent the "airflow" user to run as root, but to make sure that the "task" coming from DAG wirtten by DAG author is executed only using the "target user" (which does not have to use sudo). So it worked fine before as expected. 
   
   The FACL solution simply changes the requirements that the airflow user must fullfill to be able to use. Instead of having `sudo as root` the "${TMP_DIR}"  folder must be mounted on acl-enabled volume and the airlfow user should be able to `sudo as run_for_user`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034925023


   But it is a nice feature indeed.
   
   BTW. Just to correct your statement - the current option with sudo does NOT avert the main purpose. Maybe there is a misunderstanding about the purpose of impersonation.
   
   The purpose of impersonation is not to prevent the "airflow" user to run as root, but to make sure that the "task" coming from DAG wirtten by DAG author is executed only using the "target run_as_user" (which does not have sudo). So it worked fine before as expected. 
   
   The FACL solution simply changes the requirements that the airflow user must fullfill to be able to use impersonation. 
   
   Instead of having `sudo as root` the "${TMP_DIR}"  folder must be mounted on acl-enabled volume and the airlfow user should be able to `sudo as run_for_user`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034761172


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034919265


   Ah yeah - `acl` needs to be enabled when you mount partition as mount option, so while it does make sense, you need to add it to impersonation prerequisites and provide fallback to sudo in this case. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034919265


   Ah yeah - `acl` needs to be enabled when you mount partition, so while it does make sense, we need to add it to impersonation prerequisites and provide fallback to sudo in this case. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034925023


   But it is a nice feature indeed.
   
   BTW. Just to correct your statement - the current option with sudo does NOT avert the main purpose. Maybe there is a misunderstanding about the purpose of impersonation.
   
   The purpose of impersonation is not to prevent the "airflow" user to run as root, but to make sure that the "task" coming from DAG wirtten by DAG author is executed only using the "target run_as_user" (which does not have sudo). So it worked fine before as expected. 
   
   The FACL solution simply changes the requirements that the airflow user must fullfill to be able to use impersonation. 
   
   Instead of having `sudo as root`, the "${TMP_DIR}"  folder must be mounted on acl-enabled volume and the airlfow user should be able to `sudo as run_for_user`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034925023


   But it is a nice feature indeed.
   
   BTW. Just to correct your statement - the current option with sudo does NOT avert the main purpose. Maybe there is a misunderstanding about the purpose of impersonation.
   
   The purpose of impersonation is not to prevent the "airflow" user to run as root, but to make sure that the "task" coming from DAG wirtten by DAG author is executed only using the "target run_as_user" (which does not have sudo). So it worked fine before as expected. 
   
   The FACL solution simply changes the requirements that the airflow user must fullfill to be able to use. Instead of having `sudo as root` the "${TMP_DIR}"  folder must be mounted on acl-enabled volume and the airlfow user should be able to `sudo as run_for_user`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034910501


   Why do you want to do it? 
   Few lines below you have the "sudo" command to run the command as the other user. What woudl setfacl change in this case if you anyhow need to give `airflow` user to be added to sudoers ? 
   
   ```
   popen_prepend = ['sudo', '-E', '-H', '-u', self.run_as_user]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #21487: Requiring sudo privilages

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21487:
URL: https://github.com/apache/airflow/issues/21487#issuecomment-1034914258


   Looked a bit closer - I assume you only want to add the "airflow" user a capability to only sudo to "run_as_user" but not to root? 
   Seems doable.
   
   Did you test if all will work in this case? Does FACL not require any special permissions to set the file to be owner by the "run_as_user""? 
   
   If  so - feel free to make a PR. But please remember to update "impersonation" documentation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org