You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/27 21:17:44 UTC
[GitHub] [airflow] freget opened a new issue #12669: Cannot write logs to volume not supporting unix-style permissions
freget opened a new issue #12669:
URL: https://github.com/apache/airflow/issues/12669
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Apache Airflow version**: v2.0.0b2
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): v1.16.13
**Environment**:
- **Cloud provider or hardware configuration**: Azure AKS
- **OS** (e.g. from /etc/os-release): official airflow image
**What happened**:
When configuring the worker to write logs to a PersistantVolumeClaim in Kubernetes that does not support unix-style permissions (AzureFiles in my case), writing the logs - and hence the whole job - fails.
```
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 50, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 86, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 179, in task_run
ti.init_run_context(raw=args.raw)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1922, in init_run_context
self._set_context(self)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 54, in _set_context
set_context(self.log, context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 173, in set_context
handler.set_context(value)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 54, in set_context
local_loc = self._init_file(ti)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 256, in _init_file
os.chmod(full_path, 0o666)
PermissionError: [Errno 1] Operation not permitted: '/opt/airflow/logs/dagname/jobname/2020-11-27T20:57:00+00:00/1.log'
```
**What you expected to happen**:
Airflow should recognize that permissions are not available on the volume or allow disabling setting permissions for logfiles via config.
**How to reproduce it**:
Try setting the default log path to a volume not supporting permissions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12669: Cannot write logs to volume not supporting unix-style permissions
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12669:
URL: https://github.com/apache/airflow/issues/12669#issuecomment-742383605
Not strictly 2.0 related (same error in 1.10 I guess) but might be an easy one to fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cris-b commented on issue #12669: Cannot write logs to volume not supporting unix-style permissions
Posted by GitBox <gi...@apache.org>.
cris-b commented on issue #12669:
URL: https://github.com/apache/airflow/issues/12669#issuecomment-746596257
I have hit this with azure files in 1.10. The way I worked around it was to set the uid/gid perms on the pvc, but this effectively forces all worker images, the scheduler and the webserver to have a hard requirement on running as the same UID which may not be possible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cris-b commented on issue #12669: Cannot write logs to volume not supporting unix-style permissions
Posted by GitBox <gi...@apache.org>.
cris-b commented on issue #12669:
URL: https://github.com/apache/airflow/issues/12669#issuecomment-747108786
PR opened: https://github.com/apache/airflow/pull/13121
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] cris-b commented on issue #12669: Cannot write logs to volume not supporting unix-style permissions
Posted by GitBox <gi...@apache.org>.
cris-b commented on issue #12669:
URL: https://github.com/apache/airflow/issues/12669#issuecomment-747087279
A similar error was fixed for AIRFLOW-1563 by catching the OSError generated and logging it as a warning, its a bit clunky but the same could be done here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #12669: Cannot write logs to volume not supporting unix-style permissions
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #12669:
URL: https://github.com/apache/airflow/issues/12669#issuecomment-747094664
Sure @cris-b -> maybe you would like to make a PR fixing it ? IT won't make it in 2.0.0 but for 2.0.1 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #12669: Cannot write logs to volume not supporting unix-style permissions
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #12669:
URL: https://github.com/apache/airflow/issues/12669
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org