You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/22 08:24:55 UTC
[GitHub] [airflow] jagusztinl opened a new issue #15482: Unable to log to Azure PVC
jagusztinl opened a new issue #15482:
URL: https://github.com/apache/airflow/issues/15482
**Apache Airflow version**:
2.0.1
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
Openshift 4.5 on Azure
**Environment**:
Azure PVC
- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
Installed Airflow 2.0.1 to OKD, on Azure, defined Azure PVC, fileshare to store the logs
<!-- (please include exact error messages if you can) -->
[Yesterday 9:18 PM]
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 89, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 225, in task_run
ti.init_run_context(raw=args.raw)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1987, in init_run_context
self._set_context(self)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 54, in _set_context
set_context(self.log, context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 56, in set_context
local_loc = self._init_file(ti)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/file_task_handler.py", line 258, in _init_file
os.chmod(full_path, 0o666)
PermissionError: [Errno 1] Operation not permitted: '/opt/airflow/logs/example_bash_operator/runme_0/2021-04-20T00:00:00+00:00/1.log'
**What you expected to happen**:
Write logs to PVC
<!-- What do you think went wrong? -->
Permission problem, Airflow do not handle well cloud storage
**How to reproduce it**:
<!---
Install AIrflow to AKS or OKD on Azure, set persistent logging to PVC
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Yes
<img width="537" alt="Névtelen" src="https://user-images.githubusercontent.com/46651189/115681490-de793d80-a354-11eb-9cc4-f8e559b8326b.png">
You can see the Azire files directories created, but unable to set permissions
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #15482: Unable to log to Azure PVC
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #15482:
URL: https://github.com/apache/airflow/issues/15482#issuecomment-983351500
Ah right! great then :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15482: Unable to log to Azure PVC
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15482:
URL: https://github.com/apache/airflow/issues/15482#issuecomment-983348536
Oh actually, the try-catch was added in 2.1. So we should close this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] janvaca92 commented on issue #15482: Unable to log to Azure PVC
Posted by GitBox <gi...@apache.org>.
janvaca92 commented on issue #15482:
URL: https://github.com/apache/airflow/issues/15482#issuecomment-831105942
We have faced the same issue. The problem is that Azure fileshare does not allowed changing file permissions via os.chmod. With the managed disks is possible but they require ReadWriteOnce and its is not suitable for Airflow logs pvc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #15482: Unable to log to Azure PVC
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #15482:
URL: https://github.com/apache/airflow/issues/15482#issuecomment-983344499
> I _guess_ it's not entirely unreasonable to simply skip the chmod call (and log a warning) if it fails.
looks like we already do that but not for permission errors
https://github.com/apache/airflow/blob/f6dca1fa5e70ef08798adeb5a6bfc70f41229646/airflow/utils/log/file_task_handler.py#L277-L281
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #15482: Unable to log to Azure PVC
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #15482:
URL: https://github.com/apache/airflow/issues/15482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15482: Unable to log to Azure PVC
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15482:
URL: https://github.com/apache/airflow/issues/15482#issuecomment-831843288
I *guess* it's not entirely unreasonable to simply skip the chmod call (and log a warning) if it fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15482: Unable to log to Azure PVC
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15482:
URL: https://github.com/apache/airflow/issues/15482#issuecomment-983346994
`PermissionError` is a subclass of `OSError`, so this should cover it. But I think this needs to cover the `open().close()` line above so it catches permission errors to the _parent_ directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org