You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/21 09:08:12 UTC
[GitHub] [airflow] lgov opened a new issue #17129: All tasks fail on logging to a Windows drive (Kubernetes-Multipass-Microk8s on Windows host)
lgov opened a new issue #17129:
URL: https://github.com/apache/airflow/issues/17129
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 2.1.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): v1.19.7
**Environment**:
- **Cloud provider or hardware configuration**: Local PC
- **OS** (e.g. from /etc/os-release): Host OS: Windows 10 Pro.
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
On the Windows host we have installed Microk8s. The installer installs MultiPass with an Ubuntu VM. Airflow is installed using the helm chart from https://airflow.apache.org . We use the KubernetesExecutor and tasks are started as pods in-cluster.
Specific to this environment is that we instructed airflow to write the log files to a folder inside the Ubuntu VM, which itself is mapped to a folder on the windows host using `multipass mount` .
**What happened**:
Every task that gets started in a Kubernetes pod fails immediately with the following exception:
```
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/bin/airflow", line 8, in <module>
sys.exit(main())
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
args.func(args)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 91, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/task_command.py", line 227, in task_run
ti.init_run_context(raw=args.raw)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/models/taskinstance.py", line 2013, in init_run_context
self._set_context(self)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/logging_mixin.py", line 54, in _set_context
set_context(self.log, context)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/logging_mixin.py", line 174, in set_context
handler.set_context(value)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_task_handler.py", line 56, in set_context
local_loc = self._init_file(ti)
File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/log/file_task_handler.py", line 253, in _init_file
Path(directory).mkdir(mode=0o777, parents=True, exist_ok=True)
File "/usr/local/lib/python3.6/pathlib.py", line 1253, in mkdir
self.mkdir(mode, parents=False, exist_ok=exist_ok)
File "/usr/local/lib/python3.6/pathlib.py", line 1248, in mkdir
self._accessor.mkdir(self, mode)
File "/usr/local/lib/python3.6/pathlib.py", line 387, in wrapped
return strfunc(str(pathobj), *args)
PermissionError: [Errno 1] Operation not permitted: '/app/xxxxxxx/airflow/logs/Dag1/select_pipeline/2021-07-19T13:28:02.735574+00:00'
```
**What you expected to happen**:
When I run the exact same setup on a Mac OS X host it works without problem, so I expect the tasks to run in a pod and logs to be written.
The problem is very likely the following:
- airflow is able to make the logs folder /app/xxxxxxx/airflow/logs/Dag1/select_pipeline where Dag1 is the dag name and select_pipeline the task name.
- It fails on making the folder .../2021-07-19T13:28:02.735574+00:00 .
- I also can not create this folder manually, as the ':' character is not allowed for folder/filenames on a Windows file system.
The proposed solution then is to use log folder naming conventions that are supported on all filesystems including on Windows hosts.
**Anything else we need to know**:
This is an example of issue #10388 . While solving this issue will not make airflow compatible with Windows, at least it will allow us to use a Microk8s VM on our development machines and test environment regardless of whether the developer is using a Mac, a Windows PC or Ubuntu.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #17129: All tasks fail on logging to a Windows drive (Kubernetes-Multipass-Microk8s on Windows host)
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #17129:
URL: https://github.com/apache/airflow/issues/17129#issuecomment-888016545
> While I can make a quick fix, I have no development setup on a Windows machines available to validate.
It’s OK, we don’t currently have Windows in CI so we wouldn’t know you didn’t check if you don’t tell us :p
Jokes aside, IMO the change from `:` to `_` should be good on its own; whether it would fix log files on Windows, I don’t know—maybe we’ll find something else broken after we fix this—but this specific fix is definitely a change in the correct direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lgov commented on issue #17129: All tasks fail on logging to a Windows drive (Kubernetes-Multipass-Microk8s on Windows host)
Posted by GitBox <gi...@apache.org>.
lgov commented on issue #17129:
URL: https://github.com/apache/airflow/issues/17129#issuecomment-887669851
Thanks for the feedback @uranusjr. Changing the ':' in the time to '_' would work, or just remove them. While I can make a quick fix, I have no development setup on a Windows machines available to validate.
As a workaround, we now keep the logs inside the microk8s-vm VM (which uses Ubuntu), that works, but makes them less easy accessible. While they should be visible in the GUI, they aren't, but that's issue #9927 we are encountering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #17129: All tasks fail on logging to a Windows drive (Kubernetes-Multipass-Microk8s on Windows host)
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #17129:
URL: https://github.com/apache/airflow/issues/17129#issuecomment-884044784
I mentioned in https://github.com/apache/airflow/pull/16110#discussion_r640487904 we should just replace the `:` with `_`. Would you be interested in creating a pull request for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org