You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/12/20 22:17:10 UTC

[GitHub] [airflow] XD-DENG opened a new issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

XD-DENG opened a new issue #13200:
URL: https://github.com/apache/airflow/issues/13200


   
   
   <!--
   
   Welcome to Apache Airflow!  For a smooth issue process, try to answer the following questions.
   Don't worry if they're not all applicable; just try to include what you can :-)
   
   If you need to include code snippets or logs, please put them in fenced code
   blocks.  If they're super-long, please use the details tag like
   <details><summary>super-long log</summary> lots of stuff </details>
   
   Please delete these comment blocks before submitting the issue.
   
   -->
   
   <!--
   
   IMPORTANT!!!
   
   PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
   NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
   
   PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
   
   Please complete the next sections or the issue will be closed.
   These questions are the first thing we need to know to understand the context.
   
   -->
   
   **Apache Airflow version**: 2.0.0
   
   **Environment**: Linux & MacOS, venv
   
   - **OS** (e.g. from /etc/os-release): Ubuntu 18.04.3 LTS / MacOS 10.15.7 
   - **Kernel** (e.g. `uname -a`):
     - Linux *** 5.4.0-1029-aws #30~18.04.1-Ubuntu SMP Tue Oct 20 11:09:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
     - Darwin *** 19.6.0 Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64 x86_64
   
   **What happened**:
   
   Say I'm in my home dir, running command `airflow scheduler -D --pid test.pid` (`test.pid` is a relative path) is supposed to start the scheduler in daemon mode, and the PID will be stored in the file `test.pid` (if it doesn't exist, it should be created).
   
   However, the scheduler is NOT started. This can be validated by running `ps aux | grep airflow | grep scheduler` (no process is shown). In the whole process, I don't see any error message.
   
   However, if I change the pid file path to an absolute path, i.e. `airflow scheduler -D --pid ${PWD}/test.pid`, it successfully start the scheduler in daemon mode (can be validated via the method above).
   
   **What you expected to happen**:
   
   Even if the PID file path provided is a relative path, the scheduler should be started properly as well.
   
   <!-- What do you think went wrong? -->
   
   **How to reproduce it**:
   
   Described above
   
   <!---
   
   As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
   
   If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
   
   ## Install minikube/kind
   
   - Minikube https://minikube.sigs.k8s.io/docs/start/
   - Kind https://kind.sigs.k8s.io/docs/user/quick-start/
   
   If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
   
   You can include images using the .md style of
   ![alt text](http://url/to/img.png)
   
   To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
   
   --->
   
   
   **Anything else we need to know**:
   
   <!--
   
   How often does this problem occur? Once? Every time etc?
   
   Any relevant logs to include? Put them here in side a detail tag:
   <details><summary>x.log</summary> lots of stuff </details>
   
   -->
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749135501


   I think when you deamonize a process, you actually want to run it in a "specific" directory. The whole idea about running process as a daemon in *NIX is to detach it completely from the originating process/command (that's why for example if you want to make a really detached daemon you should fork it twice https://stackoverflow.com/questions/881388/what-is-the-reason-for-performing-a-double-fork-when-creating-a-daemon to avoid zombies (making sure that the daemon process is parented by init process). That's why also the "DaemonContext' has the "/" as working directory set. This is not a mistake - it is pretty deliberate behaviour. 
   
   For example such daemonized process should survive and work if someone decides do delete the whole "current directory" whatever it was. The current working directory is pretty much accidental, so creating an output of "daemon" there is not really what I'd expect.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749125767


   Confirmed. 
   
   
   The reason is "working_directory" parameter not set in DaemonContext https://github.com/apache/airflow/blob/97eee350e41f6f00ad3076489a5dac2e5beaa266/airflow/cli/commands/scheduler_command.py#L48 
   
   By default the "working_directory" is set to "/" which means that the pid file is attempted to be created as "/<FILE>" - which for non-root user will fail. This works for example in Breeze as we use root user there, but in productionsetup with non-root user it will fail. 
   
   I think we should assume (extending to the code in https://github.com/apache/airflow/blob/97eee350e41f6f00ad3076489a5dac2e5beaa266/airflow/utils/cli.py#L222) than when relative path is specified, it is relative to ${AIRFLOW_HOME}.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749135501


   I think when you deamonize a process, you actually want to run it in a "different" directory. The whole idea about running process as a daemon in *NIX is to detach it completely from the originating process/command (that's why for example if you want to make a really detached daemon you should fork it twice https://stackoverflow.com/questions/881388/what-is-the-reason-for-performing-a-double-fork-when-creating-a-daemon to avoid zombies (making sure that the daemon process is parented by init process). That's why also the "DaemonContext' has the "/" as working directory set. This is not a mistake - it is pretty deliberate behaviour. 
   
   For example such daemonized process should survive and work if someone decides do delete the whole "current directory" whatever it was. The current working directory is pretty much accidental, so creating an output of "daemon" there is not really what I'd expect.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
XD-DENG commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749222728


   I start to doubt if we are discussing on the same thing now :-)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749125767


   Confirmed. 
   
   
   The reason is that default "working_directory" is used in  in DaemonContext https://github.com/apache/airflow/blob/97eee350e41f6f00ad3076489a5dac2e5beaa266/airflow/cli/commands/scheduler_command.py#L48 
   
   By default the "working_directory" is set to "/" which means that the pid file is attempted to be created as "/<FILE>" - which for non-root user will fail. This works for example in Breeze as we use root user there, but in production setup with non-root user it will fail. 
   
   I think we should assume (extending to the code in https://github.com/apache/airflow/blob/97eee350e41f6f00ad3076489a5dac2e5beaa266/airflow/utils/cli.py#L222) than when relative path is specified, it is relative to ${AIRFLOW_HOME}.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749179977


   > The current working directory is pretty much accidental, so creating an output of "daemon" there is not really what I'd expect.
   
   I view it as the opposite. Every* treats a non-absolute path input as relative to the current directory. We should behave the same. (The fact the program is deamonized is an implementation detail.)
   
   * every one I can think of 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749138484


   So maybe it's better to require a non-relative file and fail with error if it is not ? 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749690895


   No strong  opinion there :). If you want to use relative from cwd, I am fine with it :).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749135501


   I think when you deamonize a process, you actually want to run it in a "different" directory. The whole idea about running process as a daemon in *NIX is to detach it completely from the originating process/command (that's why for example if you want to make a really detached daemon you should fork it twice https://stackoverflow.com/questions/881388/what-is-the-reason-for-performing-a-double-fork-when-creating-a-daemon to avoid zombies (making sure that the daemon process is parented by init process). That's why also the "DaemonContext' has the "/" as working directory set. This is not a mistake it is pretty deliberate behaviour. 
   
   For example such daemonized process should survive and work if someone decides do delete the whole "current directory" whatever it was. The current working directory is pretty much accidental, so creating an output of "daemon" there is not really what I'd expect.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
XD-DENG commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749692382


   Thanks both @potiuk  @ashb . The discussion itself was thorough and concrete.
   
   I will update PR https://github.com/apache/airflow/pull/13232 accordingly.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb edited a comment on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
ashb edited a comment on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749223383


   Since you asked though - daemonize(1) creates the pid file, then changes directory -- i.e. relative pid file would be relative to the cwd.
   
   
   https://github.com/bmc/daemonize/blob/master/daemonize.c#L461-L500


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
XD-DENG commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749128925


   Thanks @potiuk for helping identify the root cause.
   
   I have different opinion: if it's a relative path, it should be relative to current directory where the user is running the command. I.e., as I mentioned above, if it's relative path X, extend it to CURRENT_DIR/X (rather than assuming it to be AORFLOW_HOME)
   
   Let me know if you agree?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749222301


   What deamonize does is not relevant for the behaviour we want.
   
   We are accepting a file input from the command line: what is the most common pattern, and therefor least friction/least expected to a user.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749221851


   I am not sure we should reinvent the whell. I just wonder what "deamonize" does when you do not specify the working dir. I think we should do exactly the same. Not sure how relative paths are treated by it, but I believe we should do exactly what it does.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
XD-DENG commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749689102


   Hi guys @ashb @potiuk shall we come to a consensus on this?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
XD-DENG commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749137379


   I'm more thinking from an "expectation" perspective:
   
   Say I'm in directory "/home/xd" (my AIRFLOW_HOME may be `/home/airflow_user/airflow`), and I run `airflow scheduler -D --pid test.pid`. When I specify `test.pid`, I'm expecting the PID to be saved in this PID file in current directory, whose path is `/home/xd/test.pid`. I don't expect the full path to be `/home/airflow_user/airflow/test.pid`, which will be a confusion.
   
   On the other hand, when we daemonize this process, the only thing to do with path/directory is this file where PID is stored.  We don't run a process in a "specific" directory.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749221851


   I am not sure we should reinvent the wheel. I just wonder what "deamonize" does when you do not specify the working dir. I think we should do exactly the same. Not sure how relative paths are treated by it, but I believe we should do exactly what it does.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749691478


   > No strong  opinion there :). If you want to use relative from cwd, I am fine with it :).
   
   Yes please - let's do that then @XD-DENG 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
XD-DENG commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749139541


   Haha, I agree that's a "compromise" which can make both of us happy. Why not😉
   
   I will create a PR shortly tonight for this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749223383


   Since you asked though - daemonize(1) creates the pid file, then changes directory -- i.e. relative pid file would be relative to the cwd.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb edited a comment on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
ashb edited a comment on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749179977


   > The current working directory is pretty much accidental, so creating an output of "daemon" there is not really what I'd expect.
   
   I view it as the opposite. Every* unix program that deals with filenames on the command line treats a non-absolute path input as relative to the current directory. We should behave the same. (The fact the program is deamonized is an implementation detail.)
   
   \* every one I can think of 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb edited a comment on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
ashb edited a comment on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749179977


   > The current working directory is pretty much accidental, so creating an output of "daemon" there is not really what I'd expect.
   
   I view it as the opposite. Every* treats a non-absolute path input as relative to the current directory. We should behave the same. (The fact the program is deamonized is an implementation detail.)
   
   \* every one I can think of 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb edited a comment on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
ashb edited a comment on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749222301


   What deamonize does is not relevant for the behaviour we want.
   
   We are accepting a file input from the command line: what is the most common pattern, and therefor least friction/most expected to a user.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749690895


   No strong  opinion there :). If you want to use relative dir, I am fine with it :).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749215085


   I think we should take https://linux.die.net/man/1/daemonize as the canonical behaviour. 
   
   It supports both - changing the working directory (default as "/") and specifying pidfile at the command line. It is not really specified in the man page what happens if you specify relative path, but maybe we can check and do the same.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk edited a comment on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749125767


   Confirmed. 
   
   
   The reason is that default "working_directory" is used in  in DaemonContext https://github.com/apache/airflow/blob/97eee350e41f6f00ad3076489a5dac2e5beaa266/airflow/cli/commands/scheduler_command.py#L48 
   
   By default the "working_directory" is set to "/" which means that the pid file is attempted to be created as "/<FILE>" - which for non-root user will fail. This works for example in Breeze as we use root user there, but in productionsetup with non-root user it will fail. 
   
   I think we should assume (extending to the code in https://github.com/apache/airflow/blob/97eee350e41f6f00ad3076489a5dac2e5beaa266/airflow/utils/cli.py#L222) than when relative path is specified, it is relative to ${AIRFLOW_HOME}.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
XD-DENG commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-748678790


   Fixing this issue is straigth-forward: when a PID file path is provided, always extend it to a full absolute path first. But I would like to have someone to help confirm they can reproduce the error first, meanwhile I will find out the root-cause as well of course.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG closed issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
XD-DENG closed issue #13200:
URL: https://github.com/apache/airflow/issues/13200


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] XD-DENG commented on issue #13200: CLI `airflow scheduler -D --pid ` fails silently if PIDFile given is a relative path

Posted by GitBox <gi...@apache.org>.
XD-DENG commented on issue #13200:
URL: https://github.com/apache/airflow/issues/13200#issuecomment-749217097


   But for our case, it may not be a good idea to further expose "working directory" or allow changing it. Making it simple should be better.
   
   I prefer to my very first proposal (if the path is relative, expand it with the current directory) which aligns with @ashb 's suggestion. Meanwhile I'm open to different solution if we come to an agreement by majority.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org