You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/17 07:55:20 UTC
[GitHub] [airflow] dstandish opened a new pull request, #26449: WIP - Fix k8s executor when home path different
dstandish opened a new pull request, #26449:
URL: https://github.com/apache/airflow/pull/26449
Since #21877 we no longer parse the dag as part of --local task run.
As a consequence it reads dag file location from the database. But if the dag file is in a different path on the k8s executor worker, then this will be the wrong path. We can fix this by respecting the subdir from the top level command.
One problem is, k8s executor, the way it builds the pod args, it sets subdir=DAGS_FOLDER which means that all dags in the folder get parsed, which, we should avoid somehow. So, have to work on this more and see whether we can improve that.
It may be easier to just store the relative loc in the DB.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26449: WIP - Fix k8s executor when home path different
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26449:
URL: https://github.com/apache/airflow/pull/26449#issuecomment-1250924085
@pingzh - maybe you can have better idea how to fix it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #26449: WIP - Fix k8s executor when home path different
Posted by GitBox <gi...@apache.org>.
dstandish commented on PR #26449:
URL: https://github.com/apache/airflow/pull/26449#issuecomment-1251280054
> @pingzh - maybe you can have better idea how to fix it ?
@potiuk, since having a DAGS_FOLDER is _required_ full stop, why don't we always store the _relative_ fileloc? Then airflow can always reassemble the paths relative to DAGS_FOLDER and this is a non-issue....
@ashb any concerns with this idea?
It would seem possible that we could fix this as part of an upgrade (i.e. update all filelocs to be relative).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish closed pull request #26449: WIP - Fix k8s executor when home path different
Posted by GitBox <gi...@apache.org>.
dstandish closed pull request #26449: WIP - Fix k8s executor when home path different
URL: https://github.com/apache/airflow/pull/26449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #26449: WIP - Fix k8s executor when home path different
Posted by GitBox <gi...@apache.org>.
dstandish commented on PR #26449:
URL: https://github.com/apache/airflow/pull/26449#issuecomment-1252579729
i'm closing this one in favor of #26509
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dstandish commented on pull request #26449: WIP - Fix k8s executor when home path different
Posted by GitBox <gi...@apache.org>.
dstandish commented on PR #26449:
URL: https://github.com/apache/airflow/pull/26449#issuecomment-1251415534
> > @potiuk, since having a DAGS_FOLDER is _required_ full stop, why don't we always store the _relative_ fileloc? Then airflow can always reassemble the paths relative to DAGS_FOLDER and this is a non-issue....
>
> Sounds good. Actually I thought we've already fixed those some time ago to be relative :)
i will work on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #26449: WIP - Fix k8s executor when home path different
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #26449:
URL: https://github.com/apache/airflow/pull/26449#issuecomment-1251413495
> @potiuk, since having a DAGS_FOLDER is _required_ full stop, why don't we always store the _relative_ fileloc? Then airflow can always reassemble the paths relative to DAGS_FOLDER and this is a non-issue....
Sounds good. Actually I thought we've already fixed those some time ago tobe relative :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org