You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/31 12:45:53 UTC

[GitHub] [airflow] ashb opened a new pull request #15110: Use pathlib.Path for building paths in testfile

ashb opened a new pull request #15110:
URL: https://github.com/apache/airflow/pull/15110


   We do a lot of path manipulation in this test file, and it's easier to
   understand by using pathlib without all the nested `os.path.*` calls.
   
   This is a "drive-by" refactor while working on another PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb merged pull request #15110: Allow pathlib.Path in DagBag and various utils

Posted by GitBox <gi...@apache.org>.
ashb merged pull request #15110:
URL: https://github.com/apache/airflow/pull/15110


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] uranusjr commented on a change in pull request #15110: Use pathlib.Path for building paths in testfile

Posted by GitBox <gi...@apache.org>.
uranusjr commented on a change in pull request #15110:
URL: https://github.com/apache/airflow/pull/15110#discussion_r604875027



##########
File path: tests/utils/test_dag_processing.py
##########
@@ -50,7 +51,7 @@
 from tests.test_utils.config import conf_vars
 from tests.test_utils.db import clear_db_dags, clear_db_runs, clear_db_serialized_dags
 
-TEST_DAG_FOLDER = os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir, 'dags')
+TEST_DAG_FOLDER = pathlib.Path(__file__).parent.parent / 'dags'

Review comment:
       I wonder if it’d be easier to make `DagBag` accept `Union[Path, str, None]` instead and do the `str()` there. Persuambly `pathlib` would get more use as the main code base progresses as well?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on a change in pull request #15110: Use pathlib.Path for building paths in testfile

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #15110:
URL: https://github.com/apache/airflow/pull/15110#discussion_r604871660



##########
File path: tests/utils/test_dag_processing.py
##########
@@ -50,7 +51,7 @@
 from tests.test_utils.config import conf_vars
 from tests.test_utils.db import clear_db_dags, clear_db_runs, clear_db_serialized_dags
 
-TEST_DAG_FOLDER = os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir, 'dags')
+TEST_DAG_FOLDER = pathlib.Path(__file__).parent.parent / 'dags'

Review comment:
       Should we just do since everywhere we make it `str`:
   
   ```python
   TEST_DAG_FOLDER = str(pathlib.Path(__file__).parent.parent / 'dags')
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] github-actions[bot] commented on pull request #15110: Allow pathlib.Path in DagBag and various utils

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15110:
URL: https://github.com/apache/airflow/pull/15110#issuecomment-811094201


   The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest master at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #15110: Use pathlib.Path for building paths in testfile

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15110:
URL: https://github.com/apache/airflow/pull/15110#discussion_r604877090



##########
File path: tests/utils/test_dag_processing.py
##########
@@ -50,7 +51,7 @@
 from tests.test_utils.config import conf_vars
 from tests.test_utils.db import clear_db_dags, clear_db_runs, clear_db_serialized_dags
 
-TEST_DAG_FOLDER = os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir, 'dags')
+TEST_DAG_FOLDER = pathlib.Path(__file__).parent.parent / 'dags'

Review comment:
       Let me take a look.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #15110: Use pathlib.Path for building paths in testfile

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15110:
URL: https://github.com/apache/airflow/pull/15110#discussion_r604883490



##########
File path: tests/utils/test_dag_processing.py
##########
@@ -50,7 +51,7 @@
 from tests.test_utils.config import conf_vars
 from tests.test_utils.db import clear_db_dags, clear_db_runs, clear_db_serialized_dags
 
-TEST_DAG_FOLDER = os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir, 'dags')
+TEST_DAG_FOLDER = pathlib.Path(__file__).parent.parent / 'dags'

Review comment:
       A bit of a bigger change, but probably worth it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on pull request #15110: Allow pathlib.Path in DagBag and various utils

Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #15110:
URL: https://github.com/apache/airflow/pull/15110#issuecomment-811066971


   The commit message is now wrong and will be changed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #15110: Use pathlib.Path for building paths in testfile

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15110:
URL: https://github.com/apache/airflow/pull/15110#discussion_r604874752



##########
File path: tests/utils/test_dag_processing.py
##########
@@ -50,7 +51,7 @@
 from tests.test_utils.config import conf_vars
 from tests.test_utils.db import clear_db_dags, clear_db_runs, clear_db_serialized_dags
 
-TEST_DAG_FOLDER = os.path.join(os.path.dirname(os.path.realpath(__file__)), os.pardir, 'dags')
+TEST_DAG_FOLDER = pathlib.Path(__file__).parent.parent / 'dags'

Review comment:
       If we make this a str here we can't use `TEST_DAG_FOLDER / "some_dag"` that I do elsewhere.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] kaxil commented on a change in pull request #15110: Allow pathlib.Path in DagBag and various utils

Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #15110:
URL: https://github.com/apache/airflow/pull/15110#discussion_r604908035



##########
File path: airflow/models/dagbag.py
##########
@@ -450,7 +453,8 @@ def collect_dags(
         # Used to store stats around DagBag processing
         stats = []
 
-        dag_folder = correct_maybe_zipped(dag_folder)
+        # Ensure dag_folder is a str -- it may have been a pathlib.Path
+        dag_folder = correct_maybe_zipped(str(dag_folder))

Review comment:
       Should we also let `correct_maybe_zipped` accept Pathlib too and convert it to str if needed in the function itself ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] ashb commented on a change in pull request #15110: Allow pathlib.Path in DagBag and various utils

Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15110:
URL: https://github.com/apache/airflow/pull/15110#discussion_r604912680



##########
File path: airflow/models/dagbag.py
##########
@@ -450,7 +453,8 @@ def collect_dags(
         # Used to store stats around DagBag processing
         stats = []
 
-        dag_folder = correct_maybe_zipped(dag_folder)
+        # Ensure dag_folder is a str -- it may have been a pathlib.Path
+        dag_folder = correct_maybe_zipped(str(dag_folder))

Review comment:
       We could -- I wasn't sure how deep to chase this is all.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org