You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/27 22:59:55 UTC

[GitHub] [airflow] cchepelov opened a new issue #10616: modules provided and imported by packaged DAGs can get mixed up

cchepelov opened a new issue #10616:
URL: https://github.com/apache/airflow/issues/10616


   
   **Apache Airflow version**: 1.10.1, 1.10.10, code identical in 1.10.12 (reproduced with unit tests)
   
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`): N/A
   
   **Environment**:
   
   - **Cloud provider or hardware configuration**: on-prem
   - **OS** (e.g. from /etc/os-release):  CentOS 7
   - **Kernel** (e.g. `uname -a`): Linux 3.10.0-957.1.3.el7.x86_64
   - **Install tools**:
   - **Others**:
   
   **What happened**:
   
   Our DAG require some proprietary support library, which evolves from time to time. 
   The typical structure of our dag zips ("dagpacks") is the following:
   
   dag-a.zip:
   * dag-a.py
   * support_lib/
      * __init__.py
      * constants.py
      * functions.py
   
   dag-b.zip:
   * dag-b.py
   * support_lib/
      * __init__.py
      * constants.py
      * functions.py
   
   While over time, all dagpacks will tend to use the same version of the support lib, the release cycles of various DAGs are decorrelated. 
   What happens is that sometimes, and apparently randomly, dag-a will receive dag-b.zip's support_lib instead of its own.
   
   After careful analysis of the logs and the UI, it may even happen that the situation visible in the UI can be different in the scheduler process (i.e. you can have dag-a receiving dag-b's support lib in the Web UI, while the scheduler might behave as if dag-b received dag-a's support library instead !)
   
   **What you expected to happen**:
   When a zipped DAG imports a module it provides within the same zip file, it receives that module and nothing else, _ever_. Distinct DAG zips can coexist with distinct versions.
   
   
   **How to reproduce it**:
   Unit test provided in the forthcoming PR
   
   As it happens, the following structure *seems* fine:
   dag-a.zip:
   * dag-a.py
   * support_lib.py
   
   dag-b.zip:
   * dag-b.py
   * support_lib.py
   
   but migrating back to this would require us to merge back all submodules of our support library into a single file.
   
   
   **Anything else we need to know**:
   
   This problem seems to happen 100% of the time on our systems. It got unnoticed at first, as the scope of changes within the support_lib was initially timid, but it no longer is.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb edited a comment on issue #10616: modules provided and imported by packaged DAGs can get mixed up

Posted by GitBox <gi...@apache.org>.
ashb edited a comment on issue #10616:
URL: https://github.com/apache/airflow/issues/10616#issuecomment-708439516


   Is this a problem only in the webserver, or does it also affect task execution?
   
   If it's only the webserver, then this problem goes away with the enabling of DAG serialization, and that is being made mandatory from Airflow 2.0.0 onwards.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #10616: modules provided and imported by packaged DAGs can get mixed up

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #10616:
URL: https://github.com/apache/airflow/issues/10616#issuecomment-682232872


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed issue #10616: modules provided and imported by packaged DAGs can get mixed up

Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #10616:
URL: https://github.com/apache/airflow/issues/10616


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #10616: modules provided and imported by packaged DAGs can get mixed up

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10616:
URL: https://github.com/apache/airflow/issues/10616#issuecomment-823681824


   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #10616: modules provided and imported by packaged DAGs can get mixed up

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #10616:
URL: https://github.com/apache/airflow/issues/10616#issuecomment-769461865


   @cchepelov Now that Airflow 2.0 is out & following the correspondence in https://github.com/apache/airflow/pull/10617. Can you verify if it's still an issue in Airflow 2.0?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ashb commented on issue #10616: modules provided and imported by packaged DAGs can get mixed up

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #10616:
URL: https://github.com/apache/airflow/issues/10616#issuecomment-708439516


   Is this a problem only in the webserver, or does it also affect task execution?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #10616: modules provided and imported by packaged DAGs can get mixed up

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #10616:
URL: https://github.com/apache/airflow/issues/10616#issuecomment-819129305


   This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org