You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/02/18 22:50:47 UTC

[GitHub] [airflow] msb217 opened a new pull request #7456: [AIRFLOW 6817] - Move airflow __init__.py imports to sub-packages

msb217 opened a new pull request #7456: [AIRFLOW 6817]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456
 
 
   ---
   Issue link: WILL BE INSERTED BY [boring-cyborg](https://github.com/kaxil/boring-cyborg)
   
   Moved `airflow/__init__.py` imports to respective sub-packages with the exception of the `settings.initialize()` and `integrate_plugins()`
   
   All breeze unit and integration tests are passing locally.
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589765104
 
 
   Plus you will need to rebase it @msb217 !

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382725838
 
 

 ##########
 File path: airflow/__init__.py
 ##########
 @@ -30,13 +30,8 @@
 # pylint: disable=wrong-import-position
 from typing import Callable, Optional
 
-from airflow import utils
 from airflow import settings
 from airflow import version
-from airflow.utils.log.logging_mixin import LoggingMixin
-from airflow.configuration import conf
-from airflow.exceptions import AirflowException
-from airflow.models.dag import DAG
 
 Review comment:
   But no matter what - if we do it, it should be separate PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] boring-cyborg[bot] commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589927913
 
 
   Awesome work, congrats on your first merged pull request!
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589927943
 
 
   Thanks @msb217 ! First commit and so important for us !

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382572247
 
 

 ##########
 File path: airflow/providers/qubole/example_dags/example_qubole_sensor.py
 ##########
 @@ -0,0 +1,80 @@
+#
 
 Review comment:
   Unrelated?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382758098
 
 

 ##########
 File path: tests/jobs/test_local_task_job.py
 ##########
 @@ -24,10 +24,11 @@
 import pytest
 from mock import patch
 
-from airflow import AirflowException, models, settings
+from airflow import settings
+from airflow.exceptions import AirflowException
 from airflow.executors.sequential_executor import SequentialExecutor
 from airflow.jobs import LocalTaskJob
-from airflow.models import DAG, TaskInstance as TI
+from airflow.models import DAG, DagBag, TaskInstance as TI
 
 Review comment:
   This is a bit tricky, because we are relying on it in "resetdb" case. The models when imported register themselves in sqlalchemy and resetdb uses that information to know which tables to delete. However we already have some exceptions to that - some models are moved out of models package __init__.py and have to be added manually to resetdb. 
   
   I think what would make more sense is to have a separate models/all_models.py which would import all models and this one can be used by resetdb,
   
   But I think we should make a separate PR for that. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] msb217 commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
msb217 commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382699222
 
 

 ##########
 File path: airflow/providers/qubole/example_dags/example_qubole_sensor.py
 ##########
 @@ -0,0 +1,80 @@
+#
 
 Review comment:
   Yes - sorry I accidentally included this after a rebase. When I squash/rebase this should be resolved

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] msb217 commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
msb217 commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382698927
 
 

 ##########
 File path: airflow/__init__.py
 ##########
 @@ -30,13 +30,8 @@
 # pylint: disable=wrong-import-position
 from typing import Callable, Optional
 
-from airflow import utils
 from airflow import settings
 from airflow import version
-from airflow.utils.log.logging_mixin import LoggingMixin
-from airflow.configuration import conf
-from airflow.exceptions import AirflowException
-from airflow.models.dag import DAG
 
 Review comment:
   To do this I'm assuming you'd like to see the `integrate_plugins()` functionality decoupled? 
   
   So for example, `integrate_operator_plugins()` would correspond to the `airflow.operators` module. The function `integrate_sensor_plugins()` would correspond to the `airflow.sensors` module and so on for hooks/macros. 
   
   I'm sure it'd be more complex than what I'm describing, but the idea of only initializing plugins for their respective modules is what you'd be going for, correct?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589962141
 
 
   This is going to need a change to the vast majority of dag in existence - our own docs http://airflow.apache.org/docs/stable/tutorial.html say `from airflow import DAG` .
   
   Is there any corner tricks we can do to have that import still work but only import models etc when first accessed?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382734250
 
 

 ##########
 File path: tests/operators/test_latest_only_operator.py
 ##########
 @@ -21,8 +21,8 @@
 
 from freezegun import freeze_time
 
-from airflow import DAG, settings
 from airflow.models import DagRun, TaskInstance
+from airflow.models.dag import DAG, settings
 
 Review comment:
   ```suggestion
   from airflow.models.dag import DAG
   from airflow import settings
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382733633
 
 

 ##########
 File path: tests/jobs/test_local_task_job.py
 ##########
 @@ -24,10 +24,11 @@
 import pytest
 from mock import patch
 
-from airflow import AirflowException, models, settings
+from airflow import settings
+from airflow.exceptions import AirflowException
 from airflow.executors.sequential_executor import SequentialExecutor
 from airflow.jobs import LocalTaskJob
-from airflow.models import DAG, TaskInstance as TI
+from airflow.models import DAG, DagBag, TaskInstance as TI
 
 Review comment:
   Should we import DAG explicitly? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589965811
 
 
   I think we have the easiest migration path for our users to follow:
   ```
   [1.10] py2 (those still on py2) -> [1.10] py3 -> [1.10] py3 + providers (incremental) -> [1.10] py3 + providers + 2.0-compatible DAGs (incremental) -> [2.0] 
   ```
   
   This way while migrating the users will have to deal with one 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382734632
 
 

 ##########
 File path: tests/cli/commands/test_sync_perm_command.py
 ##########
 @@ -19,16 +19,16 @@
 import unittest
 from unittest import mock
 
-from airflow import DAG, models
 from airflow.bin import cli
 from airflow.cli.commands import sync_perm_command
+from airflow.models import DAG, DagBag
 
 Review comment:
   Should we import DAG directly from `airflow.models.dag`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] msb217 commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
msb217 commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382743849
 
 

 ##########
 File path: tests/jobs/test_local_task_job.py
 ##########
 @@ -24,10 +24,11 @@
 import pytest
 from mock import patch
 
-from airflow import AirflowException, models, settings
+from airflow import settings
+from airflow.exceptions import AirflowException
 from airflow.executors.sequential_executor import SequentialExecutor
 from airflow.jobs import LocalTaskJob
-from airflow.models import DAG, TaskInstance as TI
+from airflow.models import DAG, DagBag, TaskInstance as TI
 
 Review comment:
   Yes I think that would be good for consistency.
   
   Would it be worth considering a similar change to `airflow/models/__init__.py`? Removing the implicit imports of models?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] msb217 commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
msb217 commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589961780
 
 
   Thanks @potiuk , I am happy to contribute and hope that I can help with more issues in the future!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] BasPH commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
BasPH commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589962857
 
 
   I like the change but also with @ashb here, it's going to break a lot of workflows which import `from airflow import DAG`. We should at least show deprecation warnings.
   
   @ashb I don't understand your question? I know of the magic `__import__` function for changing import behaviour, but don't know if it can be applied to your question?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589963187
 
 
   For python 3 we can possibly do https://www.python.org/dev/peps/pep-0562/ do have a airflow.DAG still work, but issue a depreciation warning. I'll take a look at a warning on Monday

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589971716
 
 
   I'm raising the point about what should our API be on the mailing list.
   
   Plus this PR didn't (yet) have much of an impact on import times - https://github.com/apache/airflow/pull/7505 :) 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589224098
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=h1) Report
   > Merging [#7456](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/4e0e2f0da39626c1bc8026a33c2080a6b11f2c41?src=pr&el=desc) will **decrease** coverage by `0.17%`.
   > The diff coverage is `99%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7456/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7456      +/-   ##
   ==========================================
   - Coverage   86.79%   86.62%   -0.18%     
   ==========================================
     Files         887      887              
     Lines       41976    41991      +15     
   ==========================================
   - Hits        36432    36373      -59     
   - Misses       5544     5618      +74
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `100% <ø> (ø)` | :arrow_up: |
   | [airflow/executors/dask\_executor.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZGFza19leGVjdXRvci5weQ==) | `5.88% <0%> (ø)` | :arrow_up: |
   | [airflow/providers/imap/hooks/imap.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvaW1hcC9ob29rcy9pbWFwLnB5) | `94.95% <100%> (+0.04%)` | :arrow_up: |
   | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.52% <100%> (ø)` | :arrow_up: |
   | [airflow/providers/dingding/hooks/dingding.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZGluZ2RpbmcvaG9va3MvZGluZ2RpbmcucHk=) | `66.66% <100%> (ø)` | :arrow_up: |
   | [...ow/providers/docker/example\_dags/example\_docker.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlci5weQ==) | `100% <100%> (ø)` | :arrow_up: |
   | [.../providers/google/cloud/hooks/kubernetes\_engine.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2t1YmVybmV0ZXNfZW5naW5lLnB5) | `95.94% <100%> (+0.05%)` | :arrow_up: |
   | [airflow/providers/opsgenie/hooks/opsgenie\_alert.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvb3BzZ2VuaWUvaG9va3Mvb3BzZ2VuaWVfYWxlcnQucHk=) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/executors/local\_executor.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvbG9jYWxfZXhlY3V0b3IucHk=) | `92.48% <100%> (ø)` | :arrow_up: |
   | [.../cloud/operators/cloud\_storage\_transfer\_service.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9jbG91ZF9zdG9yYWdlX3RyYW5zZmVyX3NlcnZpY2UucHk=) | `95.63% <100%> (ø)` | :arrow_up: |
   | ... and [77 more](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=footer). Last update [4e0e2f0...0306223](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] msb217 closed pull request #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
msb217 closed pull request #7456: [AIRFLOW-6817]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589972033
 
 
   Not yet indeed :).
   
   Let's talk at the devlist indeed. I understand where you are coming from, but I think it's not good idea for the users in the future to switch to that.
   
   While I understand (and concur) we might want to add PEP-562 behavior for backwards compatibility, I don't think it should be default behavior for examples and future users:
   
   - explicit is better than implicit
   - IDE support and mypy will go crazy (I believe - I would like to check it when it's done)
   
   I think a lot of our users are not super-hackers and they use IDEs to write the code. If the PEP-562 results with having no autocomplete/verification of proper import, then it's a no-go for me.
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
ashb commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589969717
 
 
   I've thought about this a bit more, and I'm coming at this from a different point of view than simple migration - that of user facing API and what we should expose to them.
   
   Also the easier we make the migration the more likely it is to not leave users behind. So if they _can_ run old dags with new Airflow so much the better. Lets take a lesson from Django here, they've thought long and hard about how to do migrations and upgrades. In summary:
   
   If code works in one version without warnings, it MUST work in the next version but can issue deprecation warnings, and can only be removed in the release after that.
   
   Also to me, `from airflow import DAG` is nicer -- why should the DAG authors have to care about the exact internal implementation of where we've chosen to move this class to now?. Airflow and DAGS go hand in hand -- with PEP-562 we can have lazily imported classes but still present a nice interface to users. And plus we can then re-arrange things later under the hood without having to affect our users. After all, what is the harm if the importing everything is solved?
   
   My proposal here:
   
   - Add in a `_getattr_` based lazy import for DAG to airflow/__init__.py module
   - Do _NOT_ issue a deprecation warning for this
   - Revert the change to all the imports in example dags etc so that those do `from airflow import DAG`.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589965503
 
 
   One thing to consider.
   
   I would like to see if we can think slightly differently here.
   
   This is a change that is super easy to add to automated "migration assistant" tool that we will have to do anyway. I think we will have far more issues to deal with in terms of backwards compatibility (for example bash_operator and python_operator have moved).
   
   I think personally that with the 2.0 migration we should simply make people aware that they have to migrate using migration tool (or manually). Otherwise they will have a false sense of having "compatible" dags. I think telling people they will be able to move their dags to 2.0 and they will continue to work as usual, is a bit miscommunication (or even a bit of a lie - we know it's not going to work in a number of cases). 
   
   Migration to 2.0 will be anyway pain for many people but they can deal with one pain at at time - i.e. DAG migration first, then migrate to 2.0 Airflow infrastructure. 
   
   We can make it  easier for people to migrate to 2.0 - if they will have the new "back-ported" providers packages installed - the migration to the "both 1.10 and 2.0" compatible dags might be done automatically even BEFORE migration to 2.0 and our tool should be able to tell how "ready" you are to migrate to 2.0. For now - I believe none of the changes we've done prevents us from having Dags that are both 1.0 and 2.0 compatible (and if we find one - we can still easily fix it in master to keep that possibility).
   
   The "migration assistant" can produce both - 1.10 and 2.0 compatible DAGs (for example imports changed to "airflow.models.DAG" etc. This should be very easy with tools like Bowler : https://pybowler.io/  - we used it successfully to refactor tons of python code (basically all providers code!) and we can easily produce dag-migration assistant that can be used by people.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] ashb edited a comment on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
ashb edited a comment on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589962141
 
 
   This is going to need a change to the vast majority of dag in existence - our own docs http://airflow.apache.org/docs/stable/tutorial.html say `from airflow import DAG` .
   
   Is there any clever tricks we can do to have that import still work but only import models etc when first accessed?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382759655
 
 

 ##########
 File path: tests/jobs/test_local_task_job.py
 ##########
 @@ -24,10 +24,11 @@
 import pytest
 from mock import patch
 
-from airflow import AirflowException, models, settings
+from airflow import settings
+from airflow.exceptions import AirflowException
 from airflow.executors.sequential_executor import SequentialExecutor
 from airflow.jobs import LocalTaskJob
-from airflow.models import DAG, TaskInstance as TI
+from airflow.models import DAG, DagBag, TaskInstance as TI
 
 Review comment:
   I created an issue for that https://issues.apache.org/jira/browse/AIRFLOW-6870

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] msb217 commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
msb217 commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382743849
 
 

 ##########
 File path: tests/jobs/test_local_task_job.py
 ##########
 @@ -24,10 +24,11 @@
 import pytest
 from mock import patch
 
-from airflow import AirflowException, models, settings
+from airflow import settings
+from airflow.exceptions import AirflowException
 from airflow.executors.sequential_executor import SequentialExecutor
 from airflow.jobs import LocalTaskJob
-from airflow.models import DAG, TaskInstance as TI
+from airflow.models import DAG, DagBag, TaskInstance as TI
 
 Review comment:
   Yes I think that would be good for consistency.
   
   Would it be worth considering a similar change to `airflow/models/__init__.py`? Removing the implicit imports of models?
   
   The implicit imports of models are used quite frequently

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382725105
 
 

 ##########
 File path: airflow/__init__.py
 ##########
 @@ -30,13 +30,8 @@
 # pylint: disable=wrong-import-position
 from typing import Callable, Optional
 
-from airflow import utils
 from airflow import settings
 from airflow import version
-from airflow.utils.log.logging_mixin import LoggingMixin
-from airflow.configuration import conf
-from airflow.exceptions import AirflowException
-from airflow.models.dag import DAG
 
 Review comment:
   Something like that. I already had a PR for that (but I think I abandon it as there were many changes in the meantime #6696 )

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589965811
 
 
   I think we have the easiest migration path for our users to follow:
   ```
   [1.10] py2 (those still on py2) -> 
        [1.10] py3 -> 
            [1.10] py3 + providers (incremental) -> 
               [1.10] py3 + providers + 2.0-compatible DAGs (incremental) -> 
                   [2.0] 
   ```
   
   This way while migrating the users will have to deal with one problem at a time and the most difficult part (adapting DAGs) can be mostly automated and can be done incrementally while they are still running 1.10. The fact that this is incremental and that they do not have to migrate everything at once is really appealing for many users I think.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] msb217 commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
msb217 commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382698927
 
 

 ##########
 File path: airflow/__init__.py
 ##########
 @@ -30,13 +30,8 @@
 # pylint: disable=wrong-import-position
 from typing import Callable, Optional
 
-from airflow import utils
 from airflow import settings
 from airflow import version
-from airflow.utils.log.logging_mixin import LoggingMixin
-from airflow.configuration import conf
-from airflow.exceptions import AirflowException
-from airflow.models.dag import DAG
 
 Review comment:
   To do this I'm assuming you'd like to see the `integrate_plugins()` functionality decoupled? 
   
   So for example, `integrate_operator_plugins()` would correspond to the `airflow.operators` module. The function `integrate_sensor_plugins()` would correspond to the `airflow.sensors` module and so on for hooks/macros. 
   
   I'm sure it'd be more complex than what I'm describing, but the idea of only initializing plugins for their respective sub-modules is what you'd be going for, correct?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on issue #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on issue #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589775485
 
 
   I love this change! <3

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589224098
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=h1) Report
   > Merging [#7456](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6502cfa8e61cb57673df327f8d1548c4bef89bcf?src=pr&el=desc) will **increase** coverage by `0.01%`.
   > The diff coverage is `98.98%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7456/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7456      +/-   ##
   ==========================================
   + Coverage    86.7%   86.71%   +0.01%     
   ==========================================
     Files         883      883              
     Lines       41635    41650      +15     
   ==========================================
   + Hits        36099    36117      +18     
   + Misses       5536     5533       -3
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `100% <ø> (ø)` | :arrow_up: |
   | [airflow/executors/dask\_executor.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZGFza19leGVjdXRvci5weQ==) | `5.88% <0%> (ø)` | :arrow_up: |
   | [airflow/providers/imap/hooks/imap.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvaW1hcC9ob29rcy9pbWFwLnB5) | `94.95% <100%> (+0.04%)` | :arrow_up: |
   | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.52% <100%> (ø)` | :arrow_up: |
   | [airflow/providers/dingding/hooks/dingding.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZGluZ2RpbmcvaG9va3MvZGluZ2RpbmcucHk=) | `66.66% <100%> (ø)` | :arrow_up: |
   | [...ow/providers/docker/example\_dags/example\_docker.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlci5weQ==) | `100% <100%> (ø)` | :arrow_up: |
   | [.../providers/google/cloud/hooks/kubernetes\_engine.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2t1YmVybmV0ZXNfZW5naW5lLnB5) | `95.94% <100%> (+0.05%)` | :arrow_up: |
   | [airflow/providers/opsgenie/hooks/opsgenie\_alert.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvb3BzZ2VuaWUvaG9va3Mvb3BzZ2VuaWVfYWxlcnQucHk=) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/executors/local\_executor.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvbG9jYWxfZXhlY3V0b3IucHk=) | `92.48% <100%> (ø)` | :arrow_up: |
   | [.../cloud/operators/cloud\_storage\_transfer\_service.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9jbG91ZF9zdG9yYWdlX3RyYW5zZmVyX3NlcnZpY2UucHk=) | `95.63% <100%> (ø)` | :arrow_up: |
   | ... and [71 more](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=footer). Last update [6502cfa...e5b541a](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io commented on issue #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589224098
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=h1) Report
   > Merging [#7456](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/aff3a361b4092212c0757f9ce88fa2e40d25d1f4?src=pr&el=desc) will **decrease** coverage by `78.78%`.
   > The diff coverage is `5.55%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7456/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #7456       +/-   ##
   ==========================================
   - Coverage   86.57%   7.79%   -78.79%     
   ==========================================
     Files         882     882               
     Lines       41597   41608       +11     
   ==========================================
   - Hits        36013    3243    -32770     
   - Misses       5584   38365    +32781
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `100% <ø> (ø)` | :arrow_up: |
   | [airflow/providers/imap/hooks/imap.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvaW1hcC9ob29rcy9pbWFwLnB5) | `0% <0%> (-94.92%)` | :arrow_down: |
   | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `40.36% <0%> (-56.15%)` | :arrow_down: |
   | [airflow/providers/dingding/hooks/dingding.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZGluZ2RpbmcvaG9va3MvZGluZ2RpbmcucHk=) | `0% <0%> (-66.67%)` | :arrow_down: |
   | [...ow/providers/docker/example\_dags/example\_docker.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlci5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
   | [.../providers/google/cloud/hooks/kubernetes\_engine.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2t1YmVybmV0ZXNfZW5naW5lLnB5) | `0% <0%> (-95.9%)` | :arrow_down: |
   | [airflow/providers/opsgenie/hooks/opsgenie\_alert.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvb3BzZ2VuaWUvaG9va3Mvb3BzZ2VuaWVfYWxlcnQucHk=) | `0% <0%> (-100%)` | :arrow_down: |
   | [.../cloud/operators/cloud\_storage\_transfer\_service.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9jbG91ZF9zdG9yYWdlX3RyYW5zZmVyX3NlcnZpY2UucHk=) | `0% <0%> (-95.64%)` | :arrow_down: |
   | [...ow/providers/google/cloud/operators/sftp\_to\_gcs.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9zZnRwX3RvX2djcy5weQ==) | `0% <0%> (-100%)` | :arrow_down: |
   | [...flow/providers/google/cloud/operators/functions.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9mdW5jdGlvbnMucHk=) | `0% <0%> (-98.02%)` | :arrow_down: |
   | ... and [886 more](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=footer). Last update [aff3a36...18c9202](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk merged pull request #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382571494
 
 

 ##########
 File path: airflow/__init__.py
 ##########
 @@ -30,13 +30,8 @@
 # pylint: disable=wrong-import-position
 from typing import Callable, Optional
 
-from airflow import utils
 from airflow import settings
 from airflow import version
-from airflow.utils.log.logging_mixin import LoggingMixin
-from airflow.configuration import conf
-from airflow.exceptions import AirflowException
-from airflow.models.dag import DAG
 
 Review comment:
   I think we also need to remove plugins eventually - but this is a good step (and it will be a bit more complex as we have to make sure plugins are initialized where needed).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382907274
 
 

 ##########
 File path: tests/jobs/test_local_task_job.py
 ##########
 @@ -24,10 +24,11 @@
 import pytest
 from mock import patch
 
-from airflow import AirflowException, models, settings
+from airflow import settings
+from airflow.exceptions import AirflowException
 from airflow.executors.sequential_executor import SequentialExecutor
 from airflow.jobs import LocalTaskJob
-from airflow.models import DAG, TaskInstance as TI
+from airflow.models import DAG, DagBag, TaskInstance as TI
 
 Review comment:
   > I think what would make more sense is to have a separate models/all_models.py which would import all models and this one can be used by resetdb,
   
   I am happy to do that as I am pylinting core code :D
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] codecov-io edited a comment on issue #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589224098
 
 
   # [Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=h1) Report
   > Merging [#7456](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/6502cfa8e61cb57673df327f8d1548c4bef89bcf?src=pr&el=desc) will **decrease** coverage by `0.16%`.
   > The diff coverage is `98.98%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/airflow/pull/7456/graphs/tree.svg?width=650&token=WdLKlKHOAU&height=150&src=pr)](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #7456      +/-   ##
   ==========================================
   - Coverage    86.7%   86.54%   -0.17%     
   ==========================================
     Files         883      883              
     Lines       41635    41650      +15     
   ==========================================
   - Hits        36099    36045      -54     
   - Misses       5536     5605      +69
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [airflow/\_\_init\_\_.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9fX2luaXRfXy5weQ==) | `100% <ø> (ø)` | :arrow_up: |
   | [airflow/executors/dask\_executor.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvZGFza19leGVjdXRvci5weQ==) | `5.88% <0%> (ø)` | :arrow_up: |
   | [airflow/providers/imap/hooks/imap.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvaW1hcC9ob29rcy9pbWFwLnB5) | `94.95% <100%> (+0.04%)` | :arrow_up: |
   | [airflow/models/baseoperator.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9tb2RlbHMvYmFzZW9wZXJhdG9yLnB5) | `96.52% <100%> (ø)` | :arrow_up: |
   | [airflow/providers/dingding/hooks/dingding.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZGluZ2RpbmcvaG9va3MvZGluZ2RpbmcucHk=) | `66.66% <100%> (ø)` | :arrow_up: |
   | [...ow/providers/docker/example\_dags/example\_docker.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZG9ja2VyL2V4YW1wbGVfZGFncy9leGFtcGxlX2RvY2tlci5weQ==) | `100% <100%> (ø)` | :arrow_up: |
   | [.../providers/google/cloud/hooks/kubernetes\_engine.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL2hvb2tzL2t1YmVybmV0ZXNfZW5naW5lLnB5) | `95.94% <100%> (+0.05%)` | :arrow_up: |
   | [airflow/providers/opsgenie/hooks/opsgenie\_alert.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvb3BzZ2VuaWUvaG9va3Mvb3BzZ2VuaWVfYWxlcnQucHk=) | `100% <100%> (ø)` | :arrow_up: |
   | [airflow/executors/local\_executor.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9leGVjdXRvcnMvbG9jYWxfZXhlY3V0b3IucHk=) | `92.48% <100%> (ø)` | :arrow_up: |
   | [.../cloud/operators/cloud\_storage\_transfer\_service.py](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvZ29vZ2xlL2Nsb3VkL29wZXJhdG9ycy9jbG91ZF9zdG9yYWdlX3RyYW5zZmVyX3NlcnZpY2UucHk=) | `95.63% <100%> (ø)` | :arrow_up: |
   | ... and [77 more](https://codecov.io/gh/apache/airflow/pull/7456/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=footer). Last update [6502cfa...e5b541a](https://codecov.io/gh/apache/airflow/pull/7456?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] potiuk edited a comment on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589965811
 
 
   I think we have the easiest migration path for our users to follow:
   ```
   [1.10] py2 (those still on py2) -> [1.10] py3 -> [1.10] py3 + providers (incremental) -> [1.10] py3 + providers + 2.0-compatible DAGs (incremental) -> [2.0] 
   ```
   
   This way while migrating the users will have to deal with one problem at a time and the most difficult part (adapting DAGs) can be mostly automated and can be done incrementally while they are still running 1.10. The fact that this is incremental and that they do not have to migrate everything at once is really appealing for many users I think.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] msb217 opened a new pull request #7456: [AIRFLOW-6817] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
msb217 opened a new pull request #7456: [AIRFLOW-6817]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456
 
 
   ---
   Issue link: [AIRFLOW-6817](https://issues.apache.org/jira/browse/AIRFLOW-6817)
   
   Moved `airflow/__init__.py` imports to respective sub-packages with the exception of the `settings.initialize()` and `integrate_plugins()`
   
   All breeze unit and integration tests are passing locally.
   
   Make sure to mark the boxes below before creating PR: [x]
   
   - [X] Description above provides context of the change
   - [X] Commit message/PR title starts with `[AIRFLOW-NNNN]`. AIRFLOW-NNNN = JIRA ID<sup>*</sup>
   - [X] Unit tests coverage for changes (not needed for documentation changes)
   - [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
   - [X] Relevant documentation is updated including usage instructions.
   - [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
   
   <sup>*</sup> For document-only changes commit message can start with `[AIRFLOW-XXXX]`.
   
   ---
   In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
   Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] nuclearpinguin commented on a change in pull request #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
nuclearpinguin commented on a change in pull request #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#discussion_r382733921
 
 

 ##########
 File path: tests/operators/test_bash.py
 ##########
 @@ -24,7 +24,7 @@
 
 import mock
 
-from airflow import DAG, AirflowException
+from airflow.models.dag import DAG, AirflowException
 
 Review comment:
   ```suggestion
   from airflow.models.dag import DAG
   from airflow.exceptions import AirflowException
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [airflow] msb217 commented on issue #7456: [AIRFLOW-6817] [WIP] - Move airflow __init__.py imports to sub-packages

Posted by GitBox <gi...@apache.org>.
msb217 commented on issue #7456: [AIRFLOW-6817] [WIP]  - Move airflow __init__.py imports to sub-packages
URL: https://github.com/apache/airflow/pull/7456#issuecomment-589744319
 
 
   @potiuk  Great to hear you like it. I have provided an `Updating.md` memo regarding these changes and will squash/rebase my commits before the merge.  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services