You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/03 15:28:49 UTC
[GitHub] [airflow] ashb opened a new pull request #12069: Deprecate adding Operators and Sensors via plugins
ashb opened a new pull request #12069:
URL: https://github.com/apache/airflow/pull/12069
This adds two imports -- one at registration time:
```
/home/ash/airflow/plugins/example.py:27: FutureWarning: Registering operator or sensors in plugins is deprecated -- these should be treated like 'plain' python modules, and imported normally in DAGs. Airflow 2.0 has removed the ability to register these types in plugins
class AstroLinksPlugin(AirflowPlugin):
```
(The source/file shown here is the `class` line of the defining plugin.)
And a second at import time:
```
/home/ash/airflow/dags/foo.py:1: FutureWarning: Importing 'MyOperator' from under 'airflow.operators.*' has been deprecated and should be directly imported as 'example.MyOperator' instead. Support for importing from within the airflow namespace for plugins will be dropped entirely in Airflow 2.0.
from airflow.operators.astronomer_menu_links import MyOperator
```
(The source here is where the import is used - i.e. in the DAG)
Closes #9500
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb merged pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
ashb merged pull request #12069:
URL: https://github.com/apache/airflow/pull/12069
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12069:
URL: https://github.com/apache/airflow/pull/12069#issuecomment-721975160
[The Workflow run](https://github.com/apache/airflow/actions/runs/346302147) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12069:
URL: https://github.com/apache/airflow/pull/12069#issuecomment-721853101
D'oh. Using pep562 like this is barfing if the plugins aren't valid module names, such as `plugin-b.py`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb edited a comment on pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
ashb edited a comment on pull request #12069:
URL: https://github.com/apache/airflow/pull/12069#issuecomment-721853101
~D'oh. Using pep562 like this is barfing if the plugins aren't valid module names, such as `plugin-b.py`.~
No, I just did a stupid :) Hopefully an easy fix.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #12069:
URL: https://github.com/apache/airflow/pull/12069#discussion_r516796551
##########
File path: airflow/plugins_manager.py
##########
@@ -40,11 +43,29 @@
import_errors = {}
+PY37 = sys.version_info >= (3, 7)
+
+
class AirflowPluginException(Exception):
pass
-class AirflowPlugin(object):
+class _MetaPluginClass(type):
+ def __new__(cls, name, bases, props):
+ if props.get('operators', []) or props.get('sensors', []):
+ warnings.warn(
+ "Registering operator or sensors in plugins is deprecated -- these should be treated like "
+ "'plain' python modules, and imported normally in DAGs. Airflow 2.0 has removed the ability "
+ "to register these types in plugins",
Review comment:
```suggestion
"Registering operators or sensors in plugins is deprecated -- these should be treated like "
"'plain' python modules, and imported normally in DAGs. Airflow 2.0 has removed the ability "
"to register these types in plugins",
```
Looks good for now, maybe we can link it to Module Management page once we backport that doc PR to v1-10-test https://airflow.readthedocs.io/en/latest/modules_management.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12069:
URL: https://github.com/apache/airflow/pull/12069#discussion_r516834658
##########
File path: airflow/plugins_manager.py
##########
@@ -208,9 +287,9 @@ def make_module(name, objects):
for p in plugins:
operators_modules.append(
- make_module('airflow.operators.' + p.name, p.operators + p.sensors))
+ make_deprecated_module('airflow.operators.' + p.name, p.operators + p.sensors, p))
sensors_modules.append(
- make_module('airflow.sensors.' + p.name, p.sensors)
+ make_deprecated_module('airflow.sensors.' + p.name, p.sensors, p)
)
hooks_modules.append(make_module('airflow.hooks.' + p.name, p.hooks))
Review comment:
Separate ticket for hooks (cos it maybe makes sense to register them, if not import them from hooks) #9506 and #9507
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #12069:
URL: https://github.com/apache/airflow/pull/12069#discussion_r516799702
##########
File path: airflow/plugins_manager.py
##########
@@ -208,9 +287,9 @@ def make_module(name, objects):
for p in plugins:
operators_modules.append(
- make_module('airflow.operators.' + p.name, p.operators + p.sensors))
+ make_deprecated_module('airflow.operators.' + p.name, p.operators + p.sensors, p))
sensors_modules.append(
- make_module('airflow.sensors.' + p.name, p.sensors)
+ make_deprecated_module('airflow.sensors.' + p.name, p.sensors, p)
)
hooks_modules.append(make_module('airflow.hooks.' + p.name, p.hooks))
Review comment:
Should we deprecate adding Hooks and Executors too??
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12069:
URL: https://github.com/apache/airflow/pull/12069#discussion_r516818592
##########
File path: airflow/plugins_manager.py
##########
@@ -40,11 +43,29 @@
import_errors = {}
+PY37 = sys.version_info >= (3, 7)
+
+
class AirflowPluginException(Exception):
pass
-class AirflowPlugin(object):
+class _MetaPluginClass(type):
+ def __new__(cls, name, bases, props):
+ if props.get('operators', []) or props.get('sensors', []):
+ warnings.warn(
+ "Registering operator or sensors in plugins is deprecated -- these should be treated like "
+ "'plain' python modules, and imported normally in DAGs. Airflow 2.0 has removed the ability "
+ "to register these types in plugins",
Review comment:
http://airflow.apache.org/docs/stable/howto/custom-operator.html exists on 1.10 already, and is a page all about this, so I'll use that instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #12069:
URL: https://github.com/apache/airflow/pull/12069#discussion_r516832472
##########
File path: airflow/plugins_manager.py
##########
@@ -40,11 +43,29 @@
import_errors = {}
+PY37 = sys.version_info >= (3, 7)
+
+
class AirflowPluginException(Exception):
pass
-class AirflowPlugin(object):
+class _MetaPluginClass(type):
+ def __new__(cls, name, bases, props):
+ if props.get('operators', []) or props.get('sensors', []):
+ warnings.warn(
+ "Registering operator or sensors in plugins is deprecated -- these should be treated like "
+ "'plain' python modules, and imported normally in DAGs. Airflow 2.0 has removed the ability "
+ "to register these types in plugins",
Review comment:
👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #12069:
URL: https://github.com/apache/airflow/pull/12069#discussion_r516845230
##########
File path: airflow/plugins_manager.py
##########
@@ -40,11 +43,29 @@
import_errors = {}
+PY37 = sys.version_info >= (3, 7)
+
+
class AirflowPluginException(Exception):
pass
-class AirflowPlugin(object):
+class _MetaPluginClass(type):
+ def __new__(cls, name, bases, props):
+ if props.get('operators', []) or props.get('sensors', []):
+ warnings.warn(
+ "Registering operator or sensors in plugins is deprecated -- these should be treated like "
+ "'plain' python modules, and imported normally in DAGs. Airflow 2.0 has removed the ability "
+ "to register these types in plugins",
Review comment:
Done, check out new message
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #12069: Deprecate adding Operators and Sensors via plugins
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #12069:
URL: https://github.com/apache/airflow/pull/12069#issuecomment-722338578
Just random failures left. Merging now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org