You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/23 13:24:20 UTC
[GitHub] [airflow] benwatsonnandos opened a new issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
benwatsonnandos opened a new issue #14955:
URL: https://github.com/apache/airflow/issues/14955
**Apache Airflow version**: 2.0.1
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): N/A
**Environment**: Ubuntu 20.04, Python 3.9
- **Cloud provider or hardware configuration**: AWS EC2 (single instance)
- **OS** (e.g. from /etc/os-release): Ubuntu 20.04
- **Kernel** (e.g. `uname -a`): 5.4.0-1038-aws #40-Ubuntu SMP Fri Feb 5 23:50:40 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
- **Install tools**:
- **Others**: Python 3.9 is a `venv`.
**What happened**:
I'm upgrading my old Airflow 1.10.5 and Python 2.7 environment to Airflow 2.0.1 and Python 3.9. I have a file `utils.py` in `~/airflow/plugins` (referenced in `airflow.cfg::plugins_folder` that contains a macro that is used by multiple DAGs (file contents simplified for brevity):
```python
from airflow.plugins_manager import AirflowPlugin
def ret_one():
return 1
class AirflowPluginsTest(AirflowPlugin):
name = "ret_one"
macros = [ret_one]
```
When restarting Airflow then I can see `ret_one` as a macro in `Admin -> Plugins` in the UI, but using `from airflow.macros import ret_one` in a DAG or ipython results in a missing module exception and the macro can't be used. This worked in Airflow 1.10.5.
**What you expected to happen**:
The macro should be found and should be available to DAGs.
**How to reproduce it**:
Run Airflow 2.0.1 on Python 3.9. Create a custom macro (like the above), and try to import it in a DAG/ipython.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marklit edited a comment on issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
Posted by GitBox <gi...@apache.org>.
marklit edited a comment on issue #14955:
URL: https://github.com/apache/airflow/issues/14955#issuecomment-805671669
For reasons I'm unfamiliar with, it looks like the ability to import via `from airflow.macros import x` was removed a few months ago (as per 5e8b537b8). The documentation for anyone interested can be found here: https://github.com/apache/airflow/blob/2d1365444c206cbd6917a5b007222c16a05e9882/UPDATING.md#adding-operators-and-sensors-via-plugins-is-no-longer-supported
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] benwatsonnandos commented on issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
Posted by GitBox <gi...@apache.org>.
benwatsonnandos commented on issue #14955:
URL: https://github.com/apache/airflow/issues/14955#issuecomment-808370824
OK thanks for the info, I'll use macros this way then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14955:
URL: https://github.com/apache/airflow/issues/14955#issuecomment-805416772
Can you test it with Python 3.8 too please, Airflow currently "officially" does not support Python 3.9:
https://github.com/apache/airflow#requirements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marklit commented on issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
Posted by GitBox <gi...@apache.org>.
marklit commented on issue #14955:
URL: https://github.com/apache/airflow/issues/14955#issuecomment-805671669
For reasons I'm unfamiliar with, it looks like the ability to import via `from airflow.macros import x` was removed a few months ago (as per 5e8b537b8). Any documentation on what the new practice is for Airflow 2 would be greatly appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #14955:
URL: https://github.com/apache/airflow/issues/14955#issuecomment-806673423
> For reasons I'm unfamiliar with, it looks like the ability to import via `from airflow.macros import x` was removed a few months ago (as per [5e8b537](https://github.com/apache/airflow/commit/5e8b537b85ce6aa355966df08df5c1846adb4cd6)). The documentation for anyone interested can be found here: https://github.com/apache/airflow/blob/2d1365444c206cbd6917a5b007222c16a05e9882/UPDATING.md#adding-operators-and-sensors-via-plugins-is-no-longer-supported
I will take a look at it but we haven't removed ability to import macros -- it was ability to import operators and sensors. Those can be normal python modules. Plugin mechanism is not needed for it, hence it was removed. We want to keep plugin mechanism to add only UI related elements in future. So if you your python package or python module on syspath, you can import it directly via `from my_custom_module import my_class` instead of importing from plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marklit commented on issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
Posted by GitBox <gi...@apache.org>.
marklit commented on issue #14955:
URL: https://github.com/apache/airflow/issues/14955#issuecomment-805565049
I've managed to find an ugly way to import macros in Airflow 2.0.1 on Python 3.9. It's not the original `from airflow.macros import x` method from Airflow 1.x but it is working.
```
def get_macro(name):
from airflow import plugins_manager
plugins_manager.integrate_macros_plugins()
for x in plugins_manager.plugins:
if x.name == name:
return x.macros[0]
raise Exception('Unable to find macro "%s"' % name)
_exec = get_macro('_exec')
_exec('pwd') # returns: '/home/ubuntu/airflow/dags\n'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] marklit edited a comment on issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
Posted by GitBox <gi...@apache.org>.
marklit edited a comment on issue #14955:
URL: https://github.com/apache/airflow/issues/14955#issuecomment-805565049
I've managed to find an ugly way to import macros in Airflow 2.0.1 on Python 3.9. It's not the original `from airflow.macros import x` method from Airflow 1.x but it is working.
```python
def get_macro(name):
from airflow import plugins_manager
plugins_manager.integrate_macros_plugins()
for x in plugins_manager.plugins:
if x.name == name:
return x.macros[0]
raise Exception('Unable to find macro "%s"' % name)
_exec = get_macro('_exec')
_exec('pwd') # returns: '/home/ubuntu/airflow/dags\n'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] benwatsonnandos closed issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
Posted by GitBox <gi...@apache.org>.
benwatsonnandos closed issue #14955:
URL: https://github.com/apache/airflow/issues/14955
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #14955:
URL: https://github.com/apache/airflow/issues/14955#issuecomment-808342481
Macros are only designed for use in Jinja templates, not for calling directly from Python code as you are trying.
In your case you to import it directly from the plugin: something like `from utils import ret_one`
"Working as designed" in otherwords.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14955: Custom macros not working on Airflow 2.0.1 Python 3.9
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14955:
URL: https://github.com/apache/airflow/issues/14955#issuecomment-804899947
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org