You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/12/31 05:53:39 UTC
[GitHub] [airflow] gbonazzoli opened a new issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
gbonazzoli opened a new issue #20603:
URL: https://github.com/apache/airflow/issues/20603
### Apache Airflow version
2.2.3 (latest released)
### What happened
This strange behaviour started with Airflow 2.2.3, Airflow 2.2.2 was OK.
In every DAG that has a PythonOperator declared as:
```
def execute_Get_VMware_Host_Running_The_VM(ds, **kwargs):
try:
__VM_NAME__ = kwargs['dag_run'].conf['VM_NAME']
logging.info(f"Remotely received value of {__VM_NAME__} for key=VM_NAME")
except KeyError:
errorMessage = 'This DAG must be RUN with VM_NAME parameter, es: {"VM_NAME":"lxd-p2c-server"}'
raise KeyError(errorMessage)
with DAG('VMWARE_BACKUP',
description = 'VMWARE_BACKUP',
tags=['system'],
schedule_interval = None,
start_date = datetime(2019, 5, 29, tzinfo=local_tz),
default_args = default_args,
max_active_runs = 1, # maximum number of active runs for this DAG
concurrency = 1, # The Airflow scheduler will run no more than concurrency task instances for your DAG at any given time. number of active runs for this DAG
catchup = False
) as dag:
Get_VMware_Host_Running_The_VM = PythonOperator(
task_id = 'Get_VMware_Host_Running_The_VM',
python_callable = execute_Get_VMware_Host_Running_The_VM,
dag = dag
)
```
we have the following WARNINGS in the log:
```
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_start' or 'logical_date' instead.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'next_ds' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds }}' instead.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'next_ds_nodash' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds_nodash }}' instead.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'next_execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_end' instead.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_ds' from the template is deprecated and will be removed in a future version.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_ds_nodash' from the template is deprecated and will be removed in a future version.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_execution_date' from the template is deprecated and will be removed in a future version.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'prev_execution_date_success' from the template is deprecated and will be removed in a future version. Please use 'prev_data_interval_start_success' instead.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'tomorrow_ds' from the template is deprecated and will be removed in a future version.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'tomorrow_ds_nodash' from the template is deprecated and will be removed in a future version.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'yesterday_ds' from the template is deprecated and will be removed in a future version.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
[2021-12-31, 02:00:02 CET] {warnings.py:109} WARNING - /usr/local/lib/python3.8/dist-packages/airflow/utils/context.py:152: AirflowContextDeprecationWarning: Accessing 'yesterday_ds_nodash' from the template is deprecated and will be removed in a future version.
warnings.warn(_create_deprecation_warning(key, self._deprecation_replacements[key]))
```
### What you expected to happen
I think it is a bug.
### How to reproduce
Run the code posted.
### Operating System
Ubuntu 20.04.3 LTS
### Versions of Apache Airflow Providers
```
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-microsoft-mssql==2.0.1
apache-airflow-providers-microsoft-winrm==2.0.1
apache-airflow-providers-openfaas==2.0.0
apache-airflow-providers-oracle==2.0.1
apache-airflow-providers-samba==3.0.1
apache-airflow-providers-sftp==2.3.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.3.0
```
### Deployment
Virtualenv installation
### Deployment details
Airflow all in one
### Anything else
nope
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1003371066
Hmm #20213 should be part of 2.2.3. Are you sure you have 2.2.3 @gbonazzoli ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1003417894
BTW. It's fixed in `main` - most likely by https://github.com/apache/airflow/pull/20322 - I marked it for 2.2.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
jedcunningham closed issue #20603:
URL: https://github.com/apache/airflow/issues/20603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1003415795
Confirmed. Just a presence of `**kwargs` in Python Callable triggers deprecation warnings for 2.2.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1003359455
Related: https://github.com/apache/airflow/issues/20213
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] gbonazzoli commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
gbonazzoli commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1003412657
@potiuk, @kaxil, unfortunately not.
```
root@new-airflow:~# airflow version
2.2.3
```
From the UI:
```
Version: v2.2.3
Git Version: .release:2.2.3+06c82e17e9d7ff1bf261357e84c6013ccdb3c241
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1003380991
Yeah, this was the last item that was fixed for 2.2.3. Can you double-check the Airflow version please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1003309205
I believe we know it already and we fixed it but I will let @uranusjr confirm when he is back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1003389031
Yeah @gbonazzoli - maybe you have one of the RCs :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1004138339
Specifically https://github.com/apache/airflow/blob/97261c642cbf07db91d252cf6b0b7ff184cd64c6/airflow/utils/context.py#L218-L244
Initially (including when 2.2.3 was implemented) the intention was to make this the expected behaviour and discourage `**kwargs`, but ultimately we have been recommending the `**kwargs` pattern too much, and using `lazy-object-proxy` (which is not a perfect solution and brings other problems) to cover this specific case is considered worthy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1040618118
Closing this as the fix is already cherry-pick for 2.2.4 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1040618118
Closing this as the fix is already cherry-pick for 2.2.4 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham closed issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
jedcunningham closed issue #20603:
URL: https://github.com/apache/airflow/issues/20603
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on issue #20603: Airflow 2.2.3 : AirflowContextDeprecationWarning executing a DAG with PythonOperator that accesses the context
Posted by GitBox <gi...@apache.org>.
kaxil commented on issue #20603:
URL: https://github.com/apache/airflow/issues/20603#issuecomment-1003416278
Thanks for confirming @gbonazzoli , we will fix it in 2.2.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org