You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Taragolis (via GitHub)" <gi...@apache.org> on 2024/03/31 10:26:31 UTC
[I] Follow zero internal warning in unit tests [airflow]
Taragolis opened a new issue, #38642:
URL: https://github.com/apache/airflow/issues/38642
### Body
This is follow up task after https://github.com/apache/airflow/pull/38504, we have quite a few tests which raise one of the internal deprecation warning:
- `airflow.exceptions.AirflowProviderDeprecationWarning`
- `airflow.exceptions.RemovedInAirflow3Warning`
- `airflow.utils.context.AirflowContextDeprecationWarning`
All current exclusions which should be resolved stored into the [`tests/deprecations_ignore.yml`](https://github.com/apache/airflow/blob/main/tests/deprecations_ignore.yml) in format test node id: `{relative path to the test}::{ClassName}::{test_case}` or `{relative path to the test}::{test_case}` without parametrised part
How to resolve specific test suite
---------------------------------
Remove /commented out specific line in exclusion list a have a look on error by run removed node id,
```console
root@e68ed83041ce:/opt/airflow# pytest tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3
======================================================================= test session starts ========================================================================
platform linux -- Python 3.8.19, pytest-7.4.4, pluggy-1.4.0 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /opt/airflow
configfile: pyproject.toml
plugins: custom-exit-code-0.3.0, instafail-0.5.0, rerunfailures-14.0, asyncio-0.23.6, cov-5.0.0, xdist-3.5.0, timeouts-1.2.1, anyio-4.3.0, icdiff-0.9, mock-3.14.0, requests-mock-1.11.0, time-machine-2.14.1
asyncio: mode=strict
setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
collected 1 item
tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3 FAILED [100%]
============================================================================= FAILURES =============================================================================
___________________________________________________________ TestPythonVirtualenvDecorator.test_python_3 ____________________________________________________________
self = <tests.decorators.test_python_virtualenv.TestPythonVirtualenvDecorator object at 0xffff855dd760>
dag_maker = <tests.conftest.dag_maker.<locals>.DagFactory object at 0xffff8593da30>
def test_python_3(self, dag_maker):
@task.virtualenv(python_version=3, use_dill=False, requirements=["dill"])
def f():
import sys
print(sys.version)
try:
{}.iteritems()
except AttributeError:
return
raise Exception
with dag_maker():
> ret = f()
tests/decorators/test_python_virtualenv.py:171:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
airflow/decorators/base.py:372: in __call__
op = self.operator_class(
airflow/models/baseoperator.py:483: in apply_defaults
result = func(self, **kwargs, default_args=default_args)
airflow/models/baseoperator.py:483: in apply_defaults
result = func(self, **kwargs, default_args=default_args)
airflow/decorators/python.py:52: in __init__
super().__init__(
airflow/models/baseoperator.py:483: in apply_defaults
result = func(self, **kwargs, default_args=default_args)
airflow/decorators/base.py:257: in __init__
super().__init__(task_id=task_id, **kwargs_to_upstream, **kwargs)
airflow/models/baseoperator.py:483: in apply_defaults
result = func(self, **kwargs, default_args=default_args)
airflow/operators/python.py:620: in __init__
warnings.warn(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <airflow.utils.decorators._autostacklevel_warn object at 0xffff85ebb820>
message = 'Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.'
category = <class 'airflow.exceptions.RemovedInAirflow3Warning'>, stacklevel = 2, source = None
def warn(self, message, category=None, stacklevel=1, source=None):
> self.warnings.warn(message, category, stacklevel + 2, source)
E airflow.exceptions.RemovedInAirflow3Warning: Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.
airflow/utils/decorators.py:108: RemovedInAirflow3Warning
---------------------------------------------------------------------- Captured stdout setup -----------------------------------------------------------------------
========================= AIRFLOW ==========================
Home of the user: /root
Airflow home /root/airflow
Skipping initializing of the DB as it was initialized already.
You can re-initialize the database by adding --with-db-init flag when running tests.
[2024-03-31T10:06:44.945+0000] {dagbag.py:540} INFO - Filling up the DagBag from /dev/null
------------------------------------------------------------------------ Captured log setup ------------------------------------------------------------------------
INFO airflow.models.dagbag.DagBag:dagbag.py:540 Filling up the DagBag from /dev/null
===================================================================== short test summary info ======================================================================
FAILED tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3 - airflow.exceptions.RemovedInAirflow3Warning: Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.
```
After that need to determine which kind of a problem here
**Option 1**: It might be test deprecation functional without explicit usage of [`pytest.warn`](https://docs.pytest.org/en/latest/how-to/capture-warnings.html#warns). This case is pretty simple and required to add `pytest.warn` context manager wit appropriate matcher.
**Option 2**: Use deprecated functional in tests, time moves on and not all test updated. In this case need to rewrite test to avoid use deprecated part
**Option 3**: Use deprecated functional in tests utilities and fixtures, the same as option 2, however it might spread across multiple tests suits, replace/add functional to use not deprecated functional it this utilities/fixtures. This might required to use `full test needed` label in pr, If you not a Commiter or PMC please ask to add this label.
**Option 4**: Incorrect depreciation in code base. This one could be cause if one part of functional deprecated in code base, but the other still use it. There is no a universal solution for this situation, it might required to avoid use deprecated functional in codebase or even undeprecate.
**Other cases**: It might be other cases, feel free to share you experience with non listed above in this issue or into the Slack
> [!TIP]
> Feel free to ask a suggestion/helps in slack channels `#contributors` or `#new-contributors`
> [!NOTE]
> There main target of this task it resolve deprecation warnings, but not hide all of them under [`pytest.warn`](https://docs.pytest.org/en/latest/how-to/capture-warnings.html#warns), use this context manager, as well as `warnings.catch_warnings` context manager only for test suites around deprecated functional
> [!WARNING]
> Please do not mixin changes in code base and only tests into the same PR. Better to split it into the separate PRs.
### Committer
- [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow zero internal warning in unit tests [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #38642: Follow zero internal warning in unit tests
URL: https://github.com/apache/airflow/issues/38642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow zero internal warning in unit tests [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #38642:
URL: https://github.com/apache/airflow/issues/38642#issuecomment-2679751327
I assume we cleaned most of the named deprecations. in context of the names I _think_:
- airflow.exceptions.AirflowProviderDeprecationWarning - we attempted to clean deprecations last month - there might be some left. I assume some careful cleaning is possible but depends on the provider.
- airflow.exceptions.RemovedInAirflow3Warning - Almost all should be gone and the leftovers were mostly remaining for TaskSDK changes to be completed. Might be blocked to be removed. I think the leftovers are not easy.
- airflow.utils.context.AirflowContextDeprecationWarning - I assume this is only existing in a local flavor in OpenLineage provider - all is cleaned already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow zero internal warning in unit tests [airflow]
Posted by "tsafacjo (via GitHub)" <gi...@apache.org>.
tsafacjo commented on issue #38642:
URL: https://github.com/apache/airflow/issues/38642#issuecomment-2316283628
can I pick it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Follow zero internal warning in unit tests [airflow]
Posted by "cocameron (via GitHub)" <gi...@apache.org>.
cocameron commented on issue #38642:
URL: https://github.com/apache/airflow/issues/38642#issuecomment-2679671614
I'm looking to pick this up as a good first issue. From poking around it, it looks like everything referenced in `deprecations_ignore.yml` either does not exist or does not generate a forbidden warning any longer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org