You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Taragolis (via GitHub)" <gi...@apache.org> on 2024/03/31 10:26:31 UTC

[I] Follow zero internal warning in unit tests [airflow]

Taragolis opened a new issue, #38642:
URL: https://github.com/apache/airflow/issues/38642

   ### Body
   
   This is follow up task after https://github.com/apache/airflow/pull/38504, we have quite a few tests which raise one of the internal deprecation warning:
   - `airflow.exceptions.AirflowProviderDeprecationWarning`
   - `airflow.exceptions.RemovedInAirflow3Warning`
   - `airflow.utils.context.AirflowContextDeprecationWarning`
   
   All current exclusions which should be resolved stored into the [`tests/deprecations_ignore.yml`](https://github.com/apache/airflow/blob/main/tests/deprecations_ignore.yml) in format test node id: `{relative path to the test}::{ClassName}::{test_case}` or  `{relative path to the test}::{test_case}` without parametrised part
   
   How to resolve specific test suite
   ---------------------------------
   
   Remove /commented out specific line in exclusion list a have a look on error by run removed node id, 
   
   ```console
   root@e68ed83041ce:/opt/airflow# pytest tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3
   ======================================================================= test session starts ========================================================================
   platform linux -- Python 3.8.19, pytest-7.4.4, pluggy-1.4.0 -- /usr/local/bin/python
   cachedir: .pytest_cache
   rootdir: /opt/airflow
   configfile: pyproject.toml
   plugins: custom-exit-code-0.3.0, instafail-0.5.0, rerunfailures-14.0, asyncio-0.23.6, cov-5.0.0, xdist-3.5.0, timeouts-1.2.1, anyio-4.3.0, icdiff-0.9, mock-3.14.0, requests-mock-1.11.0, time-machine-2.14.1
   asyncio: mode=strict
   setup timeout: 0.0s, execution timeout: 0.0s, teardown timeout: 0.0s
   collected 1 item                                                                                                                                                   
   
   tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3 FAILED                                                              [100%]
   
   ============================================================================= FAILURES =============================================================================
   ___________________________________________________________ TestPythonVirtualenvDecorator.test_python_3 ____________________________________________________________
   
   self = <tests.decorators.test_python_virtualenv.TestPythonVirtualenvDecorator object at 0xffff855dd760>
   dag_maker = <tests.conftest.dag_maker.<locals>.DagFactory object at 0xffff8593da30>
   
       def test_python_3(self, dag_maker):
           @task.virtualenv(python_version=3, use_dill=False, requirements=["dill"])
           def f():
               import sys
       
               print(sys.version)
               try:
                   {}.iteritems()
               except AttributeError:
                   return
               raise Exception
       
           with dag_maker():
   >           ret = f()
   
   tests/decorators/test_python_virtualenv.py:171: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   airflow/decorators/base.py:372: in __call__
       op = self.operator_class(
   airflow/models/baseoperator.py:483: in apply_defaults
       result = func(self, **kwargs, default_args=default_args)
   airflow/models/baseoperator.py:483: in apply_defaults
       result = func(self, **kwargs, default_args=default_args)
   airflow/decorators/python.py:52: in __init__
       super().__init__(
   airflow/models/baseoperator.py:483: in apply_defaults
       result = func(self, **kwargs, default_args=default_args)
   airflow/decorators/base.py:257: in __init__
       super().__init__(task_id=task_id, **kwargs_to_upstream, **kwargs)
   airflow/models/baseoperator.py:483: in apply_defaults
       result = func(self, **kwargs, default_args=default_args)
   airflow/operators/python.py:620: in __init__
       warnings.warn(
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   
   self = <airflow.utils.decorators._autostacklevel_warn object at 0xffff85ebb820>
   message = 'Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.'
   category = <class 'airflow.exceptions.RemovedInAirflow3Warning'>, stacklevel = 2, source = None
   
       def warn(self, message, category=None, stacklevel=1, source=None):
   >       self.warnings.warn(message, category, stacklevel + 2, source)
   E       airflow.exceptions.RemovedInAirflow3Warning: Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.
   
   airflow/utils/decorators.py:108: RemovedInAirflow3Warning
   ---------------------------------------------------------------------- Captured stdout setup -----------------------------------------------------------------------
   ========================= AIRFLOW ==========================
   Home of the user: /root
   Airflow home /root/airflow
   Skipping initializing of the DB as it was initialized already.
   You can re-initialize the database by adding --with-db-init flag when running tests.
   [2024-03-31T10:06:44.945+0000] {dagbag.py:540} INFO - Filling up the DagBag from /dev/null
   ------------------------------------------------------------------------ Captured log setup ------------------------------------------------------------------------
   INFO     airflow.models.dagbag.DagBag:dagbag.py:540 Filling up the DagBag from /dev/null
   ===================================================================== short test summary info ======================================================================
   FAILED tests/decorators/test_python_virtualenv.py::TestPythonVirtualenvDecorator::test_python_3 - airflow.exceptions.RemovedInAirflow3Warning: Passing non-string types (e.g. int or float) as python_version is deprecated. Please use string value instead.
   ```
   
   After that need to determine which kind of a problem here
   
   **Option 1**: It might be test deprecation functional without explicit usage of [`pytest.warn`](https://docs.pytest.org/en/latest/how-to/capture-warnings.html#warns). This case is pretty simple and required to add `pytest.warn` context manager wit appropriate matcher.
   
   **Option 2**: Use deprecated functional in tests, time moves on and not all test updated. In this case need to rewrite test to avoid use deprecated part
   
   **Option 3**: Use deprecated functional in tests utilities and fixtures, the same as option 2, however it might spread across multiple tests suits, replace/add functional to use not deprecated functional it this utilities/fixtures. This might required to use `full test needed` label in pr, If you not a Commiter or PMC please ask to add this label.
   
   **Option 4**: Incorrect depreciation in code base. This one could be cause if one part of functional deprecated in code base, but the other still use it. There is no a universal solution for this situation, it might required to avoid use deprecated functional in codebase or even undeprecate. 
   
   **Other cases**: It might be other cases, feel free to share you experience with non listed above in this issue or into the Slack
   
   > [!TIP]
   > Feel free to ask a suggestion/helps in slack channels `#contributors` or `#new-contributors`
   
   > [!NOTE]
   > There main target of this task it resolve deprecation warnings, but not hide all of them under [`pytest.warn`](https://docs.pytest.org/en/latest/how-to/capture-warnings.html#warns), use this context manager, as well as `warnings.catch_warnings` context manager only for test suites around deprecated functional
   
   > [!WARNING]
   > Please do not mixin changes in code base and only tests into the same PR. Better to split it into the separate PRs. 
   
   
   ### Committer
   
   - [X] I acknowledge that I am a maintainer/committer of the Apache Airflow project.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Follow zero internal warning in unit tests [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal closed issue #38642: Follow zero internal warning in unit tests
URL: https://github.com/apache/airflow/issues/38642


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Follow zero internal warning in unit tests [airflow]

Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #38642:
URL: https://github.com/apache/airflow/issues/38642#issuecomment-2679751327

   I assume we cleaned most of the named deprecations. in context of the names I _think_:
   - airflow.exceptions.AirflowProviderDeprecationWarning - we attempted to clean deprecations last month - there might be some left. I assume some careful cleaning is possible but depends on the provider.
   - airflow.exceptions.RemovedInAirflow3Warning - Almost all should be gone and the leftovers were mostly remaining for TaskSDK changes to be completed. Might be blocked to be removed. I think the leftovers are not easy.
   - airflow.utils.context.AirflowContextDeprecationWarning - I assume this is only existing in a local flavor in OpenLineage provider - all is cleaned already.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Follow zero internal warning in unit tests [airflow]

Posted by "tsafacjo (via GitHub)" <gi...@apache.org>.
tsafacjo commented on issue #38642:
URL: https://github.com/apache/airflow/issues/38642#issuecomment-2316283628

   can I pick it ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Follow zero internal warning in unit tests [airflow]

Posted by "cocameron (via GitHub)" <gi...@apache.org>.
cocameron commented on issue #38642:
URL: https://github.com/apache/airflow/issues/38642#issuecomment-2679671614

   I'm looking to pick this up as a good first issue. From poking around it, it looks like everything referenced in `deprecations_ignore.yml` either does not exist or does not generate a forbidden warning any longer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org