You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/04/02 19:24:24 UTC

[GitHub] [airflow] potiuk opened a new pull request, #30434: Increase timeout of cli connection add

potiuk opened a new pull request, #30434:
URL: https://github.com/apache/airflow/pull/30434

   Recently we had a number of flaky failures for cli connection add test. The problem is that those tests initialize providers manager which loads all available hooks and imports their dependencies. This might take a lot of time when the test run in parallel and might exceed the timeout of 60 seconds (usually the test fall around importing Snowflake provider which is towards the end of all provider list, 120 seconds should be enough.
   
   <!--
   Thank you for contributing! Please make sure that your code changes
   are covered with tests. And in case of new features or big changes
   remember to adjust the documentation.
   
   Feel free to ping committers for the review!
   
   In case of an existing issue, reference it using one of the following:
   
   closes: #ISSUE
   related: #ISSUE
   
   How to write a good git commit message:
   http://chris.beams.io/posts/git-commit/
   -->
   
   ---
   **^ Add meaningful description above**
   
   Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
   In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] pierrejeambrun merged pull request #30434: Increase timeout of cli connection add test

Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun merged PR #30434:
URL: https://github.com/apache/airflow/pull/30434


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on pull request #30434: Increase timeout of cli connection add test

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #30434:
URL: https://github.com/apache/airflow/pull/30434#issuecomment-1493420867

   This is the example stacktrace:
   
   https://github.com/apache/airflow/actions/runs/4590022900/jobs/8105387439?pr=30308
   
   Pytest rewrite seems to contribute a lot to the slowness of the test:
   
   ```
   tests/cli/commands/test_connection_command.py:550: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   airflow/utils/cli.py:110: in wrapper
       return f(*args, **kwargs)
   airflow/cli/commands/connection_command.py:212: in connections_add
       if has_type and args.conn_type not in _get_connection_types():
   airflow/cli/commands/connection_command.py:144: in _get_connection_types
       for connection_type, provider_info in providers_manager.hooks.items():
   /usr/local/lib/python3.9/_collections_abc.py:851: in __iter__
       yield (key, self._mapping[key])
   airflow/providers_manager.py:110: in __getitem__
       value = value()
   airflow/providers_manager.py:796: in _import_hook
       hook_class = _sanity_check(package_name, hook_class_name, provider_info)
   airflow/providers_manager.py:284: in _sanity_check
       imported_class = import_string(class_name)
   airflow/utils/module_loading.py:36: in import_string
       module = import_module(module_path)
   /usr/local/lib/python3.9/importlib/__init__.py:127: in import_module
       return _bootstrap._gcd_import(name[level:], package, level)
   <frozen importlib._bootstrap>:1030: in _gcd_import
       ???
   <frozen importlib._bootstrap>:1007: in _find_and_load
       ???
   <frozen importlib._bootstrap>:986: in _find_and_load_unlocked
       ???
   <frozen importlib._bootstrap>:680: in _load_unlocked
       ???
   /usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
       exec(co, module.__dict__)
   airflow/providers/snowflake/hooks/snowflake.py:29: in <module>
       from snowflake import connector
   <frozen importlib._bootstrap>:1007: in _find_and_load
       ???
   <frozen importlib._bootstrap>:986: in _find_and_load_unlocked
       ???
   <frozen importlib._bootstrap>:680: in _load_unlocked
       ???
   /usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
       exec(co, module.__dict__)
   /usr/local/lib/python3.9/site-packages/snowflake/connector/azure_storage_client.py:18: in <module>
       from .encryption_util import EncryptionMetadata
   <frozen importlib._bootstrap>:1007: in _find_and_load
       ???
   <frozen importlib._bootstrap>:986: in _find_and_load_unlocked
       ???
   <frozen importlib._bootstrap>:680: in _load_unlocked
       ???
   /usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
       exec(co, module.__dict__)
   /usr/local/lib/python3.9/site-packages/snowflake/connector/encryption_util.py:15: in <module>
       from Cryptodome.Cipher import AES
   <frozen importlib._bootstrap>:1007: in _find_and_load
       ???
   <frozen importlib._bootstrap>:986: in _find_and_load_unlocked
       ???
   <frozen importlib._bootstrap>:680: in _load_unlocked
       ???
   /usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
       exec(co, module.__dict__)
   /usr/local/lib/python3.9/site-packages/Cryptodome/Cipher/__init__.py:35: in <module>
       from Cryptodome.Cipher._mode_siv import _create_siv_cipher
   <frozen importlib._bootstrap>:1007: in _find_and_load
       ???
   <frozen importlib._bootstrap>:986: in _find_and_load_unlocked
       ???
   <frozen importlib._bootstrap>:680: in _load_unlocked
       ???
   /usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
       exec(co, module.__dict__)
   /usr/local/lib/python3.9/site-packages/Cryptodome/Cipher/_mode_siv.py:44: in <module>
       from Cryptodome.Protocol.KDF import _S2V
   <frozen importlib._bootstrap>:1007: in _find_and_load
       ???
   <frozen importlib._bootstrap>:986: in _find_and_load_unlocked
       ???
   <frozen importlib._bootstrap>:680: in _load_unlocked
       ???
   /usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
       exec(co, module.__dict__)
   /usr/local/lib/python3.9/site-packages/Cryptodome/Protocol/KDF.py:47: in <module>
       _raw_scrypt_lib = load_pycryptodome_raw_lib("Cryptodome.Protocol._scrypt",
   /usr/local/lib/python3.9/site-packages/Cryptodome/Util/_raw_api.py:306: in load_pycryptodome_raw_lib
       return load_lib(full_name, cdecl)
   /usr/local/lib/python3.9/site-packages/Cryptodome/Util/_raw_api.py:100: in load_lib
       ffi.cdef(cdecl)
   /usr/local/lib/python3.9/site-packages/cffi/api.py:112: in cdef
       self._cdef(csource, override=override, packed=packed, pack=pack)
   /usr/local/lib/python3.9/site-packages/cffi/api.py:126: in _cdef
       self._parser.parse(csource, override=override, **options)
   /usr/local/lib/python3.9/site-packages/cffi/cparser.py:389: in parse
       self._internal_parse(csource)
   /usr/local/lib/python3.9/site-packages/cffi/cparser.py:394: in _internal_parse
       ast, macros, csource = self._parse(csource)
   /usr/local/lib/python3.9/site-packages/cffi/cparser.py:336: in _parse
       ast = _get_parser().parse(fullcsource)
   /usr/local/lib/python3.9/site-packages/pycparser/c_parser.py:147: in parse
       return self.cparser.parse(
   /usr/local/lib/python3.9/site-packages/pycparser/ply/yacc.py:331: in parse
       return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
   /usr/local/lib/python3.9/site-packages/pycparser/ply/yacc.py:1061: in parseopt_notrack
       lookahead = get_token()     # Get the next token
   /usr/local/lib/python3.9/site-packages/pycparser/c_lexer.py:76: in token
       self.last_token = self.lexer.token()
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org