You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/04/02 19:24:24 UTC
[GitHub] [airflow] potiuk opened a new pull request, #30434: Increase timeout of cli connection add
potiuk opened a new pull request, #30434:
URL: https://github.com/apache/airflow/pull/30434
Recently we had a number of flaky failures for cli connection add test. The problem is that those tests initialize providers manager which loads all available hooks and imports their dependencies. This might take a lot of time when the test run in parallel and might exceed the timeout of 60 seconds (usually the test fall around importing Snowflake provider which is towards the end of all provider list, 120 seconds should be enough.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] pierrejeambrun merged pull request #30434: Increase timeout of cli connection add test
Posted by "pierrejeambrun (via GitHub)" <gi...@apache.org>.
pierrejeambrun merged PR #30434:
URL: https://github.com/apache/airflow/pull/30434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #30434: Increase timeout of cli connection add test
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #30434:
URL: https://github.com/apache/airflow/pull/30434#issuecomment-1493420867
This is the example stacktrace:
https://github.com/apache/airflow/actions/runs/4590022900/jobs/8105387439?pr=30308
Pytest rewrite seems to contribute a lot to the slowness of the test:
```
tests/cli/commands/test_connection_command.py:550:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
airflow/utils/cli.py:110: in wrapper
return f(*args, **kwargs)
airflow/cli/commands/connection_command.py:212: in connections_add
if has_type and args.conn_type not in _get_connection_types():
airflow/cli/commands/connection_command.py:144: in _get_connection_types
for connection_type, provider_info in providers_manager.hooks.items():
/usr/local/lib/python3.9/_collections_abc.py:851: in __iter__
yield (key, self._mapping[key])
airflow/providers_manager.py:110: in __getitem__
value = value()
airflow/providers_manager.py:796: in _import_hook
hook_class = _sanity_check(package_name, hook_class_name, provider_info)
airflow/providers_manager.py:284: in _sanity_check
imported_class = import_string(class_name)
airflow/utils/module_loading.py:36: in import_string
module = import_module(module_path)
/usr/local/lib/python3.9/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1030: in _gcd_import
???
<frozen importlib._bootstrap>:1007: in _find_and_load
???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:680: in _load_unlocked
???
/usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
exec(co, module.__dict__)
airflow/providers/snowflake/hooks/snowflake.py:29: in <module>
from snowflake import connector
<frozen importlib._bootstrap>:1007: in _find_and_load
???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:680: in _load_unlocked
???
/usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
exec(co, module.__dict__)
/usr/local/lib/python3.9/site-packages/snowflake/connector/azure_storage_client.py:18: in <module>
from .encryption_util import EncryptionMetadata
<frozen importlib._bootstrap>:1007: in _find_and_load
???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:680: in _load_unlocked
???
/usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
exec(co, module.__dict__)
/usr/local/lib/python3.9/site-packages/snowflake/connector/encryption_util.py:15: in <module>
from Cryptodome.Cipher import AES
<frozen importlib._bootstrap>:1007: in _find_and_load
???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:680: in _load_unlocked
???
/usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
exec(co, module.__dict__)
/usr/local/lib/python3.9/site-packages/Cryptodome/Cipher/__init__.py:35: in <module>
from Cryptodome.Cipher._mode_siv import _create_siv_cipher
<frozen importlib._bootstrap>:1007: in _find_and_load
???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:680: in _load_unlocked
???
/usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
exec(co, module.__dict__)
/usr/local/lib/python3.9/site-packages/Cryptodome/Cipher/_mode_siv.py:44: in <module>
from Cryptodome.Protocol.KDF import _S2V
<frozen importlib._bootstrap>:1007: in _find_and_load
???
<frozen importlib._bootstrap>:986: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:680: in _load_unlocked
???
/usr/local/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:168: in exec_module
exec(co, module.__dict__)
/usr/local/lib/python3.9/site-packages/Cryptodome/Protocol/KDF.py:47: in <module>
_raw_scrypt_lib = load_pycryptodome_raw_lib("Cryptodome.Protocol._scrypt",
/usr/local/lib/python3.9/site-packages/Cryptodome/Util/_raw_api.py:306: in load_pycryptodome_raw_lib
return load_lib(full_name, cdecl)
/usr/local/lib/python3.9/site-packages/Cryptodome/Util/_raw_api.py:100: in load_lib
ffi.cdef(cdecl)
/usr/local/lib/python3.9/site-packages/cffi/api.py:112: in cdef
self._cdef(csource, override=override, packed=packed, pack=pack)
/usr/local/lib/python3.9/site-packages/cffi/api.py:126: in _cdef
self._parser.parse(csource, override=override, **options)
/usr/local/lib/python3.9/site-packages/cffi/cparser.py:389: in parse
self._internal_parse(csource)
/usr/local/lib/python3.9/site-packages/cffi/cparser.py:394: in _internal_parse
ast, macros, csource = self._parse(csource)
/usr/local/lib/python3.9/site-packages/cffi/cparser.py:336: in _parse
ast = _get_parser().parse(fullcsource)
/usr/local/lib/python3.9/site-packages/pycparser/c_parser.py:147: in parse
return self.cparser.parse(
/usr/local/lib/python3.9/site-packages/pycparser/ply/yacc.py:331: in parse
return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
/usr/local/lib/python3.9/site-packages/pycparser/ply/yacc.py:1061: in parseopt_notrack
lookahead = get_token() # Get the next token
/usr/local/lib/python3.9/site-packages/pycparser/c_lexer.py:76: in token
self.last_token = self.lexer.token()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org