You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Taragolis (via GitHub)" <gi...@apache.org> on 2024/04/03 12:00:52 UTC
[PR] Add fallback `region_name` value to AWS Executors [airflow]
Taragolis opened a new pull request, #38704:
URL: https://github.com/apache/airflow/pull/38704
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
Add fallback to `region_name` it might not obtained if configuration not persists in `airflow.cfg` or in env vars or configuration loaded without provider configurations
In addition use `conf_vars` helper for configure tests, it will automatically undo configuration in the end of test, so it reduce side effect when one test allow to pass/fail next text tests.
Originally it cause flakey behaviour in parallel non-db tests when it might run tests agains not properly configured tests in `main` or PRs
```console
________________ TestAwsBatchExecutor.test_health_check_failure ________________
[gw6] linux -- Python 3.12.2 /usr/local/bin/python
self = <tests.providers.amazon.aws.executors.batch.test_batch_executor.TestAwsBatchExecutor object at 0x7effd436bc20>
mock_executor = <MagicMock name='load' id='139637296591360'>
@mock.patch(
"airflow.providers.amazon.aws.executors.batch.boto_schema.BatchDescribeJobsResponseSchema.load"
)
def test_health_check_failure(self, mock_executor):
mock_executor.batch.describe_jobs.side_effect = Exception("Test_failure")
> executor = AwsBatchExecutor()
tests/providers/amazon/aws/executors/batch/test_batch_executor.py:519:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
airflow/providers/amazon/aws/executors/batch/batch_executor.py:102: in __init__
self.load_batch_connection(check_connection=False)
airflow/providers/amazon/aws/executors/batch/batch_executor.py:156: in load_batch_connection
region_name = conf.get(CONFIG_GROUP_NAME, AllBatchConfigKeys.REGION_NAME)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <airflow.configuration.AirflowConfigParser object at 0x7f0002c10500>
section = 'aws_batch_executor', key = 'region_name', suppress_warnings = False
_extra_stacklevel = 0, kwargs = {}, warning_emitted = False
option_description = {}, deprecated_section = None, deprecated_key = None
def get( # type: ignore[override,misc]
self,
section: str,
key: str,
suppress_warnings: bool = False,
_extra_stacklevel: int = 0,
**kwargs,
) -> str | None:
section = section.lower()
key = key.lower()
warning_emitted = False
deprecated_section: str | None
deprecated_key: str | None
option_description = self.configuration_description.get(section, {}).get(key, {})
if option_description.get("deprecated"):
deprecation_reason = option_description.get("deprecation_reason", "")
warnings.warn(
f"The '{key}' option in section {section} is deprecated. {deprecation_reason}",
DeprecationWarning,
stacklevel=2 + _extra_stacklevel,
)
# For when we rename whole sections
if section in self.inversed_deprecated_sections:
deprecated_section, deprecated_key = (section, key)
section = self.inversed_deprecated_sections[section]
if not self._suppress_future_warnings:
warnings.warn(
f"The config section [{deprecated_section}] has been renamed to "
f"[{section}]. Please update your `conf.get*` call to use the new name",
FutureWarning,
stacklevel=2 + _extra_stacklevel,
)
# Don't warn about individual rename if the whole section is renamed
warning_emitted = True
elif (section, key) in self.inversed_deprecated_options:
# Handle using deprecated section/key instead of the new section/key
new_section, new_key = self.inversed_deprecated_options[(section, key)]
if not self._suppress_future_warnings and not warning_emitted:
warnings.warn(
f"section/key [{section}/{key}] has been deprecated, you should use"
f"[{new_section}/{new_key}] instead. Please update your `conf.get*` call to use the "
"new name",
FutureWarning,
stacklevel=2 + _extra_stacklevel,
)
warning_emitted = True
deprecated_section, deprecated_key = section, key
section, key = (new_section, new_key)
elif section in self.deprecated_sections:
# When accessing the new section name, make sure we check under the old config name
deprecated_key = key
deprecated_section = self.deprecated_sections[section][0]
else:
deprecated_section, deprecated_key, _ = self.deprecated_options.get(
(section, key), (None, None, None)
)
# first check environment variables
option = self._get_environment_variables(
deprecated_key,
deprecated_section,
key,
section,
issue_warning=not warning_emitted,
extra_stacklevel=_extra_stacklevel,
)
if option is not None:
return option
# ...then the config file
option = self._get_option_from_config_file(
deprecated_key,
deprecated_section,
key,
kwargs,
section,
issue_warning=not warning_emitted,
extra_stacklevel=_extra_stacklevel,
)
if option is not None:
return option
# ...then commands
option = self._get_option_from_commands(
deprecated_key,
deprecated_section,
key,
section,
issue_warning=not warning_emitted,
extra_stacklevel=_extra_stacklevel,
)
if option is not None:
return option
# ...then from secret backends
option = self._get_option_from_secrets(
deprecated_key,
deprecated_section,
key,
section,
issue_warning=not warning_emitted,
extra_stacklevel=_extra_stacklevel,
)
if option is not None:
return option
# ...then the default config
if self.get_default_value(section, key) is not None or "fallback" in kwargs:
return expand_env_var(self.get_default_value(section, key, **kwargs))
if self.get_default_pre_2_7_value(section, key) is not None:
# no expansion needed
return self.get_default_pre_2_7_value(section, key, **kwargs)
if not suppress_warnings:
log.warning("section/key [%s/%s] not found in config", section, key)
> raise AirflowConfigException(f"section/key [{section}/{key}] not found in config")
E airflow.exceptions.AirflowConfigException: section/key [aws_batch_executor/region_name] not found in config
airflow/configuration.py:1052: AirflowConfigException
----------------------------- Captured stdout call -----------------------------
[2024-04-03T10:14:13.235+0000] {batch_executor.py:150} INFO - Loading Connection information
[2024-04-03T10:14:13.237+0000] {configuration.py:1050} WARNING - section/*** [aws_batch_executor/region_name] not found in config
FAILED tests/providers/amazon/aws/executors/batch/test_batch_executor.py::TestAwsBatchExecutor::test_health_check_failure - airflow.exceptions.AirflowConfigException: section/key [aws_batch_executor/region_name] not found in config
```
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add fallback `region_name` value to AWS Executors [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #38704:
URL: https://github.com/apache/airflow/pull/38704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org