You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/05/16 06:27:02 UTC
[GitHub] [airflow] jhtimmins opened a new pull request #8884: Use env_vars to for reload config test.
jhtimmins opened a new pull request #8884:
URL: https://github.com/apache/airflow/pull/8884
Updates the test logging configuration to use the `env_vars` to set the environment variables instead of setting them explicitly within the `settings_context` context manager.
---
Make sure to mark the boxes below before creating PR: [x]
- [ ] Description above provides context of the change
- [ ] Unit tests coverage for changes (not needed for documentation changes)
- [ ] Target Github ISSUE in description if exists
- [ ] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [ ] Relevant documentation is updated including usage instructions.
- [ ] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #8884: Use env_vars to for reload config test.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8884:
URL: https://github.com/apache/airflow/pull/8884#discussion_r426137598
##########
File path: tests/utils/test_dag_processing.py
##########
@@ -377,34 +377,35 @@ class path, thus when reloading logging module the airflow.processor_manager
logger should not be configured.
"""
with settings_context(SETTINGS_FILE_VALID):
- # Launch a process through DagFileProcessorAgent, which will try
- # reload the logging module.
- test_dag_path = os.path.join(TEST_DAG_FOLDER, 'test_scheduler_dags.py')
- async_mode = 'sqlite' not in conf.get('core', 'sql_alchemy_conn')
- log_file_loc = conf.get('logging', 'DAG_PROCESSOR_MANAGER_LOG_LOCATION')
-
- try:
- os.remove(log_file_loc)
- except OSError:
- pass
-
- # Starting dag processing with 0 max_runs to avoid redundant operations.
- processor_agent = DagFileProcessorAgent(test_dag_path,
- 0,
- type(self)._processor_factory,
- timedelta.max,
- [],
- False,
- async_mode)
- processor_agent.start()
- if not async_mode:
- processor_agent.run_single_parsing_loop()
-
- processor_agent._process.join()
- # Since we are reloading logging config not creating this file,
- # we should expect it to be nonexistent.
-
- self.assertFalse(os.path.isfile(log_file_loc))
+ with env_vars({('logging', 'logging_config_class'): 'custom_airflow_local_settings.LOGGING_CONFIG'}):
Review comment:
```
tests/utils/test_dag_processing.py:380:0: C0301: Line too long (113/110) (line-too-long)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stale[bot] commented on pull request #8884: Use env_vars to for reload config test.
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #8884:
URL: https://github.com/apache/airflow/pull/8884#issuecomment-653726263
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on a change in pull request #8884: Use env_vars to for reload config test.
Posted by GitBox <gi...@apache.org>.
kaxil commented on a change in pull request #8884:
URL: https://github.com/apache/airflow/pull/8884#discussion_r426139517
##########
File path: tests/utils/test_dag_processing.py
##########
@@ -377,34 +377,35 @@ class path, thus when reloading logging module the airflow.processor_manager
logger should not be configured.
"""
with settings_context(SETTINGS_FILE_VALID):
- # Launch a process through DagFileProcessorAgent, which will try
- # reload the logging module.
- test_dag_path = os.path.join(TEST_DAG_FOLDER, 'test_scheduler_dags.py')
- async_mode = 'sqlite' not in conf.get('core', 'sql_alchemy_conn')
- log_file_loc = conf.get('logging', 'DAG_PROCESSOR_MANAGER_LOG_LOCATION')
-
- try:
- os.remove(log_file_loc)
- except OSError:
- pass
-
- # Starting dag processing with 0 max_runs to avoid redundant operations.
- processor_agent = DagFileProcessorAgent(test_dag_path,
- 0,
- type(self)._processor_factory,
- timedelta.max,
- [],
- False,
- async_mode)
- processor_agent.start()
- if not async_mode:
- processor_agent.run_single_parsing_loop()
-
- processor_agent._process.join()
- # Since we are reloading logging config not creating this file,
- # we should expect it to be nonexistent.
-
- self.assertFalse(os.path.isfile(log_file_loc))
+ with env_vars({('logging', 'logging_config_class'): 'custom_airflow_local_settings.LOGGING_CONFIG'}):
Review comment:
```suggestion
with env_vars({
('logging', 'logging_config_class'): 'custom_airflow_local_settings.LOGGING_CONFIG'
}):
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jhtimmins commented on a change in pull request #8884: Use env_vars to for reload config test.
Posted by GitBox <gi...@apache.org>.
jhtimmins commented on a change in pull request #8884:
URL: https://github.com/apache/airflow/pull/8884#discussion_r426124504
##########
File path: tests/test_logging_config.py
##########
@@ -128,43 +128,41 @@ def settings_context(content, directory=None, name='LOGGING_CONFIG'):
:param content:
The content of the settings file
"""
- initial_logging_config = os.environ.get("AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS", "")
- try:
- settings_root = tempfile.mkdtemp()
- filename = f"{SETTINGS_DEFAULT_NAME}.py"
- if directory:
- # Replace slashes by dots
- module = directory.replace('/', '.') + '.' + SETTINGS_DEFAULT_NAME + '.' + name
-
- # Create the directory structure
- dir_path = os.path.join(settings_root, directory)
- pathlib.Path(dir_path).mkdir(parents=True, exist_ok=True)
-
- # Add the __init__ for the directories
- # This is required for Python 2.7
- basedir = settings_root
- for part in directory.split('/'):
- open(os.path.join(basedir, '__init__.py'), 'w').close()
- basedir = os.path.join(basedir, part)
+ # initial_logging_config = os.environ.get("AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS", "")
+ # try:
+ settings_root = tempfile.mkdtemp()
+ filename = f"{SETTINGS_DEFAULT_NAME}.py"
+ if directory:
+ # Replace slashes by dots
+ module = directory.replace('/', '.') + '.' + SETTINGS_DEFAULT_NAME + '.' + name
+ # breakpoint()
Review comment:
Oops, my bad
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] stale[bot] closed pull request #8884: Use env_vars to for reload config test.
Posted by GitBox <gi...@apache.org>.
stale[bot] closed pull request #8884:
URL: https://github.com/apache/airflow/pull/8884
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #8884: Use env_vars to for reload config test.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8884:
URL: https://github.com/apache/airflow/pull/8884#discussion_r426124472
##########
File path: tests/test_logging_config.py
##########
@@ -128,43 +128,41 @@ def settings_context(content, directory=None, name='LOGGING_CONFIG'):
:param content:
The content of the settings file
"""
- initial_logging_config = os.environ.get("AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS", "")
- try:
- settings_root = tempfile.mkdtemp()
- filename = f"{SETTINGS_DEFAULT_NAME}.py"
- if directory:
- # Replace slashes by dots
- module = directory.replace('/', '.') + '.' + SETTINGS_DEFAULT_NAME + '.' + name
-
- # Create the directory structure
- dir_path = os.path.join(settings_root, directory)
- pathlib.Path(dir_path).mkdir(parents=True, exist_ok=True)
-
- # Add the __init__ for the directories
- # This is required for Python 2.7
- basedir = settings_root
- for part in directory.split('/'):
- open(os.path.join(basedir, '__init__.py'), 'w').close()
- basedir = os.path.join(basedir, part)
+ # initial_logging_config = os.environ.get("AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS", "")
+ # try:
+ settings_root = tempfile.mkdtemp()
+ filename = f"{SETTINGS_DEFAULT_NAME}.py"
+ if directory:
+ # Replace slashes by dots
+ module = directory.replace('/', '.') + '.' + SETTINGS_DEFAULT_NAME + '.' + name
+ # breakpoint()
Review comment:
Can you remove it?
##########
File path: tests/test_logging_config.py
##########
@@ -128,43 +128,41 @@ def settings_context(content, directory=None, name='LOGGING_CONFIG'):
:param content:
The content of the settings file
"""
- initial_logging_config = os.environ.get("AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS", "")
- try:
- settings_root = tempfile.mkdtemp()
- filename = f"{SETTINGS_DEFAULT_NAME}.py"
- if directory:
- # Replace slashes by dots
- module = directory.replace('/', '.') + '.' + SETTINGS_DEFAULT_NAME + '.' + name
-
- # Create the directory structure
- dir_path = os.path.join(settings_root, directory)
- pathlib.Path(dir_path).mkdir(parents=True, exist_ok=True)
-
- # Add the __init__ for the directories
- # This is required for Python 2.7
- basedir = settings_root
- for part in directory.split('/'):
- open(os.path.join(basedir, '__init__.py'), 'w').close()
- basedir = os.path.join(basedir, part)
+ # initial_logging_config = os.environ.get("AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS", "")
+ # try:
Review comment:
Can you remove it?
##########
File path: tests/test_logging_config.py
##########
@@ -128,43 +128,41 @@ def settings_context(content, directory=None, name='LOGGING_CONFIG'):
:param content:
The content of the settings file
"""
- initial_logging_config = os.environ.get("AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS", "")
- try:
- settings_root = tempfile.mkdtemp()
- filename = f"{SETTINGS_DEFAULT_NAME}.py"
- if directory:
- # Replace slashes by dots
- module = directory.replace('/', '.') + '.' + SETTINGS_DEFAULT_NAME + '.' + name
-
- # Create the directory structure
- dir_path = os.path.join(settings_root, directory)
- pathlib.Path(dir_path).mkdir(parents=True, exist_ok=True)
-
- # Add the __init__ for the directories
- # This is required for Python 2.7
- basedir = settings_root
- for part in directory.split('/'):
- open(os.path.join(basedir, '__init__.py'), 'w').close()
- basedir = os.path.join(basedir, part)
+ # initial_logging_config = os.environ.get("AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS", "")
+ # try:
+ settings_root = tempfile.mkdtemp()
+ filename = f"{SETTINGS_DEFAULT_NAME}.py"
+ if directory:
+ # Replace slashes by dots
+ module = directory.replace('/', '.') + '.' + SETTINGS_DEFAULT_NAME + '.' + name
+ # breakpoint()
+
+ # Create the directory structure
+ dir_path = os.path.join(settings_root, directory)
+ pathlib.Path(dir_path).mkdir(parents=True, exist_ok=True)
+
+ # Add the __init__ for the directories
+ # This is required for Python 2.7
+ basedir = settings_root
+ for part in directory.split('/'):
open(os.path.join(basedir, '__init__.py'), 'w').close()
+ basedir = os.path.join(basedir, part)
+ open(os.path.join(basedir, '__init__.py'), 'w').close()
- settings_file = os.path.join(dir_path, filename)
- else:
- module = SETTINGS_DEFAULT_NAME + '.' + name
- settings_file = os.path.join(settings_root, filename)
+ settings_file = os.path.join(dir_path, filename)
+ else:
+ module = SETTINGS_DEFAULT_NAME + '.' + name
+ # breakpoint()
Review comment:
Can you remove it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org