You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/10 09:56:20 UTC
[GitHub] [airflow] arvindeybram opened a new issue #22141: Dump configurations in airflow scheduler logs based on the config it reads
arvindeybram opened a new issue #22141:
URL: https://github.com/apache/airflow/issues/22141
### Description
We don't have any way to cross-verify the configs that the airflow scheduler uses. It would be good to have it logged somewhere so that users can cross-verify it.
### Use case/motivation
How do you know for sure that the configs in airflow.cfg are being correctly parsed by airflow scheduler?
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #22141: Dump configurations in airflow scheduler logs based on the config it reads
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #22141:
URL: https://github.com/apache/airflow/issues/22141#issuecomment-1066098676
Yep. we even have some functions for this (used in CLI) - It also masks sensitive values, so this is important to use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22141: Dump configurations in airflow scheduler logs based on the config it reads
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22141:
URL: https://github.com/apache/airflow/issues/22141#issuecomment-1063885639
I thhini it would be super helpful. I've seen many people who had problems with understanding what their configuraiton is.
The `airflow config list` is not enough - the problem is that many people run airflow using systemd or other solutions and it would be great to know what ACTUALLY was used by the process that started.
In many cases people do not realise that variables they have set in .bashrc are not set in systemd for example, and there might be plenty other reasons why running airflow config list provides different results than what the running airflow process sees.
Big +1 on this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #22141: Dump configurations in airflow scheduler logs based on the config it reads
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #22141:
URL: https://github.com/apache/airflow/issues/22141#issuecomment-1066047007
Just dump everything to the log file when the scheduler launches, then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tirkarthi commented on issue #22141: Dump configurations in airflow scheduler logs based on the config it reads
Posted by GitBox <gi...@apache.org>.
tirkarthi commented on issue #22141:
URL: https://github.com/apache/airflow/issues/22141#issuecomment-1080560527
Hi, I would like to work on this feature and this is my first patch. I looked into `airflow conf list` that has `show_config` to print config. It doesn't seem to mask sensitive values. I wrote another function that initializes `AirflowConfigParser` and uses `as_dict` that masks sensitive values to read the masked config again using `read_dict`. I am not sure if the approach is correct. Patch with test as below
https://github.com/apache/airflow/blob/ae6570ffefc124fd4ba7cf16648ce7dd21dc6aec/airflow/cli/commands/config_command.py#L23-L35
Patch :
```diff
diff --git a/airflow/cli/commands/scheduler_command.py b/airflow/cli/commands/scheduler_command.py
index bc6e983ee..4fae84f43 100644
--- a/airflow/cli/commands/scheduler_command.py
+++ b/airflow/cli/commands/scheduler_command.py
@@ -16,6 +16,7 @@
# under the License.
"""Scheduler command"""
+import io
import signal
from multiprocessing import Process
from typing import Optional
@@ -24,6 +25,7 @@ import daemon
from daemon.pidfile import TimeoutPIDLockFile
from airflow import settings
+from airflow.configuration import AirflowConfigParser, conf
from airflow.jobs.scheduler_job import SchedulerJob
from airflow.utils import cli as cli_utils
from airflow.utils.cli import process_subdir, setup_locations, setup_logging, sigint_handler, sigquit_handler
@@ -38,6 +40,16 @@ def _create_scheduler_job(args):
return job
+def _get_masked_config():
+ parser = AirflowConfigParser(strict=False, interpolation=None)
+ config_dict = conf.as_dict()
+ parser.read_dict(config_dict)
+
+ with io.StringIO() as output:
+ parser.write(output)
+ return output.getvalue()
+
+
def _run_scheduler_job(args):
skip_serve_logs = args.skip_serve_logs
job = _create_scheduler_job(args)
@@ -53,6 +65,7 @@ def _run_scheduler_job(args):
def scheduler(args):
"""Starts Airflow Scheduler"""
print(settings.HEADER)
+ print(_get_masked_config())
if args.daemon:
pid, stdout, stderr, log_file = setup_locations(
diff --git a/tests/cli/commands/test_scheduler_command.py b/tests/cli/commands/test_scheduler_command.py
index 00b851f55..8946e6c52 100644
--- a/tests/cli/commands/test_scheduler_command.py
+++ b/tests/cli/commands/test_scheduler_command.py
@@ -15,6 +15,9 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+import contextlib
+import io
+import os
import unittest
from unittest import mock
@@ -87,3 +90,14 @@ class TestSchedulerCommand(unittest.TestCase):
scheduler_command.scheduler(args)
finally:
mock_process().terminate.assert_called()
+
+ @mock.patch.dict(os.environ, {}, clear=True)
+ @mock.patch("airflow.cli.commands.scheduler_command.SchedulerJob")
+ @mock.patch("airflow.cli.commands.scheduler_command.Process")
+ def test_scheduler_print_config(self, mock_process, mock_scheduler_job):
+ args = self.parser.parse_args(['scheduler'])
+ with conf_vars({("core", "sql_alchemy_conn_cmd"): 'echo hello'}):
+ with contextlib.redirect_stdout(io.StringIO()) as temp_stdout:
+ scheduler_command.scheduler(args)
+ output = temp_stdout.getvalue()
+ assert "sql_alchemy_conn = < hidden >" in output
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #22141: Dump configurations in airflow scheduler logs based on the config it reads
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #22141:
URL: https://github.com/apache/airflow/issues/22141#issuecomment-1066098676
Yep. we even have some functions for this (used in CLI).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tirkarthi commented on issue #22141: Dump configurations in airflow scheduler logs based on the config it reads
Posted by GitBox <gi...@apache.org>.
tirkarthi commented on issue #22141:
URL: https://github.com/apache/airflow/issues/22141#issuecomment-1081961229
Thanks, created https://github.com/apache/airflow/pull/22588
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22141: Dump configurations in airflow scheduler logs based on the config it reads
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22141:
URL: https://github.com/apache/airflow/issues/22141#issuecomment-1066098676
Yep. we even have some functions for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #22141: Dump configurations in airflow scheduler logs based on the config it reads
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #22141:
URL: https://github.com/apache/airflow/issues/22141#issuecomment-1081919580
Just open a PR with it it (referring to this one - it will be easier to dicuss it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org