You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/04/14 04:11:09 UTC
[GitHub] [airflow] mik-laj opened a new pull request #8290: Add airflow info
command
mik-laj opened a new pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290
Hi.
Users often have problems with the runtime environment or with several basic configuration options. To make it easier to gather information, I have created an additional command that allows us to collect the necessary data also in anonymized form.
For now, I've added only a few basic keys, but I think we can easily extend based on observation. In particular, I'd like to see keys that get some data from the database (e.g. jobs). However, this can be done in the future.
I would like to cherry-pick this command to Airflow 1.10, so I didn't use the latest syntax from Python 3.
Best regards,
Kamil
---
Make sure to mark the boxes below before creating PR: [x]
- [X] Description above provides context of the change
- [X] Unit tests coverage for changes (not needed for documentation changes)
- [X] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [X] Relevant documentation is updated including usage instructions.
- [X] I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8290: Add airflow
info command
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613226464
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=h1) Report
> Merging [#8290](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45c8983306ab1c54abdacd8f870e790fad25cb37&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `1.69%`.
[](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8290 +/- ##
==========================================
- Coverage 88.45% 87.95% -0.50%
==========================================
Files 937 938 +1
Lines 45234 45352 +118
==========================================
- Hits 40011 39891 -120
- Misses 5223 5461 +238
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/cli/commands/info\_command.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvaW5mb19jb21tYW5kLnB5) | `0.00% <0.00%> (ø)` | |
| [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `97.30% <100.00%> (+0.02%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `75.18% <0.00%> (-3.76%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=footer). Last update [45c8983...437dae1](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r409479059
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,427 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import logging
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+from urllib.parse import urlsplit, urlunsplit
+
+import requests
+import tenacity
+from typing_extensions import Protocol
Review comment:
https://github.com/python/typing/blob/master/typing_extensions/src_py3/typing_extensions.py#L1116-L1119
I can not see the difference. We can't delete typing_extension anyway. This is still needed in many places.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8290: Add airflow
info command
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613226464
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=h1) Report
> Merging [#8290](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45c8983306ab1c54abdacd8f870e790fad25cb37&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `1.69%`.
[](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8290 +/- ##
==========================================
- Coverage 88.45% 87.95% -0.50%
==========================================
Files 937 938 +1
Lines 45234 45352 +118
==========================================
- Hits 40011 39891 -120
- Misses 5223 5461 +238
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/cli/commands/info\_command.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvaW5mb19jb21tYW5kLnB5) | `0.00% <0.00%> (ø)` | |
| [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `97.30% <100.00%> (+0.02%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `75.18% <0.00%> (-3.76%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=footer). Last update [45c8983...437dae1](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r409476067
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,427 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import logging
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+from urllib.parse import urlsplit, urlunsplit
+
+import requests
+import tenacity
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+log = logging.getLogger(__name__)
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+ def process_username(self, value):
+ """Remove pii from ussername"""
+
+ def process_url(self, value):
+ """Remove pii from URL"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def _identity(self, value):
+ return value
+
+ process_path = process_username = process_url = _identity
+
+ del _identity
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+ def process_username(self, value):
+ if not value:
+ return value
+ return value[0] + "..." + value[-1]
+
+ def process_url(self, value):
+ if not value:
+ return value
+
+ url_parts = urlsplit(value)
+ netloc = None
+ if url_parts.netloc:
+ # unpack
+ userinfo = None
+ host = None
+ username = None
+ password = None
+
+ if "@" in url_parts.netloc:
+ userinfo, _, host = url_parts.netloc.partition("@")
+ else:
+ host = url_parts.netloc
+ if userinfo:
+ if ":" in userinfo:
+ username, _, password = userinfo.partition(":")
+ else:
+ username = userinfo
+
+ # anonymize
+ username = self.process_username(username) if username else None
+ password = "PASSWORD" if password else None
+
+ # pack
+ if username and password and host:
+ netloc = username + ":" + password + "@" + host
+ elif username and host:
+ netloc = username + "@" + host
+ elif password and host:
+ netloc = ":" + password + "@" + host
+ elif host:
+ netloc = host
+ else:
+ netloc = ""
+
+ return urlunsplit((url_parts.scheme, netloc, url_parts.path, url_parts.query, url_parts.fragment))
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
+
+
+_MACHINE_TO_ARCHITECTURE = {
+ "amd64": Architecture.X86_64,
+ "x86_64": Architecture.X86_64,
+ "i686-64": Architecture.X86_64,
+ "i386": Architecture.X86,
+ "i686": Architecture.X86,
+ "x86": Architecture.X86,
+ "ia64": Architecture.X86, # Itanium is different x64 arch, treat it as the common x86.
+ "powerpc": Architecture.PPC,
+ "power macintosh": Architecture.PPC,
+ "ppc64": Architecture.PPC,
+ "armv6": Architecture.ARM,
+ "armv6l": Architecture.ARM,
+ "arm64": Architecture.ARM,
+ "armv7": Architecture.ARM,
+ "armv7l": Architecture.ARM,
+}
+
+
+class AirflowInfo:
+ """All information related to Airflow, system and other."""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.airflow_version = airflow_version
+ self.system = SystemInfo(anonymizer)
+ self.tools = ToolsInfo(anonymizer)
+ self.paths = PathsInfo(anonymizer)
+ self.config = ConfigInfo(anonymizer)
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Apache Airflow [{version}]
+
+ {system}
+
+ {tools}
+
+ {paths}
+
+ {config}
+ """
+ )
+ .strip()
+ .format(
+ version=self.airflow_version,
+ system=self.system,
+ tools=self.tools,
+ paths=self.paths,
+ config=self.config,
+ )
+ )
+
+
+class SystemInfo:
+ """Basic system and python information"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.operating_system = OperatingSystem.get_current()
+ self.arch = Architecture.get_current()
+ self.uname = platform.uname()
+ self.locale = locale.getdefaultlocale()
+ self.python_location = anonymizer.process_path(sys.executable)
+ self.python_version = sys.version.replace("\n", " ")
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Platform: [{os}, {arch}] {uname}
+ Locale: {locale}
+ Python Version: [{python_version}]
+ Python Location: [{python_location}]
+ """
+ )
+ .strip()
+ .format(
+ os=self.operating_system or "NOT AVAILABLE",
+ arch=self.arch or "NOT AVAILABLE",
+ uname=self.uname,
+ locale=self.locale,
+ python_version=self.python_version,
+ python_location=self.python_location,
+ )
+ )
+
+
+class PathsInfo:
+ """Path informaation"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ system_path = os.environ.get("PATH", "").split(os.pathsep)
+
+ self.airflow_home = anonymizer.process_path(configuration.get_airflow_home())
+ self.system_path = [anonymizer.process_path(p) for p in system_path]
+ self.python_path = [anonymizer.process_path(p) for p in sys.path]
+ self.airflow_on_path = any(
+ os.path.exists(os.path.join(path_elem, "airflow")) for path_elem in system_path
+ )
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Airflow Home: [{airflow_home}]
+ System PATH: [{system_path}]
+ Python PATH: [{python_path}]
+ airflow on PATH: [{airflow_on_path}]
+ """
+ )
+ .strip()
+ .format(
+ airflow_home=self.airflow_home,
+ system_path=os.pathsep.join(self.system_path),
+ python_path=os.pathsep.join(self.python_path),
+ airflow_on_path=self.airflow_on_path,
+ )
+ )
+
+
+class ConfigInfo:
+ """"Most critical config properties"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.executor = configuration.conf.get("core", "executor")
+ self.dags_folder = anonymizer.process_path(
+ configuration.conf.get("core", "dags_folder", fallback="NOT AVAILABLE")
+ )
+ self.plugins_folder = anonymizer.process_path(
+ configuration.conf.get("core", "plugins_folder", fallback="NOT AVAILABLE")
+ )
+ self.base_log_folder = anonymizer.process_path(
+ configuration.conf.get("logging", "base_log_folder", fallback="NOT AVAILABLE")
+ )
+ self.base_log_folder = anonymizer.process_path(
+ configuration.conf.get("logging", "base_log_folder", fallback="NOT AVAILABLE")
+ )
+ self.sql_alchemy_conn = anonymizer.process_url(
+ configuration.conf.get("core", "SQL_ALCHEMY_CONN", fallback="NOT AVAILABLE")
+ )
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Executor: [{executor}]
+ SQL Alchemy Conn: [{sql_alchemy_conn}]
+ DAGS Folder: [{dags_folder}]
+ Plugins Folder: [{plugins_folder}]
+ Base Log Folder: [{base_log_folder}]
+ """
+ )
+ .strip()
+ .format(
+ executor=self.executor,
+ sql_alchemy_conn=self.sql_alchemy_conn,
+ dags_folder=self.dags_folder,
+ plugins_folder=self.plugins_folder,
+ base_log_folder=self.base_log_folder,
+ )
+ )
+
+
+class ToolsInfo:
+ """The versions of the tools that Airflow uses"""
+
+ def __init__(self, anonymize: Anonymizer):
+ del anonymize # Nothing to anonymize here.
+ self.git_version = self._get_version(["git", "--version"])
+ self.ssh_version = self._get_version(["ssh", "-V"])
+ self.kubectl_version = self._get_version(["kubectl", "version", "--short=True", "--client=True"])
+ self.gcloud_version = self._get_version(["gcloud", "version"], grep=b"Google Cloud SDK")
+ self.cloud_sql_proxy_version = self._get_version(["cloud_sql_proxy", "--version"])
+ self.mysql_version = self._get_version(["mysql", "--version"])
+ self.sqlite3_version = self._get_version(["sqlite3", "--version"])
+ self.psql_version = self._get_version(["psql", "--version"])
Review comment:
Airflow uses psql here: https://github.com/apache/airflow/blob/master/airflow/cli/commands/db_command.py#L91
I did not check any version of the Python library. For this we have a very other command - `pip freeze`. In the future, we can add selected libraries if necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r408255802
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -310,9 +375,50 @@ def __str__(self):
)
+class FileIoException(Exception):
+ """Raises when error happens in FileIo.io integration"""
+
+ pass
+
+
+@tenacity.retry(
+ stop=tenacity.stop_after_attempt(5),
+ wait=tenacity.wait_exponential(multiplier=1, max=10),
+ retry=tenacity.retry_if_exception_type(FileIoException),
+ before=tenacity.before_log(log, logging.DEBUG),
+ after=tenacity.after_log(log, logging.DEBUG),
+)
+def upload_text_to_fileio(content):
+ """Uload text file to File.io service and return lnk"""
+ resp = requests.post("https://file.io", files={"file": ("airflow-report.txt", content)})
+ if not resp.ok:
+ raise FileIoException("Failed to send report to file.io service.")
+ try:
+ return resp.json()["link"]
+ except ValueError as e:
+ log.debug(e)
+ raise FileIoException("Failed to send report to file.io service.")
+
+
+def send_report_to_fileio(info):
+ print("Uploading report to file.io service.")
+ try:
+ link = upload_text_to_fileio(str(info))
+ print("Report uploaded.")
+ print()
+ print("Link:\t", link)
+ print()
+ except FileIoException as ex:
+ print(str(ex))
+
+
def show_info(args):
"""
Show information related to Airflow, system and other.
"""
anonymizer = PiiAnonymizer() if args.anonymize else NullAnonymizer()
- print(AirflowInfo(anonymizer))
+ info = AirflowInfo(anonymizer)
+ if args.file_io:
Review comment:
I think we should force anonymisation when file.io is used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8290: Add airflow info command
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613634334
@kaxil -> FYI ^^
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r407967308
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,318 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def process_path(self, value):
+ return value
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
+
+
+_MACHINE_TO_ARCHITECTURE = {
+ "amd64": Architecture.X86_64,
+ "x86_64": Architecture.X86_64,
+ "i686-64": Architecture.X86_64,
+ "i386": Architecture.X86,
+ "i686": Architecture.X86,
+ "x86": Architecture.X86,
+ "ia64": Architecture.X86, # Itanium is different x64 arch, treat it as the common x86.
+ "powerpc": Architecture.PPC,
+ "power macintosh": Architecture.PPC,
+ "ppc64": Architecture.PPC,
+ "armv6": Architecture.ARM,
+ "armv6l": Architecture.ARM,
+ "arm64": Architecture.ARM,
+ "armv7": Architecture.ARM,
+ "armv7l": Architecture.ARM,
+}
+
+
+class AirflowInfo:
+ """All information related to Airflow, system and other."""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.airflow_version = airflow_version
+ self.system = SystemInfo(anonymizer)
+ self.tools = ToolsInfo(anonymizer)
+ self.paths = PathsInfo(anonymizer)
+ self.config = ConfigInfo(anonymizer)
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Apache Airflow [{version}]
+
+ {system}
+
+ {tools}
+
+ {paths}
+
+ {config}
+ """
+ )
+ .strip()
+ .format(
+ version=airflow_version,
+ system=self.system,
+ tools=self.tools,
+ paths=self.paths,
+ config=self.config,
+ )
+ )
+
+
+class SystemInfo:
+ """Basic system and python information"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.operating_system = OperatingSystem.get_current()
+ self.arch = Architecture.get_current()
+ self.uname = platform.uname()
+ self.locale = locale.getdefaultlocale()
+ self.python_location = anonymizer.process_path(sys.executable)
+ self.python_version = sys.version.replace("\n", " ")
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Platform: [{os}, {arch}] {uname}
+ Locale: {locale}
+ Python Version: [{python_version}]
+ Python Location: [{python_location}]
+ """
+ )
+ .strip()
+ .format(
+ version=airflow_version,
+ os=self.operating_system or "NOT AVAILABLE",
+ arch=self.arch or "NOT AVAILABLE",
+ uname=self.uname,
+ locale=self.locale,
+ python_version=self.python_version,
+ python_location=self.python_location,
+ )
+ )
+
+
+class PathsInfo:
+ """Path informaation"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ system_path = os.environ.get("PATH", "").split(os.pathsep)
+
+ self.airflow_home = anonymizer.process_path(configuration.get_airflow_home())
+ self.system_path = [anonymizer.process_path(p) for p in system_path]
+ self.python_path = [anonymizer.process_path(p) for p in sys.path]
+ self.airflow_on_path = any(
+ os.path.exists(os.path.join(path_elem, "airflow")) for path_elem in system_path
+ )
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Airflow Home: [{airflow_home}]
+ System PATH: [{system_path}]
+ Python PATH: [{python_path}]
Review comment:
This option is crucial for me. Many people do not understand how Python modules are loaded. Python really [tries hard to intelligently](https://stackoverflow.com/questions/897792/where-is-pythons-sys-path-initialized-from/38403654#38403654) set sys.path. Additionally, Airflow modifies the state of this variable, so determining the content of this variable is very difficult.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r407964667
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,318 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def process_path(self, value):
+ return value
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
Review comment:
And we don't display None. Missing values is handled in __str__. https://github.com/apache/airflow/pull/8290/files#diff-1877860355e4eb978a17b0b48b6ba073R178
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io commented on issue #8290: Add airflow info
command
Posted by GitBox <gi...@apache.org>.
codecov-io commented on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613226464
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=h1) Report
> Merging [#8290](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45c8983306ab1c54abdacd8f870e790fad25cb37&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `1.69%`.
[](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8290 +/- ##
==========================================
- Coverage 88.45% 87.95% -0.50%
==========================================
Files 937 938 +1
Lines 45234 45352 +118
==========================================
- Hits 40011 39891 -120
- Misses 5223 5461 +238
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/cli/commands/info\_command.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvaW5mb19jb21tYW5kLnB5) | `0.00% <0.00%> (ø)` | |
| [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `97.30% <100.00%> (+0.02%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `75.18% <0.00%> (-3.76%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=footer). Last update [45c8983...437dae1](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on issue #8290: Add airflow info
command
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-614429276
It seems that the build failed because postgres was not up
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #8290: Add airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613631254
Ready for review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r407951101
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,318 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def process_path(self, value):
+ return value
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
Review comment:
I wanted this function to return only constants that are defined. Other results are erroneous and we should improve the function so that it returns correct values rather than random values from the code's point of view. It is worth adding that the detailed information you are asking for is already displayed on the screen.
```
Platform: [Mac OS, x86_64] uname_result(system='Darwin', node='Kamils-MacBook-Pro.local', release='19.4.0', version='Darwin Kernel Version 19.4.0: Wed Mar 4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64', machine='x86_64', processor='i386')
```
```
Platform: [Linux, x86_64] uname_result(system='Linux', node='eeefbb427a9c', release='4.19.76-linuxkit', version='#1 SMP Thu Oct 17 19:31:58 UTC 2019', machine='x86_64', processor='')
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r409453786
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,427 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import logging
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+from urllib.parse import urlsplit, urlunsplit
+
+import requests
+import tenacity
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+log = logging.getLogger(__name__)
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+ def process_username(self, value):
+ """Remove pii from ussername"""
+
+ def process_url(self, value):
+ """Remove pii from URL"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def _identity(self, value):
+ return value
+
+ process_path = process_username = process_url = _identity
+
+ del _identity
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+ def process_username(self, value):
+ if not value:
+ return value
+ return value[0] + "..." + value[-1]
+
+ def process_url(self, value):
+ if not value:
+ return value
+
+ url_parts = urlsplit(value)
+ netloc = None
+ if url_parts.netloc:
+ # unpack
+ userinfo = None
+ host = None
+ username = None
+ password = None
+
+ if "@" in url_parts.netloc:
+ userinfo, _, host = url_parts.netloc.partition("@")
+ else:
+ host = url_parts.netloc
+ if userinfo:
+ if ":" in userinfo:
+ username, _, password = userinfo.partition(":")
+ else:
+ username = userinfo
+
+ # anonymize
+ username = self.process_username(username) if username else None
+ password = "PASSWORD" if password else None
+
+ # pack
+ if username and password and host:
+ netloc = username + ":" + password + "@" + host
+ elif username and host:
+ netloc = username + "@" + host
+ elif password and host:
+ netloc = ":" + password + "@" + host
+ elif host:
+ netloc = host
+ else:
+ netloc = ""
+
+ return urlunsplit((url_parts.scheme, netloc, url_parts.path, url_parts.query, url_parts.fragment))
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
+
+
+_MACHINE_TO_ARCHITECTURE = {
+ "amd64": Architecture.X86_64,
+ "x86_64": Architecture.X86_64,
+ "i686-64": Architecture.X86_64,
+ "i386": Architecture.X86,
+ "i686": Architecture.X86,
+ "x86": Architecture.X86,
+ "ia64": Architecture.X86, # Itanium is different x64 arch, treat it as the common x86.
+ "powerpc": Architecture.PPC,
+ "power macintosh": Architecture.PPC,
+ "ppc64": Architecture.PPC,
+ "armv6": Architecture.ARM,
+ "armv6l": Architecture.ARM,
+ "arm64": Architecture.ARM,
+ "armv7": Architecture.ARM,
+ "armv7l": Architecture.ARM,
+}
+
+
+class AirflowInfo:
+ """All information related to Airflow, system and other."""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.airflow_version = airflow_version
+ self.system = SystemInfo(anonymizer)
+ self.tools = ToolsInfo(anonymizer)
+ self.paths = PathsInfo(anonymizer)
+ self.config = ConfigInfo(anonymizer)
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Apache Airflow [{version}]
+
+ {system}
+
+ {tools}
+
+ {paths}
+
+ {config}
+ """
+ )
+ .strip()
+ .format(
+ version=self.airflow_version,
+ system=self.system,
+ tools=self.tools,
+ paths=self.paths,
+ config=self.config,
+ )
+ )
+
+
+class SystemInfo:
+ """Basic system and python information"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.operating_system = OperatingSystem.get_current()
+ self.arch = Architecture.get_current()
+ self.uname = platform.uname()
+ self.locale = locale.getdefaultlocale()
+ self.python_location = anonymizer.process_path(sys.executable)
+ self.python_version = sys.version.replace("\n", " ")
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Platform: [{os}, {arch}] {uname}
+ Locale: {locale}
+ Python Version: [{python_version}]
+ Python Location: [{python_location}]
+ """
+ )
+ .strip()
+ .format(
+ os=self.operating_system or "NOT AVAILABLE",
+ arch=self.arch or "NOT AVAILABLE",
+ uname=self.uname,
+ locale=self.locale,
+ python_version=self.python_version,
+ python_location=self.python_location,
+ )
+ )
+
+
+class PathsInfo:
+ """Path informaation"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ system_path = os.environ.get("PATH", "").split(os.pathsep)
+
+ self.airflow_home = anonymizer.process_path(configuration.get_airflow_home())
+ self.system_path = [anonymizer.process_path(p) for p in system_path]
+ self.python_path = [anonymizer.process_path(p) for p in sys.path]
+ self.airflow_on_path = any(
+ os.path.exists(os.path.join(path_elem, "airflow")) for path_elem in system_path
+ )
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Airflow Home: [{airflow_home}]
+ System PATH: [{system_path}]
+ Python PATH: [{python_path}]
+ airflow on PATH: [{airflow_on_path}]
+ """
+ )
+ .strip()
+ .format(
+ airflow_home=self.airflow_home,
+ system_path=os.pathsep.join(self.system_path),
+ python_path=os.pathsep.join(self.python_path),
+ airflow_on_path=self.airflow_on_path,
+ )
+ )
+
+
+class ConfigInfo:
+ """"Most critical config properties"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.executor = configuration.conf.get("core", "executor")
+ self.dags_folder = anonymizer.process_path(
+ configuration.conf.get("core", "dags_folder", fallback="NOT AVAILABLE")
+ )
+ self.plugins_folder = anonymizer.process_path(
+ configuration.conf.get("core", "plugins_folder", fallback="NOT AVAILABLE")
+ )
+ self.base_log_folder = anonymizer.process_path(
+ configuration.conf.get("logging", "base_log_folder", fallback="NOT AVAILABLE")
+ )
+ self.base_log_folder = anonymizer.process_path(
+ configuration.conf.get("logging", "base_log_folder", fallback="NOT AVAILABLE")
+ )
+ self.sql_alchemy_conn = anonymizer.process_url(
+ configuration.conf.get("core", "SQL_ALCHEMY_CONN", fallback="NOT AVAILABLE")
+ )
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Executor: [{executor}]
+ SQL Alchemy Conn: [{sql_alchemy_conn}]
+ DAGS Folder: [{dags_folder}]
+ Plugins Folder: [{plugins_folder}]
+ Base Log Folder: [{base_log_folder}]
+ """
+ )
+ .strip()
+ .format(
+ executor=self.executor,
+ sql_alchemy_conn=self.sql_alchemy_conn,
+ dags_folder=self.dags_folder,
+ plugins_folder=self.plugins_folder,
+ base_log_folder=self.base_log_folder,
+ )
+ )
+
+
+class ToolsInfo:
+ """The versions of the tools that Airflow uses"""
+
+ def __init__(self, anonymize: Anonymizer):
+ del anonymize # Nothing to anonymize here.
+ self.git_version = self._get_version(["git", "--version"])
+ self.ssh_version = self._get_version(["ssh", "-V"])
+ self.kubectl_version = self._get_version(["kubectl", "version", "--short=True", "--client=True"])
+ self.gcloud_version = self._get_version(["gcloud", "version"], grep=b"Google Cloud SDK")
+ self.cloud_sql_proxy_version = self._get_version(["cloud_sql_proxy", "--version"])
+ self.mysql_version = self._get_version(["mysql", "--version"])
+ self.sqlite3_version = self._get_version(["sqlite3", "--version"])
+ self.psql_version = self._get_version(["psql", "--version"])
Review comment:
Why did we pick psql, and not the version of psycopg2, say?
(Airflow never uses `psql`)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj edited a comment on issue #8290: Add airflow info
command
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613481597
Hello,
I do the following changes:
* add tests
* add `--file-io` flags
* add SQL Alchemy conn with anonymization (`postgresql+psycopg2://postgres:airflow@postgres/airflow` => `postgresql+psycopg2://p...s:PASSWORD@postgres/airflow`)
Example output:
<details>
```
Apache Airflow [2.0.0.dev0]
Platform: [Linux, x86_64] uname_result(system='Linux', node='eeefbb427a9c', release='4.19.76-linuxkit', version='#1 SMP Thu Oct 17 19:31:58 UTC 2019', machine='x86_64', processor='')
Locale: ('en_US', 'UTF-8')
Python Version: [3.6.10 (default, Mar 31 2020, 16:33:25) [GCC 8.3.0]]
Python Location: [/usr/local/bin/python]
git: [git version 2.20.1]
ssh: [OpenSSH_8.2p1 Debian-4, OpenSSL 1.1.1d 10 Sep 2019]
kubectl: [Client Version: v1.15.3]
gcloud: [Google Cloud SDK 274.0.1]
cloud_sql_proxy: [NOT AVAILABLE]
mysql: [mysql Ver 14.14 Distrib 5.7.29, for Linux (x86_64) using EditLine wrapper]
sqlite3: [3.31.1 2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837balt1]
psql: [psql (PostgreSQL) 12.2 (Debian 12.2-4)]
Airflow Home: [${HOME}/airflow]
System PATH: [${HOME}:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/hive/bin:/usr/local/go/bin:/opt/gcloud/bin]
Python PATH: [/usr/local/bin:/opt/airflow/scripts/perf:/usr/local/lib/python36.zip:/usr/local/lib/python3.6:/usr/local/lib/python3.6/lib-dynload:/usr/local/lib/python3.6/site-packages:/opt/airflow:/opt/airflow/airflow/providers/google/cloud/example_dags:${HOME}/airflow/config:${HOME}/airflow/plugins]
airflow on PATH: [True]
Executor: [SequentialExecutor]
SQL Alchemy Conn: [postgresql+psycopg2://p...s:PASSWORD@postgres/airflow]
DAGS Folder: [/opt/airflow/airflow/providers/google/cloud/example_dags]
Plugins Folder: [${HOME}/airflow/plugins]
Base Log Folder: [${HOME}/airflow/logs]
```
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8290:
Add airflow info command
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r407942261
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,318 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def process_path(self, value):
+ return value
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
+
+
+_MACHINE_TO_ARCHITECTURE = {
+ "amd64": Architecture.X86_64,
+ "x86_64": Architecture.X86_64,
+ "i686-64": Architecture.X86_64,
+ "i386": Architecture.X86,
+ "i686": Architecture.X86,
+ "x86": Architecture.X86,
+ "ia64": Architecture.X86, # Itanium is different x64 arch, treat it as the common x86.
+ "powerpc": Architecture.PPC,
+ "power macintosh": Architecture.PPC,
+ "ppc64": Architecture.PPC,
+ "armv6": Architecture.ARM,
+ "armv6l": Architecture.ARM,
+ "arm64": Architecture.ARM,
+ "armv7": Architecture.ARM,
+ "armv7l": Architecture.ARM,
+}
Review comment:
@mik-laj have you checked if there's already a nice package to collect such information?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8290: Add airflow
info command
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613226464
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=h1) Report
> Merging [#8290](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45c8983306ab1c54abdacd8f870e790fad25cb37&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `1.69%`.
[](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8290 +/- ##
==========================================
- Coverage 88.45% 87.95% -0.50%
==========================================
Files 937 938 +1
Lines 45234 45352 +118
==========================================
- Hits 40011 39891 -120
- Misses 5223 5461 +238
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/cli/commands/info\_command.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvaW5mb19jb21tYW5kLnB5) | `0.00% <0.00%> (ø)` | |
| [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `97.30% <100.00%> (+0.02%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `75.18% <0.00%> (-3.76%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=footer). Last update [45c8983...437dae1](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8290:
Add airflow info command
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r407942261
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,318 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def process_path(self, value):
+ return value
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
+
+
+_MACHINE_TO_ARCHITECTURE = {
+ "amd64": Architecture.X86_64,
+ "x86_64": Architecture.X86_64,
+ "i686-64": Architecture.X86_64,
+ "i386": Architecture.X86,
+ "i686": Architecture.X86,
+ "x86": Architecture.X86,
+ "ia64": Architecture.X86, # Itanium is different x64 arch, treat it as the common x86.
+ "powerpc": Architecture.PPC,
+ "power macintosh": Architecture.PPC,
+ "ppc64": Architecture.PPC,
+ "armv6": Architecture.ARM,
+ "armv6l": Architecture.ARM,
+ "arm64": Architecture.ARM,
+ "armv7": Architecture.ARM,
+ "armv7l": Architecture.ARM,
+}
Review comment:
@mik-laj have you checked if there's already a nice package to collect such information?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8290: Add airflow
info command
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613226464
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=h1) Report
> Merging [#8290](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45c8983306ab1c54abdacd8f870e790fad25cb37&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `1.69%`.
[](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8290 +/- ##
==========================================
- Coverage 88.45% 87.95% -0.50%
==========================================
Files 937 938 +1
Lines 45234 45352 +118
==========================================
- Hits 40011 39891 -120
- Misses 5223 5461 +238
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/cli/commands/info\_command.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvaW5mb19jb21tYW5kLnB5) | `0.00% <0.00%> (ø)` | |
| [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `97.30% <100.00%> (+0.02%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `75.18% <0.00%> (-3.76%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=footer). Last update [45c8983...437dae1](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] codecov-io edited a comment on issue #8290: Add airflow
info command
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613226464
# [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=h1) Report
> Merging [#8290](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=desc) into [master](https://codecov.io/gh/apache/airflow/commit/45c8983306ab1c54abdacd8f870e790fad25cb37&el=desc) will **decrease** coverage by `0.49%`.
> The diff coverage is `1.69%`.
[](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #8290 +/- ##
==========================================
- Coverage 88.45% 87.95% -0.50%
==========================================
Files 937 938 +1
Lines 45234 45352 +118
==========================================
- Hits 40011 39891 -120
- Misses 5223 5461 +238
```
| [Impacted Files](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [airflow/cli/commands/info\_command.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY29tbWFuZHMvaW5mb19jb21tYW5kLnB5) | `0.00% <0.00%> (ø)` | |
| [airflow/cli/cli\_parser.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9jbGkvY2xpX3BhcnNlci5weQ==) | `97.30% <100.00%> (+0.02%)` | :arrow_up: |
| [airflow/kubernetes/volume\_mount.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZV9tb3VudC5weQ==) | `44.44% <0.00%> (-55.56%)` | :arrow_down: |
| [airflow/kubernetes/volume.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3ZvbHVtZS5weQ==) | `52.94% <0.00%> (-47.06%)` | :arrow_down: |
| [airflow/kubernetes/pod\_launcher.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3BvZF9sYXVuY2hlci5weQ==) | `47.18% <0.00%> (-45.08%)` | :arrow_down: |
| [...viders/cncf/kubernetes/operators/kubernetes\_pod.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9wcm92aWRlcnMvY25jZi9rdWJlcm5ldGVzL29wZXJhdG9ycy9rdWJlcm5ldGVzX3BvZC5weQ==) | `69.69% <0.00%> (-25.26%)` | :arrow_down: |
| [airflow/kubernetes/refresh\_config.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy9rdWJlcm5ldGVzL3JlZnJlc2hfY29uZmlnLnB5) | `50.98% <0.00%> (-23.53%)` | :arrow_down: |
| [airflow/utils/process\_utils.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9wcm9jZXNzX3V0aWxzLnB5) | `75.18% <0.00%> (-3.76%)` | :arrow_down: |
| [airflow/utils/dag\_processing.py](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree#diff-YWlyZmxvdy91dGlscy9kYWdfcHJvY2Vzc2luZy5weQ==) | `87.94% <0.00%> (-0.76%)` | :arrow_down: |
| ... and [2 more](https://codecov.io/gh/apache/airflow/pull/8290/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=footer). Last update [45c8983...437dae1](https://codecov.io/gh/apache/airflow/pull/8290?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r407964667
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,318 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def process_path(self, value):
+ return value
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
Review comment:
And we don't display None. Missing values is handled in ``__str__``. https://github.com/apache/airflow/pull/8290/files#diff-1877860355e4eb978a17b0b48b6ba073R178
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8290: Add airflow info command
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613634169
LGTM! Great tool that we shoudl start using soon! Maybe a good candidate for 1.10.11 cherry-pick.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on issue #8290: Add airflow info command
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-614565074
Nice, should help debugging a lot
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8290:
Add airflow info command
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r407943642
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,318 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def process_path(self, value):
+ return value
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
Review comment:
Should we use `platform.machine().lower()` as fallback value in `get`? It's probably better than None
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r408075141
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,318 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def process_path(self, value):
+ return value
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
Review comment:
Yeah. Since we have all the values printed directly, having fixed set of values in here is good for any kind of automation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r408284496
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -310,9 +375,50 @@ def __str__(self):
)
+class FileIoException(Exception):
+ """Raises when error happens in FileIo.io integration"""
+
+ pass
+
+
+@tenacity.retry(
+ stop=tenacity.stop_after_attempt(5),
+ wait=tenacity.wait_exponential(multiplier=1, max=10),
+ retry=tenacity.retry_if_exception_type(FileIoException),
+ before=tenacity.before_log(log, logging.DEBUG),
+ after=tenacity.after_log(log, logging.DEBUG),
+)
+def upload_text_to_fileio(content):
+ """Uload text file to File.io service and return lnk"""
+ resp = requests.post("https://file.io", files={"file": ("airflow-report.txt", content)})
+ if not resp.ok:
+ raise FileIoException("Failed to send report to file.io service.")
+ try:
+ return resp.json()["link"]
+ except ValueError as e:
+ log.debug(e)
+ raise FileIoException("Failed to send report to file.io service.")
+
+
+def send_report_to_fileio(info):
+ print("Uploading report to file.io service.")
+ try:
+ link = upload_text_to_fileio(str(info))
+ print("Report uploaded.")
+ print()
+ print("Link:\t", link)
+ print()
+ except FileIoException as ex:
+ print(str(ex))
+
+
def show_info(args):
"""
Show information related to Airflow, system and other.
"""
anonymizer = PiiAnonymizer() if args.anonymize else NullAnonymizer()
- print(AirflowInfo(anonymizer))
+ info = AirflowInfo(anonymizer)
+ if args.file_io:
Review comment:
Agree. I updated the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r409476067
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,427 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import logging
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+from urllib.parse import urlsplit, urlunsplit
+
+import requests
+import tenacity
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+log = logging.getLogger(__name__)
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+ def process_username(self, value):
+ """Remove pii from ussername"""
+
+ def process_url(self, value):
+ """Remove pii from URL"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def _identity(self, value):
+ return value
+
+ process_path = process_username = process_url = _identity
+
+ del _identity
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+ def process_username(self, value):
+ if not value:
+ return value
+ return value[0] + "..." + value[-1]
+
+ def process_url(self, value):
+ if not value:
+ return value
+
+ url_parts = urlsplit(value)
+ netloc = None
+ if url_parts.netloc:
+ # unpack
+ userinfo = None
+ host = None
+ username = None
+ password = None
+
+ if "@" in url_parts.netloc:
+ userinfo, _, host = url_parts.netloc.partition("@")
+ else:
+ host = url_parts.netloc
+ if userinfo:
+ if ":" in userinfo:
+ username, _, password = userinfo.partition(":")
+ else:
+ username = userinfo
+
+ # anonymize
+ username = self.process_username(username) if username else None
+ password = "PASSWORD" if password else None
+
+ # pack
+ if username and password and host:
+ netloc = username + ":" + password + "@" + host
+ elif username and host:
+ netloc = username + "@" + host
+ elif password and host:
+ netloc = ":" + password + "@" + host
+ elif host:
+ netloc = host
+ else:
+ netloc = ""
+
+ return urlunsplit((url_parts.scheme, netloc, url_parts.path, url_parts.query, url_parts.fragment))
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
+
+
+_MACHINE_TO_ARCHITECTURE = {
+ "amd64": Architecture.X86_64,
+ "x86_64": Architecture.X86_64,
+ "i686-64": Architecture.X86_64,
+ "i386": Architecture.X86,
+ "i686": Architecture.X86,
+ "x86": Architecture.X86,
+ "ia64": Architecture.X86, # Itanium is different x64 arch, treat it as the common x86.
+ "powerpc": Architecture.PPC,
+ "power macintosh": Architecture.PPC,
+ "ppc64": Architecture.PPC,
+ "armv6": Architecture.ARM,
+ "armv6l": Architecture.ARM,
+ "arm64": Architecture.ARM,
+ "armv7": Architecture.ARM,
+ "armv7l": Architecture.ARM,
+}
+
+
+class AirflowInfo:
+ """All information related to Airflow, system and other."""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.airflow_version = airflow_version
+ self.system = SystemInfo(anonymizer)
+ self.tools = ToolsInfo(anonymizer)
+ self.paths = PathsInfo(anonymizer)
+ self.config = ConfigInfo(anonymizer)
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Apache Airflow [{version}]
+
+ {system}
+
+ {tools}
+
+ {paths}
+
+ {config}
+ """
+ )
+ .strip()
+ .format(
+ version=self.airflow_version,
+ system=self.system,
+ tools=self.tools,
+ paths=self.paths,
+ config=self.config,
+ )
+ )
+
+
+class SystemInfo:
+ """Basic system and python information"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.operating_system = OperatingSystem.get_current()
+ self.arch = Architecture.get_current()
+ self.uname = platform.uname()
+ self.locale = locale.getdefaultlocale()
+ self.python_location = anonymizer.process_path(sys.executable)
+ self.python_version = sys.version.replace("\n", " ")
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Platform: [{os}, {arch}] {uname}
+ Locale: {locale}
+ Python Version: [{python_version}]
+ Python Location: [{python_location}]
+ """
+ )
+ .strip()
+ .format(
+ os=self.operating_system or "NOT AVAILABLE",
+ arch=self.arch or "NOT AVAILABLE",
+ uname=self.uname,
+ locale=self.locale,
+ python_version=self.python_version,
+ python_location=self.python_location,
+ )
+ )
+
+
+class PathsInfo:
+ """Path informaation"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ system_path = os.environ.get("PATH", "").split(os.pathsep)
+
+ self.airflow_home = anonymizer.process_path(configuration.get_airflow_home())
+ self.system_path = [anonymizer.process_path(p) for p in system_path]
+ self.python_path = [anonymizer.process_path(p) for p in sys.path]
+ self.airflow_on_path = any(
+ os.path.exists(os.path.join(path_elem, "airflow")) for path_elem in system_path
+ )
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Airflow Home: [{airflow_home}]
+ System PATH: [{system_path}]
+ Python PATH: [{python_path}]
+ airflow on PATH: [{airflow_on_path}]
+ """
+ )
+ .strip()
+ .format(
+ airflow_home=self.airflow_home,
+ system_path=os.pathsep.join(self.system_path),
+ python_path=os.pathsep.join(self.python_path),
+ airflow_on_path=self.airflow_on_path,
+ )
+ )
+
+
+class ConfigInfo:
+ """"Most critical config properties"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.executor = configuration.conf.get("core", "executor")
+ self.dags_folder = anonymizer.process_path(
+ configuration.conf.get("core", "dags_folder", fallback="NOT AVAILABLE")
+ )
+ self.plugins_folder = anonymizer.process_path(
+ configuration.conf.get("core", "plugins_folder", fallback="NOT AVAILABLE")
+ )
+ self.base_log_folder = anonymizer.process_path(
+ configuration.conf.get("logging", "base_log_folder", fallback="NOT AVAILABLE")
+ )
+ self.base_log_folder = anonymizer.process_path(
+ configuration.conf.get("logging", "base_log_folder", fallback="NOT AVAILABLE")
+ )
+ self.sql_alchemy_conn = anonymizer.process_url(
+ configuration.conf.get("core", "SQL_ALCHEMY_CONN", fallback="NOT AVAILABLE")
+ )
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Executor: [{executor}]
+ SQL Alchemy Conn: [{sql_alchemy_conn}]
+ DAGS Folder: [{dags_folder}]
+ Plugins Folder: [{plugins_folder}]
+ Base Log Folder: [{base_log_folder}]
+ """
+ )
+ .strip()
+ .format(
+ executor=self.executor,
+ sql_alchemy_conn=self.sql_alchemy_conn,
+ dags_folder=self.dags_folder,
+ plugins_folder=self.plugins_folder,
+ base_log_folder=self.base_log_folder,
+ )
+ )
+
+
+class ToolsInfo:
+ """The versions of the tools that Airflow uses"""
+
+ def __init__(self, anonymize: Anonymizer):
+ del anonymize # Nothing to anonymize here.
+ self.git_version = self._get_version(["git", "--version"])
+ self.ssh_version = self._get_version(["ssh", "-V"])
+ self.kubectl_version = self._get_version(["kubectl", "version", "--short=True", "--client=True"])
+ self.gcloud_version = self._get_version(["gcloud", "version"], grep=b"Google Cloud SDK")
+ self.cloud_sql_proxy_version = self._get_version(["cloud_sql_proxy", "--version"])
+ self.mysql_version = self._get_version(["mysql", "--version"])
+ self.sqlite3_version = self._get_version(["sqlite3", "--version"])
+ self.psql_version = self._get_version(["psql", "--version"])
Review comment:
Airflow uses psql here: https://github.com/apache/airflow/blob/master/airflow/cli/commands/db_command.py#L91
I did not check any version of the Python library. For this, we have another command - `pip freeze`. In the future, we can add selected libraries if necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #8290: Add airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613481597
Hello,
I do the following changes:
* add tests
* add `file-io` flags
* add SQL Alchemy conn with anonymization (`postgresql+psycopg2://postgres:airflow@postgres/airflow` => `postgresql+psycopg2://p...s:PASSWORD@postgres/airflow`)
Example output:
<details>
```
Apache Airflow [2.0.0.dev0]
Platform: [Linux, x86_64] uname_result(system='Linux', node='eeefbb427a9c', release='4.19.76-linuxkit', version='#1 SMP Thu Oct 17 19:31:58 UTC 2019', machine='x86_64', processor='')
Locale: ('en_US', 'UTF-8')
Python Version: [3.6.10 (default, Mar 31 2020, 16:33:25) [GCC 8.3.0]]
Python Location: [/usr/local/bin/python]
git: [git version 2.20.1]
ssh: [OpenSSH_8.2p1 Debian-4, OpenSSL 1.1.1d 10 Sep 2019]
kubectl: [Client Version: v1.15.3]
gcloud: [Google Cloud SDK 274.0.1]
cloud_sql_proxy: [NOT AVAILABLE]
mysql: [mysql Ver 14.14 Distrib 5.7.29, for Linux (x86_64) using EditLine wrapper]
sqlite3: [3.31.1 2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837balt1]
psql: [psql (PostgreSQL) 12.2 (Debian 12.2-4)]
Airflow Home: [${HOME}/airflow]
System PATH: [${HOME}:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/hive/bin:/usr/local/go/bin:/opt/gcloud/bin]
Python PATH: [/usr/local/bin:/opt/airflow/scripts/perf:/usr/local/lib/python36.zip:/usr/local/lib/python3.6:/usr/local/lib/python3.6/lib-dynload:/usr/local/lib/python3.6/site-packages:/opt/airflow:/opt/airflow/airflow/providers/google/cloud/example_dags:${HOME}/airflow/config:${HOME}/airflow/plugins]
airflow on PATH: [True]
Executor: [SequentialExecutor]
SQL Alchemy Conn: [postgresql+psycopg2://p...s:PASSWORD@postgres/airflow]
DAGS Folder: [/opt/airflow/airflow/providers/google/cloud/example_dags]
Plugins Folder: [${HOME}/airflow/plugins]
Base Log Folder: [${HOME}/airflow/logs]
```
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on issue #8290: Add airflow info command
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613400920
https://www.file.io/ -> I am using that to submit logs from our CI builds.
They have a nice feature that it's truly ephemeral - i.e. only one person can download it after it's posted and the files disappear after 2 weeks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] turbaszek commented on a change in pull request #8290:
Add airflow info command
Posted by GitBox <gi...@apache.org>.
turbaszek commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r407942261
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,318 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def process_path(self, value):
+ return value
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
+
+
+_MACHINE_TO_ARCHITECTURE = {
+ "amd64": Architecture.X86_64,
+ "x86_64": Architecture.X86_64,
+ "i686-64": Architecture.X86_64,
+ "i386": Architecture.X86,
+ "i686": Architecture.X86,
+ "x86": Architecture.X86,
+ "ia64": Architecture.X86, # Itanium is different x64 arch, treat it as the common x86.
+ "powerpc": Architecture.PPC,
+ "power macintosh": Architecture.PPC,
+ "ppc64": Architecture.PPC,
+ "armv6": Architecture.ARM,
+ "armv6l": Architecture.ARM,
+ "arm64": Architecture.ARM,
+ "armv7": Architecture.ARM,
+ "armv7l": Architecture.ARM,
+}
Review comment:
@mik-laj have you checked if there's already a nice package to collect such information? Or at least should consider using https://docs.python.org/3/library/platform.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj commented on issue #8290: Add airflow info command
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#issuecomment-613399205
@potiuk Do you know any service that allows us to add notes anonymously? I suspect that most services will require an API KEY.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] mik-laj merged pull request #8290: Add airflow info
command
Posted by GitBox <gi...@apache.org>.
mik-laj merged pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] ashb commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r409452447
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,427 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import logging
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+from urllib.parse import urlsplit, urlunsplit
+
+import requests
+import tenacity
+from typing_extensions import Protocol
Review comment:
This should be `from airflow.typing_compat import Protocol` -- on Py 3.8+ it will use the built in one then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [airflow] potiuk commented on a change in pull request #8290: Add
airflow info command
Posted by GitBox <gi...@apache.org>.
potiuk commented on a change in pull request #8290: Add airflow info command
URL: https://github.com/apache/airflow/pull/8290#discussion_r408073303
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -0,0 +1,318 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""Config sub-commands"""
+import getpass
+import locale
+import os
+import platform
+import subprocess
+import sys
+import textwrap
+from typing import Optional
+
+from typing_extensions import Protocol
+
+from airflow import configuration
+from airflow.version import version as airflow_version
+
+
+class Anonymizer(Protocol):
+ """Anonymizer protocol."""
+
+ def process_path(self, value):
+ """Remove pii from paths"""
+
+
+class NullAnonymizer(Anonymizer):
+ """Do nothing."""
+
+ def process_path(self, value):
+ return value
+
+
+class PiiAnonymizer(Anonymizer):
+ """Remove personally identifiable info from path."""
+
+ def __init__(self):
+ home_path = os.path.expanduser("~")
+ username = getpass.getuser()
+ self._path_replacements = {home_path: "${HOME}", username: "${USER}"}
+
+ def process_path(self, value):
+ if not value:
+ return value
+ for src, target in self._path_replacements.items():
+ value = value.replace(src, target)
+ return value
+
+
+class OperatingSystem:
+ """Operating system"""
+
+ WINDOWS = "Windows"
+ LINUX = "Linux"
+ MACOSX = "Mac OS"
+ CYGWIN = "Cygwin"
+
+ @staticmethod
+ def get_current() -> Optional[str]:
+ """Get current operating system"""
+ if os.name == "nt":
+ return OperatingSystem.WINDOWS
+ elif "linux" in sys.platform:
+ return OperatingSystem.LINUX
+ elif "darwin" in sys.platform:
+ return OperatingSystem.MACOSX
+ elif "cygwin" in sys.platform:
+ return OperatingSystem.CYGWIN
+ return None
+
+
+class Architecture:
+ """Compute architecture"""
+
+ X86_64 = "x86_64"
+ X86 = "x86"
+ PPC = "ppc"
+ ARM = "arm"
+
+ @staticmethod
+ def get_current():
+ """Get architecture"""
+ return _MACHINE_TO_ARCHITECTURE.get(platform.machine().lower())
+
+
+_MACHINE_TO_ARCHITECTURE = {
+ "amd64": Architecture.X86_64,
+ "x86_64": Architecture.X86_64,
+ "i686-64": Architecture.X86_64,
+ "i386": Architecture.X86,
+ "i686": Architecture.X86,
+ "x86": Architecture.X86,
+ "ia64": Architecture.X86, # Itanium is different x64 arch, treat it as the common x86.
+ "powerpc": Architecture.PPC,
+ "power macintosh": Architecture.PPC,
+ "ppc64": Architecture.PPC,
+ "armv6": Architecture.ARM,
+ "armv6l": Architecture.ARM,
+ "arm64": Architecture.ARM,
+ "armv7": Architecture.ARM,
+ "armv7l": Architecture.ARM,
+}
+
+
+class AirflowInfo:
+ """All information related to Airflow, system and other."""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.airflow_version = airflow_version
+ self.system = SystemInfo(anonymizer)
+ self.tools = ToolsInfo(anonymizer)
+ self.paths = PathsInfo(anonymizer)
+ self.config = ConfigInfo(anonymizer)
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Apache Airflow [{version}]
+
+ {system}
+
+ {tools}
+
+ {paths}
+
+ {config}
+ """
+ )
+ .strip()
+ .format(
+ version=airflow_version,
+ system=self.system,
+ tools=self.tools,
+ paths=self.paths,
+ config=self.config,
+ )
+ )
+
+
+class SystemInfo:
+ """Basic system and python information"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ self.operating_system = OperatingSystem.get_current()
+ self.arch = Architecture.get_current()
+ self.uname = platform.uname()
+ self.locale = locale.getdefaultlocale()
+ self.python_location = anonymizer.process_path(sys.executable)
+ self.python_version = sys.version.replace("\n", " ")
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Platform: [{os}, {arch}] {uname}
+ Locale: {locale}
+ Python Version: [{python_version}]
+ Python Location: [{python_location}]
+ """
+ )
+ .strip()
+ .format(
+ version=airflow_version,
+ os=self.operating_system or "NOT AVAILABLE",
+ arch=self.arch or "NOT AVAILABLE",
+ uname=self.uname,
+ locale=self.locale,
+ python_version=self.python_version,
+ python_location=self.python_location,
+ )
+ )
+
+
+class PathsInfo:
+ """Path informaation"""
+
+ def __init__(self, anonymizer: Anonymizer):
+ system_path = os.environ.get("PATH", "").split(os.pathsep)
+
+ self.airflow_home = anonymizer.process_path(configuration.get_airflow_home())
+ self.system_path = [anonymizer.process_path(p) for p in system_path]
+ self.python_path = [anonymizer.process_path(p) for p in sys.path]
+ self.airflow_on_path = any(
+ os.path.exists(os.path.join(path_elem, "airflow")) for path_elem in system_path
+ )
+
+ def __str__(self):
+ return (
+ textwrap.dedent(
+ """\
+ Airflow Home: [{airflow_home}]
+ System PATH: [{system_path}]
+ Python PATH: [{python_path}]
Review comment:
I really like the considerations to add all anonymisations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services