You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/05 14:50:00 UTC
[GitHub] [airflow] mik-laj opened a new pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
mik-laj opened a new pull request #15206:
URL: https://github.com/apache/airflow/pull/15206
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r608209583
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -27,11 +27,11 @@
import requests
import tenacity
+from typing_extensions import Protocol
from airflow import configuration
from airflow.cli.simple_table import AirflowConsole
from airflow.providers_manager import ProvidersManager
-from airflow.typing_compat import Protocol
Review comment:
I install `typing_extensions` on every pythton version, because `sphinx-autoapi` has problems with it.
```
WARNING: Cannot resolve import of airflow.typing_compat.Protocol in airflow.stats
```
Hopefully we can fix this differently, but for now, I wanted to take a step forward to see if we had other critical issues. Now the documentation for the apache-airflow package builds up fine, so I think it's a pretty cool start.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#issuecomment-813438922
[The Workflow run](https://github.com/apache/airflow/actions/runs/719514094) is cancelling this PR. Building images for the PR has failed. Follow the workflow link to check the reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r607133504
##########
File path: airflow/providers/qubole/hooks/qubole.py
##########
@@ -103,6 +103,8 @@ def build_command_args() -> Tuple[Dict[str, list], list]:
return command_args, list(hyphen_args)
+COMMAND_ARGS: List[str]
+HYPHEN_ARGS: List[str]
Review comment:
We need to explicitly create these variables so that sphinx-autoapi can see and import these variables in other module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r607710835
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -27,11 +27,11 @@
import requests
import tenacity
+from typing_extensions import Protocol
from airflow import configuration
from airflow.cli.simple_table import AirflowConsole
from airflow.providers_manager import ProvidersManager
-from airflow.typing_compat import Protocol
Review comment:
This is done this way for compatibility across multiple python versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r607135114
##########
File path: docs/exts/docs_build/run_patched_sphinx.py
##########
@@ -0,0 +1,220 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+import logging as python_logging
+import os
+import sys
+
+import autoapi
+from autoapi.extension import (
+ DEFAULT_FILE_PATTERNS,
+ DEFAULT_IGNORE_PATTERNS,
+ LANGUAGE_MAPPERS,
+ LANGUAGE_REQUIREMENTS,
+ ExtensionError,
+ RemovedInAutoAPI2Warning,
+ _normalise_autoapi_dirs,
+ warnings,
+)
+from autoapi.mappers.python import mapper
+from sphinx.cmd.build import main
+from sphinx.util import logging as sphinx_logging
+
+
+def run_autoapi(app): # pylint: disable=too-many-branches
+ """
+ Load AutoAPI data from the filesystem.
+ """
+ if app.config.autoapi_type not in LANGUAGE_MAPPERS:
+ raise ExtensionError(
+ "Invalid autoapi_type setting, "
+ "following values is allowed: {}".format(
+ ", ".join(f'"{api_type}"' for api_type in sorted(LANGUAGE_MAPPERS))
+ )
+ )
+
+ if not app.config.autoapi_dirs:
+ raise ExtensionError("You must configure an autoapi_dirs setting")
+
+ if app.config.autoapi_include_summaries is not None:
+ warnings.warn(
+ "autoapi_include_summaries has been replaced by the show-module-summary AutoAPI option\n",
+ RemovedInAutoAPI2Warning,
+ )
+ if app.config.autoapi_include_summaries:
+ app.config.autoapi_options.append("show-module-summary")
+
+ # Make sure the paths are full
+ normalised_dirs = _normalise_autoapi_dirs(app.config.autoapi_dirs, app.confdir)
+ for _dir in normalised_dirs:
+ if not os.path.exists(_dir):
+ raise ExtensionError(
+ "AutoAPI Directory `{dir}` not found. "
+ "Please check your `autoapi_dirs` setting.".format(dir=_dir)
+ )
+
+ normalized_root = os.path.normpath(os.path.join(app.srcdir, app.config.autoapi_root))
+ url_root = os.path.join("/", app.config.autoapi_root)
+
+ if not all(
+ import_name in sys.modules for _, import_name in LANGUAGE_REQUIREMENTS[app.config.autoapi_type]
+ ):
+ raise ExtensionError(
+ "AutoAPI of type `{type}` requires following "
+ "packages to be installed and included in extensions list: "
+ "{packages}".format(
+ type=app.config.autoapi_type,
+ packages=", ".join(
+ '{import_name} (available as "{pkg_name}" on PyPI)'.format(
+ pkg_name=pkg_name, import_name=import_name
+ )
+ for pkg_name, import_name in LANGUAGE_REQUIREMENTS[app.config.autoapi_type]
+ ),
+ )
+ )
+
+ sphinx_mapper = LANGUAGE_MAPPERS[app.config.autoapi_type]
+ template_dir = app.config.autoapi_template_dir
+ if template_dir and not os.path.isabs(template_dir):
+ if not os.path.isdir(template_dir):
+ template_dir = os.path.join(app.confdir, app.config.autoapi_template_dir)
+ elif app.confdir != os.getcwd():
+ warnings.warn(
+ "autoapi_template_dir will be expected to be "
+ " relative to conf.py instead of "
+ "relative to where sphinx-build is run\n",
+ RemovedInAutoAPI2Warning,
+ )
+ sphinx_mapper_obj = sphinx_mapper(app, template_dir=template_dir, url_root=url_root)
+
+ if app.config.autoapi_file_patterns:
+ file_patterns = app.config.autoapi_file_patterns
+ else:
+ file_patterns = DEFAULT_FILE_PATTERNS.get(app.config.autoapi_type, [])
+
+ if app.config.autoapi_ignore:
+ ignore_patterns = app.config.autoapi_ignore
+ else:
+ ignore_patterns = DEFAULT_IGNORE_PATTERNS.get(app.config.autoapi_type, [])
+
+ if ".rst" in app.config.source_suffix:
+ out_suffix = ".rst"
+ elif ".txt" in app.config.source_suffix:
+ out_suffix = ".txt"
+ else:
+ # Fallback to first suffix listed
+ out_suffix = app.config.source_suffix[0]
+
+ if sphinx_mapper_obj.load(patterns=file_patterns, dirs=normalised_dirs, ignore=ignore_patterns):
+ sphinx_mapper_obj.map(options=app.config.autoapi_options)
+
+ if app.config.autoapi_generate_api_docs:
+ sphinx_mapper_obj.output_rst(root=normalized_root, source_suffix=out_suffix)
+
+ if app.config.autoapi_type == "python":
+ app.env.autoapi_objects = sphinx_mapper_obj.objects
+ app.env.autoapi_all_objects = sphinx_mapper_obj.all_objects
+
+
+# HACK: sphinx-auto map did not correctly use the confdir attribute instead of srcdir when specifying the
+# directory to contain the generated files.
+# Unfortunately we have a problem updating to a newer version of this library and we have to use
+# sphinx-autoapi v1.0.0, so I am monkeypatching this library to fix this one problem.
+autoapi.extension.run_autoapi = run_autoapi
+
+
+def _resolve_module_placeholders(modules, module_name, visit_path, resolved):
+ """Resolve all placeholder children under a module.
+
+ :param modules: A mapping of module names to their data dictionary.
+ Placeholders are resolved in place.
+ :type modules: dict(str, dict)
+ :param module_name: The name of the module to resolve.
+ :type module_name: str
+ :param visit_path: An ordered set of visited module names.
+ :type visited: collections.OrderedDict
+ :param resolved: A set of already resolved module names.
+ :type resolved: set(str)
+ """
+ if module_name in resolved:
+ return
+
+ visit_path[module_name] = True
+
+ module, children = modules[module_name]
+ for child in list(children.values()):
+ if child["type"] != "placeholder":
+ continue
+
+ if child["original_path"] in modules:
+ module["children"].remove(child)
+ children.pop(child["name"])
+ continue
+
+ imported_from, original_name = child["original_path"].rsplit(".", 1)
+ if imported_from in visit_path:
+ msg = f"Cannot resolve cyclic import: {', '.join(visit_path)}, {imported_from}"
+ mapper.LOGGER.warning(msg)
+ module["children"].remove(child)
+ children.pop(child["name"])
+ continue
+
+ if imported_from not in modules:
+ if not imported_from.startswith("airflow.providers") or module_name.startswith(
Review comment:
Here is the only change to this function. I added this if statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r607711504
##########
File path: airflow/providers/jira/sensors/jira.py
##########
@@ -17,9 +17,10 @@
# under the License.
from typing import Any, Callable, Dict, Optional
+from jira.exceptions import JIRAError
from jira.resources import Issue, Resource
-from airflow.providers.jira.operators.jira import JIRAError, JiraOperator
+from airflow.providers.jira.operators.jira import JiraOperator
Review comment:
What was the sort of error this gave with the import as it was?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r607133875
##########
File path: airflow/sensors/base.py
##########
@@ -322,9 +321,3 @@ def mode_setter(_, value):
return cls_type
return decorate(cls)
-
-
-if 'BUILDING_AIRFLOW_DOCS' in os.environ:
Review comment:
It's is duplicated code. See: https://github.com/apache/airflow/blob/4099108f554130cf3f87ba33b9d6084a74e70231/airflow/utils/decorators.py#L100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#issuecomment-813666957
[The Workflow run](https://github.com/apache/airflow/actions/runs/720460399) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r608206748
##########
File path: airflow/providers/jira/sensors/jira.py
##########
@@ -17,9 +17,10 @@
# under the License.
from typing import Any, Callable, Dict, Optional
+from jira.exceptions import JIRAError
from jira.resources import Issue, Resource
-from airflow.providers.jira.operators.jira import JIRAError, JiraOperator
+from airflow.providers.jira.operators.jira import JiraOperator
Review comment:
```
WARNING: Cannot resolve import of airflow.providers.jira.operators.jira.JIRAError in airflow.providers.jira.sensors.jira
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #15206:
URL: https://github.com/apache/airflow/pull/15206
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#issuecomment-814489232
[The Workflow run](https://github.com/apache/airflow/actions/runs/724151114) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r607132519
##########
File path: airflow/cli/commands/info_command.py
##########
@@ -27,11 +27,11 @@
import requests
import tenacity
+from typing_extensions import Protocol
from airflow import configuration
from airflow.cli.simple_table import AirflowConsole
from airflow.providers_manager import ProvidersManager
-from airflow.typing_compat import Protocol
Review comment:
We have to import Prootcol directly for sphinx-autoapi to understand this correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#issuecomment-814489351
[The Workflow run](https://github.com/apache/airflow/actions/runs/724156059) is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r607132747
##########
File path: airflow/providers/google/cloud/utils/credentials_provider.py
##########
@@ -36,10 +35,13 @@
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.utils.process_utils import patch_environ
-log = logging.getLogger(__name__)
+log: logging.Logger = logging.getLogger(__name__)
+"""Logger"""
AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT = "AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT"
+"""A"""
_DEFAULT_SCOPES: Sequence[str] = ('https://www.googleapis.com/auth/cloud-platform',)
+"""B"""
Review comment:
```suggestion
log: logging.Logger = logging.getLogger(__name__)
AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT = "AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT"
_DEFAULT_SCOPES: Sequence[str] = ('https://www.googleapis.com/auth/cloud-platform',)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
mik-laj commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r608223072
##########
File path: airflow/providers/google/cloud/utils/credentials_provider.py
##########
@@ -36,7 +35,7 @@
from airflow.utils.log.logging_mixin import LoggingMixin
from airflow.utils.process_utils import patch_environ
-log = logging.getLogger(__name__)
+log: logging.Logger = logging.getLogger(__name__)
Review comment:
```suggestion
log = logging.getLogger(__name__)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
ashb commented on pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#issuecomment-813990198
Ah they've finially fixed the pickle issue https://github.com/readthedocs/sphinx-autoapi/commit/7541f922ae862e7de052040ae69749006ac9692f that was stopping us upgrading, nice!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#issuecomment-813667286
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.0
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#issuecomment-846315960
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
mik-laj commented on pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#issuecomment-814478114
> Ah they've finially fixed the pickle issue readthedocs/sphinx-autoapi@7541f92 that was stopping us upgrading, nice!
I think I found a workaround for this problem. I deleted annotations that contained types equal to `None`. In most cases it is used incorrectly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on a change in pull request #15206: [WIP] Bump sphinx-autoapi from 1.0.0 to 1.7.5
Posted by GitBox <gi...@apache.org>.
ashb commented on a change in pull request #15206:
URL: https://github.com/apache/airflow/pull/15206#discussion_r607713445
##########
File path: docs/exts/docs_build/run_patched_sphinx.py
##########
@@ -0,0 +1,220 @@
+#!/usr/bin/env python
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+import logging as python_logging
+import os
+import sys
+
+import autoapi
+from autoapi.extension import (
+ DEFAULT_FILE_PATTERNS,
+ DEFAULT_IGNORE_PATTERNS,
+ LANGUAGE_MAPPERS,
+ LANGUAGE_REQUIREMENTS,
+ ExtensionError,
+ RemovedInAutoAPI2Warning,
+ _normalise_autoapi_dirs,
+ warnings,
+)
+from autoapi.mappers.python import mapper
+from sphinx.cmd.build import main
+from sphinx.util import logging as sphinx_logging
+
+
+def run_autoapi(app): # pylint: disable=too-many-branches
+ """
+ Load AutoAPI data from the filesystem.
+ """
+ if app.config.autoapi_type not in LANGUAGE_MAPPERS:
+ raise ExtensionError(
+ "Invalid autoapi_type setting, "
+ "following values is allowed: {}".format(
+ ", ".join(f'"{api_type}"' for api_type in sorted(LANGUAGE_MAPPERS))
+ )
+ )
+
+ if not app.config.autoapi_dirs:
+ raise ExtensionError("You must configure an autoapi_dirs setting")
+
+ if app.config.autoapi_include_summaries is not None:
+ warnings.warn(
+ "autoapi_include_summaries has been replaced by the show-module-summary AutoAPI option\n",
+ RemovedInAutoAPI2Warning,
+ )
+ if app.config.autoapi_include_summaries:
+ app.config.autoapi_options.append("show-module-summary")
+
+ # Make sure the paths are full
+ normalised_dirs = _normalise_autoapi_dirs(app.config.autoapi_dirs, app.confdir)
+ for _dir in normalised_dirs:
+ if not os.path.exists(_dir):
+ raise ExtensionError(
+ "AutoAPI Directory `{dir}` not found. "
+ "Please check your `autoapi_dirs` setting.".format(dir=_dir)
+ )
+
+ normalized_root = os.path.normpath(os.path.join(app.srcdir, app.config.autoapi_root))
+ url_root = os.path.join("/", app.config.autoapi_root)
+
+ if not all(
+ import_name in sys.modules for _, import_name in LANGUAGE_REQUIREMENTS[app.config.autoapi_type]
+ ):
+ raise ExtensionError(
+ "AutoAPI of type `{type}` requires following "
+ "packages to be installed and included in extensions list: "
+ "{packages}".format(
+ type=app.config.autoapi_type,
+ packages=", ".join(
+ '{import_name} (available as "{pkg_name}" on PyPI)'.format(
+ pkg_name=pkg_name, import_name=import_name
+ )
+ for pkg_name, import_name in LANGUAGE_REQUIREMENTS[app.config.autoapi_type]
+ ),
+ )
+ )
+
+ sphinx_mapper = LANGUAGE_MAPPERS[app.config.autoapi_type]
+ template_dir = app.config.autoapi_template_dir
+ if template_dir and not os.path.isabs(template_dir):
+ if not os.path.isdir(template_dir):
+ template_dir = os.path.join(app.confdir, app.config.autoapi_template_dir)
+ elif app.confdir != os.getcwd():
+ warnings.warn(
+ "autoapi_template_dir will be expected to be "
+ " relative to conf.py instead of "
+ "relative to where sphinx-build is run\n",
+ RemovedInAutoAPI2Warning,
+ )
+ sphinx_mapper_obj = sphinx_mapper(app, template_dir=template_dir, url_root=url_root)
+
+ if app.config.autoapi_file_patterns:
+ file_patterns = app.config.autoapi_file_patterns
+ else:
+ file_patterns = DEFAULT_FILE_PATTERNS.get(app.config.autoapi_type, [])
+
+ if app.config.autoapi_ignore:
+ ignore_patterns = app.config.autoapi_ignore
+ else:
+ ignore_patterns = DEFAULT_IGNORE_PATTERNS.get(app.config.autoapi_type, [])
+
+ if ".rst" in app.config.source_suffix:
+ out_suffix = ".rst"
+ elif ".txt" in app.config.source_suffix:
+ out_suffix = ".txt"
+ else:
+ # Fallback to first suffix listed
+ out_suffix = app.config.source_suffix[0]
+
+ if sphinx_mapper_obj.load(patterns=file_patterns, dirs=normalised_dirs, ignore=ignore_patterns):
+ sphinx_mapper_obj.map(options=app.config.autoapi_options)
+
+ if app.config.autoapi_generate_api_docs:
+ sphinx_mapper_obj.output_rst(root=normalized_root, source_suffix=out_suffix)
+
+ if app.config.autoapi_type == "python":
+ app.env.autoapi_objects = sphinx_mapper_obj.objects
+ app.env.autoapi_all_objects = sphinx_mapper_obj.all_objects
+
+
+# HACK: sphinx-auto map did not correctly use the confdir attribute instead of srcdir when specifying the
+# directory to contain the generated files.
+# Unfortunately we have a problem updating to a newer version of this library and we have to use
+# sphinx-autoapi v1.0.0, so I am monkeypatching this library to fix this one problem.
+autoapi.extension.run_autoapi = run_autoapi
+
+
+def _resolve_module_placeholders(modules, module_name, visit_path, resolved):
+ """Resolve all placeholder children under a module.
+
+ :param modules: A mapping of module names to their data dictionary.
+ Placeholders are resolved in place.
+ :type modules: dict(str, dict)
+ :param module_name: The name of the module to resolve.
+ :type module_name: str
+ :param visit_path: An ordered set of visited module names.
+ :type visited: collections.OrderedDict
+ :param resolved: A set of already resolved module names.
+ :type resolved: set(str)
+ """
+ if module_name in resolved:
+ return
+
+ visit_path[module_name] = True
+
+ module, children = modules[module_name]
+ for child in list(children.values()):
+ if child["type"] != "placeholder":
+ continue
+
+ if child["original_path"] in modules:
+ module["children"].remove(child)
+ children.pop(child["name"])
+ continue
+
+ imported_from, original_name = child["original_path"].rsplit(".", 1)
+ if imported_from in visit_path:
+ msg = f"Cannot resolve cyclic import: {', '.join(visit_path)}, {imported_from}"
+ mapper.LOGGER.warning(msg)
+ module["children"].remove(child)
+ children.pop(child["name"])
+ continue
+
+ if imported_from not in modules:
+ if not imported_from.startswith("airflow.providers") or module_name.startswith(
Review comment:
If we need to keep this function could you "bracket" this added/changed section with a `# begin change`, `# end change` please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org