You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2024/12/28 01:58:04 UTC
[PR] Move first provider (airbyte) to a separate project [airflow]
potiuk opened a new pull request, #45259:
URL: https://github.com/apache/airflow/pull/45259
This is the first step to move providers to separate projects inside our monorepo.
Airbyte is a first provider that is separated out to a project under "providers/<PROVIDER_ID>" directory. This has the nice property that all files belonging to the same provider are under a single directory that is part of the Airflow workspace. For now the code is more complex because we are handling providers being in either "old" or "new" structure, but once we move old providers to the new structure, a lot of code could be removed and simplified.
The new structure for provider code is:
```
providers
|- PROVIDER_ID
| |- src
| | |-airflow
| | |- providers
| | |- PROVIDER_ID
| |- tests
| | |- providers
| | |- PROVIDER_ID
| |- docs
| | |- .latest-doc-only-changes.txt
| |- pyproject.toml
| |- CHANGELOG.rst
| |- provider.yaml
| |- README.rst
|- PROVIDER_ID2
...
```
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #45259:
URL: https://github.com/apache/airflow/pull/45259#issuecomment-2567539071
> Once we sort out what needs to be done to move one provider, an automated script to move each one is maybe possible.
That's exactly the plan (and that's how I've done it in very old POC. But only after 3 or 4 selected proividers that exhibit different "variants" - for example one common provider and one provider with auth manager etc. I have not yet chosen the next providers. There will be **some** problems uncovered only after we do such three different providers.
And then we will incrementally move one-provider-by-one-provider with the aid of that script (ideally by a number of different people so that they can also laarne about the structure, conseqences, packaging and solve smaller problems along the way)
My plan how to do this is described in detail (including the automation and incremental move) in https://github.com/apache/airflow/issues/44511#issue-2707381976 - see the end where I even linked to the old POC of the script that was used in the past.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "uranusjr (via GitHub)" <gi...@apache.org>.
uranusjr commented on PR #45259:
URL: https://github.com/apache/airflow/pull/45259#issuecomment-2567384208
Once we sort out what needs to be done to move one provider, an automated script to move each one is maybe possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #45259:
URL: https://github.com/apache/airflow/pull/45259#issuecomment-2572427916
Indeed - worth to describe the proposal how tests should work - this is one of the changes that is not yet "final" setup but more of a step in the right direction.
It's not obvious and worth mentioning what's the setup I propose now (and how it can slightly evolve).
I propose that we move all the provider tests in the "tests" folder or "tests/system" folder as `providers.<PROVIDER>`. Main reason is so that we can mark "tests" as the root for all tests similarly as "src" is root for all sources.
This will allow to avoid problems like "from email import something" failing which was often the problem that we had when we wanted to treat "tests" as PYTHONPATh entry - because it was clashing with built-in email package (for example - there were more packages like that).
So Inside provider folders we (I use google as an example with sytem tests) will have this:
```
tests
|- conftest.py -> here we import common plugins for both "tests" and system tests"
tests/providers/google/
|- -> all google tests are here
tests/system/providers/google
|- conftest.py -> here we have common stuff for system tests of google provider
```
This should help with making nice separation between regular and "system" tests - but also allow to have unambiguous imports:
```python
from email import ....
from providers.email import test_whatever
```
But at the same time, even if someone will have "tests" added to PYTHONPATH (some IDEs will do it automatically) we will avoid this to be interpreted as "from tests.email import" (which happened to a number of people in the past - and it's pretty confusing).
```python
from email
```
We will still have to figure out what to do with "tests_common" - currently we use plugins in the common conftest.py:
```python
pytest_plugins = "tests_common.pytest_plugin"
```
This allows to run provider tests from the root of breeze, and where we also install airflow: `pip install -e.[]` (or by default when you run `uv sync` and have `.venv` environment created. This **just works**:
```bash
pytest providers/airbyte/tests/providers/airbyte/test_something.py
```
But after we complete the provider split we have the possibility of moving "airflow" to subfolder of the main repo - we have not decided yet, but that would make sense - and in this case we will be able to refactor slightly the way how tests_common is imported. One problem with the proposal now is that you cannot run easily tests when you create venv directly in the "provider" folder.
This does **NOT** work yet (you really need to run provider tests in context of Airflow venv):
```
cd providers/airbyte
uv sync
uv run pytest tests/providers/airfbyte/test_something.py
```
But when we move airflow out and make "test_common" a "referrable" uv-workspae project, we should be able to make it possible to make it "common" project that will be imported by all projects: airflow, task_sdk, each of the provider etc. I did not want to make those changes now - that would require a number of import changes in tests , so I want to do it as a separte refactor after all providers have been moved.
This will also make each "provider" project "truly" standalone - i.e. you will be able to easily run tests there without even creating a venv in "." top-level project - and essentially work on provider project separately.
At the same time (thanks to workspaces) we will allow to have all - task_sdk + providers + test_common installed together and run and test them togetther as well. This is the promise of workspaces - that you can work separately on each project but you can also install everything together and implemnt - at the same time - changes in all the projects (which was so far the only way we could develop airflow).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on PR #45259:
URL: https://github.com/apache/airflow/pull/45259#issuecomment-2589157319
Woohoo nice Jarek :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45259:
URL: https://github.com/apache/airflow/pull/45259#discussion_r1907732152
##########
airflow/models/abstractoperator.py:
##########
@@ -44,7 +44,7 @@
if TYPE_CHECKING:
from collections.abc import Mapping
- import jinja2 # Slow import.
+ import jinja2 # Slow imports.
Review Comment:
Yeah. I could not help myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45259:
URL: https://github.com/apache/airflow/pull/45259#discussion_r1907964190
##########
providers/airbyte/pyproject.toml:
##########
@@ -0,0 +1,73 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+[build-system]
+requires = ["flit_core==3.10.1"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "apache-airflow-providers-airbyte"
+version = "5.0.0"
+description = "Provider package apache-airflow-providers-airbyte for Apache Airflow"
+readme = "README.rst"
+authors = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+maintainers = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+keywords = [ "airflow-provider", "airbyte", "airflow", "integration" ]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Environment :: Web Environment",
+ "Intended Audience :: Developers",
+ "Intended Audience :: System Administrators",
+ "Framework :: Apache Airflow",
+ "Framework :: Apache Airflow :: Provider",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Topic :: System :: Monitoring",
+]
+requires-python = "~=3.9"
+dependencies = [
+ "apache-airflow>=2.9.0",
+ "airbyte-api>=0.52.0",
+]
Review Comment:
Very good question - not stupid at all because this is not at all obviois but - there are few reasons.
BTW. I have not planned doing it but while implementing it I found quite a few reasons why its better to just regenerate it.
First of all, there is a problem with versions in URLs - we link to the version of Provider in the URLs directly embedded in pyproject.toml and some of the URLs contains versions of provider (for example link to documentation). Those cannot be templated, we could make them potentially dynamic via "hatch_build" or similar but that's not really worth the effort.
Secondly - we have support for different Python versions in different providers (currently we keep python excludes in Provider.yaml - and this will become very useful when we add Python 3.13 support). While we could rely on trove classifiers as a source of truth, there is a number of CI scripts and breeze ifs that depend on python versions supported in provider and implement skipping or removing the providers, having a single source of truth in `provider.yaml` with exceptions from the "currently supported Python versions list" is way more convenient - there are few providers to have those exceptions usually and we can manage it more easily.
Also in the future we would like to apply some common changes to all providers (say for example flit_core version bump). Also we might add - in the future - different way of referring to workspace projects, regenerating pyproject.toml in this case is quite a bit better than only modifying parts of it in 90+ provider packages.
Finally it's much more maintainable for our 90+ providers to have one place where the "main" template for all pyproject.toml files for provider is kept and always override the pyproject.toml using it. Otherwise there is a temptation taht if somone wants to do a quick-fix and add single provider feature/exception in it's provider.yaml - it could be added in a way that will prevent global changes to work - some of those 90+ pyproject.toml might diverge in differnt direction, that we won't be able to converge.
So - it's much better to always regenerate them. This way you will prevent people from making "quick" changes - they will always have to add the changes in more maintainable way - they will have to implement the template changes so that whatever they implement in the single provider could be re-applied to others. Which will automatically keep all the 90+ providers maintainable in a "common" way in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on code in PR #45259:
URL: https://github.com/apache/airflow/pull/45259#discussion_r1910641006
##########
scripts/ci/pre_commit/update_providers_dependencies.py:
##########
@@ -95,18 +99,46 @@ def visit_ImportFrom(self, node: ImportFrom):
self.process_import(fullname)
+def load_pyproject_toml(pyproject_toml_file_path: Path) -> dict[str, Any]:
+ try:
+ import tomllib
+ except ImportError:
+ import tomli as tomllib
+ return tomllib.loads(pyproject_toml_file_path.read_text())
+
+
def find_all_providers_and_provider_files():
- for root, _, filenames in os.walk(AIRFLOW_PROVIDERS_SRC_DIR):
+ for root, dirs, filenames in os.walk(AIRFLOW_PROVIDERS_DIR):
for filename in filenames:
if filename == "provider.yaml":
- provider_file = Path(root, filename)
- provider_name = str(provider_file.parent.relative_to(AIRFLOW_PROVIDERS_SRC_DIR)).replace(
- os.sep, "."
- )
- provider_info = yaml.safe_load(provider_file.read_text())
+ provider_yaml_file = Path(root, filename)
+ if provider_yaml_file.is_relative_to(AIRFLOW_PROVIDERS_SRC_DIR):
+ # TODO(potiuk) - remove this when we move all providers to the new structure
+ provider_name = str(
+ provider_yaml_file.parent.relative_to(AIRFLOW_PROVIDERS_SRC_DIR)
+ ).replace(os.sep, ".")
+ else:
+ provider_name = str(provider_yaml_file.parent.relative_to(AIRFLOW_PROVIDERS_DIR)).replace(
+ os.sep, "."
+ )
+ NEW_STRUCTURE_PROVIDERS.add(provider_name)
+ PYPROJECT_TOML_CONTENT[provider_name] = load_pyproject_toml(
+ provider_yaml_file.parent / "pyproject.toml"
+ )
+ # only descend to "src" directory in the new structure
+ # this avoids descending into .venv or "build" directories in case
+ # someone works on providers in a separate virtualenv
+ if "src" in dirs:
Review Comment:
Nice check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on code in PR #45259:
URL: https://github.com/apache/airflow/pull/45259#discussion_r1907790732
##########
providers/airbyte/pyproject.toml:
##########
@@ -0,0 +1,73 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+[build-system]
+requires = ["flit_core==3.10.1"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "apache-airflow-providers-airbyte"
+version = "5.0.0"
+description = "Provider package apache-airflow-providers-airbyte for Apache Airflow"
+readme = "README.rst"
+authors = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+maintainers = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+keywords = [ "airflow-provider", "airbyte", "airflow", "integration" ]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Environment :: Web Environment",
+ "Intended Audience :: Developers",
+ "Intended Audience :: System Administrators",
+ "Framework :: Apache Airflow",
+ "Framework :: Apache Airflow :: Provider",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Topic :: System :: Monitoring",
+]
+requires-python = "~=3.9"
+dependencies = [
+ "apache-airflow>=2.9.0",
+ "airbyte-api>=0.52.0",
+]
Review Comment:
Perhaps a stupid question, but after the pyproject.toml is first generated, why does it need to be regenerated? No need to answer unless it is going to be generally relevant to provider developers.
##########
providers/airbyte/pyproject.toml:
##########
@@ -0,0 +1,73 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+[build-system]
+requires = ["flit_core==3.10.1"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "apache-airflow-providers-airbyte"
+version = "5.0.0"
+description = "Provider package apache-airflow-providers-airbyte for Apache Airflow"
+readme = "README.rst"
+authors = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+maintainers = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+keywords = [ "airflow-provider", "airbyte", "airflow", "integration" ]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Environment :: Web Environment",
+ "Intended Audience :: Developers",
+ "Intended Audience :: System Administrators",
+ "Framework :: Apache Airflow",
+ "Framework :: Apache Airflow :: Provider",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Topic :: System :: Monitoring",
+]
+requires-python = "~=3.9"
+dependencies = [
+ "apache-airflow>=2.9.0",
+ "airbyte-api>=0.52.0",
+]
Review Comment:
Perhaps a stupid question, but after the pyproject.toml is first generated, why does it need to be regenerated? No need to answer unless it is going to be generally relevant to provider developers.
##########
providers/airbyte/pyproject.toml:
##########
@@ -0,0 +1,73 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+[build-system]
+requires = ["flit_core==3.10.1"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "apache-airflow-providers-airbyte"
+version = "5.0.0"
+description = "Provider package apache-airflow-providers-airbyte for Apache Airflow"
+readme = "README.rst"
+authors = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+maintainers = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+keywords = [ "airflow-provider", "airbyte", "airflow", "integration" ]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Environment :: Web Environment",
+ "Intended Audience :: Developers",
+ "Intended Audience :: System Administrators",
+ "Framework :: Apache Airflow",
+ "Framework :: Apache Airflow :: Provider",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Topic :: System :: Monitoring",
+]
+requires-python = "~=3.9"
+dependencies = [
+ "apache-airflow>=2.9.0",
+ "airbyte-api>=0.52.0",
+]
Review Comment:
Perhaps a stupid question, but after the pyproject.toml is first generated, why does it need to be regenerated? No need to answer unless it is going to be generally relevant to provider developers.
##########
providers/airbyte/pyproject.toml:
##########
@@ -0,0 +1,73 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+[build-system]
+requires = ["flit_core==3.10.1"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "apache-airflow-providers-airbyte"
+version = "5.0.0"
+description = "Provider package apache-airflow-providers-airbyte for Apache Airflow"
+readme = "README.rst"
+authors = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+maintainers = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+keywords = [ "airflow-provider", "airbyte", "airflow", "integration" ]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Environment :: Web Environment",
+ "Intended Audience :: Developers",
+ "Intended Audience :: System Administrators",
+ "Framework :: Apache Airflow",
+ "Framework :: Apache Airflow :: Provider",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Topic :: System :: Monitoring",
+]
+requires-python = "~=3.9"
+dependencies = [
+ "apache-airflow>=2.9.0",
+ "airbyte-api>=0.52.0",
+]
Review Comment:
Perhaps a stupid question, but after the pyproject.toml is first generated, why does it need to be regenerated? No need to answer unless it is going to be generally relevant to provider developers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45259:
URL: https://github.com/apache/airflow/pull/45259#discussion_r1907732882
##########
providers/airbyte/pyproject.toml:
##########
@@ -0,0 +1,73 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+[build-system]
+requires = ["flit_core==3.10.1"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "apache-airflow-providers-airbyte"
+version = "5.0.0"
+description = "Provider package apache-airflow-providers-airbyte for Apache Airflow"
+readme = "README.rst"
+authors = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+maintainers = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+keywords = [ "airflow-provider", "airbyte", "airflow", "integration" ]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Environment :: Web Environment",
+ "Intended Audience :: Developers",
+ "Intended Audience :: System Administrators",
+ "Framework :: Apache Airflow",
+ "Framework :: Apache Airflow :: Provider",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Topic :: System :: Monitoring",
+]
+requires-python = "~=3.9"
+dependencies = [
+ "apache-airflow>=2.9.0",
+ "airbyte-api>=0.52.0",
+]
Review Comment:
It's the source of truth here - they are both read and written from here (see above).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45259:
URL: https://github.com/apache/airflow/pull/45259#discussion_r1907964190
##########
providers/airbyte/pyproject.toml:
##########
@@ -0,0 +1,73 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+[build-system]
+requires = ["flit_core==3.10.1"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "apache-airflow-providers-airbyte"
+version = "5.0.0"
+description = "Provider package apache-airflow-providers-airbyte for Apache Airflow"
+readme = "README.rst"
+authors = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+maintainers = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+keywords = [ "airflow-provider", "airbyte", "airflow", "integration" ]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Environment :: Web Environment",
+ "Intended Audience :: Developers",
+ "Intended Audience :: System Administrators",
+ "Framework :: Apache Airflow",
+ "Framework :: Apache Airflow :: Provider",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Topic :: System :: Monitoring",
+]
+requires-python = "~=3.9"
+dependencies = [
+ "apache-airflow>=2.9.0",
+ "airbyte-api>=0.52.0",
+]
Review Comment:
Very good question - not stupid at all because this is not at all obviois but - there are few reasons.
BTW. I have not planned doing it but while implementing it I found quite a few reasons why its better to just regenerate it.
* First of all, there is a problem with versions in URLs - we link to the version of Provider in the URLs directly embedded in pyproject.toml and some of the URLs contains versions of provider (for example link to documentation). Those cannot be templated, we could make them potentially dynamic via "hatch_build" or similar but that's not really worth the effort.
* Secondly - we have support for different Python versions in different providers (currently we keep python excludes in Provider.yaml - and this will become very useful when we add Python 3.13 support). While we could rely on trove classifiers as a source of truth, there is a number of CI scripts and breeze ifs that depend on python versions supported in provider and implement skipping or removing the providers, having a single source of truth in `provider.yaml` with exceptions from the "currently supported Python versions list" is way more convenient - there are few providers to have those exceptions usually and we can manage it more easily.
* Also in the future we would like to apply some common changes to all providers (say for example flit_core version bump). Also we might add - in the future - different way of referring to workspace projects, regenerating pyproject.toml in this case is quite a bit better than only modifying parts of it in 90+ provider packages.
* Finally it's much more maintainable for our 90+ providers to have one place where the "main" template for all pyproject.toml files for provider is kept and always override the pyproject.toml using it. Otherwise there is a temptation taht if somone wants to do a quick-fix and add single provider feature/exception in it's provider.yaml - it could be added in a way that will prevent global changes to work - some of those 90+ pyproject.toml might diverge in differnt direction, that we won't be able to converge.
So - it's much better to always regenerate them. This way you will prevent people from making "quick" changes - they will always have to add the changes in more maintainable way - they will have to implement the template changes so that whatever they implement in the single provider could be re-applied to others. Which will automatically keep all the 90+ providers maintainable in a "common" way in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #45259:
URL: https://github.com/apache/airflow/pull/45259#issuecomment-2582276299
> Looks good! Our sphinx friend is angry though :)
Yeah. And as usual speaking riddles, so I have to somehow walk around the beast.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #45259:
URL: https://github.com/apache/airflow/pull/45259#issuecomment-2572263862
What about system tests? Is it going to stay cenerlized or per provider?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #45259:
URL: https://github.com/apache/airflow/pull/45259#issuecomment-2572315952
> What about system tests? Is it going to stay cenerlized or per provider?
They are under "tests/system" in the provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #45259:
URL: https://github.com/apache/airflow/pull/45259
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #45259:
URL: https://github.com/apache/airflow/pull/45259#issuecomment-2564247959
This is not yet complete, and I also extract some things out of that to separate PR to make it smaller/simpler. Just wanted to test the image build process.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #45259:
URL: https://github.com/apache/airflow/pull/45259#issuecomment-2588500262
OK. Sphinx riddles and resulting Drachenfutter continued ....
I **think** I got it right now. I had to make some "dirty" solution to speed it up for docs building. There are far too many moving parts now to fix it for docs building individually "per-package" now, so for now I am just dynamically copying the docs to "docs/apache-airflow-providers-*/ and build docs from there. (those new providers are gitignored and by default the generated files are deleted after building to avoid the docs to show up in your IDE in two places normally (you can add `--skip-deletion` flag to leave the copied docs in place).
Eventually, I think we shoudl be able to generate docs separately for each provider - in each provider on it's own. But that would require quite a bit more heavy sphinx scripts refactoring (or rather generally writing them from the scratch in much simpler form).
But for now I think it should be "good enough". Let's see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on code in PR #45259:
URL: https://github.com/apache/airflow/pull/45259#discussion_r1906082730
##########
airflow/new_provider.yaml.schema.json:
##########
Review Comment:
I would recommend not calling this "new" but rather rename the existing one to "legacy_structure" or so.
##########
providers/airbyte/pyproject.toml:
##########
@@ -0,0 +1,73 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+[build-system]
+requires = ["flit_core==3.10.1"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "apache-airflow-providers-airbyte"
+version = "5.0.0"
+description = "Provider package apache-airflow-providers-airbyte for Apache Airflow"
+readme = "README.rst"
+authors = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+maintainers = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+keywords = [ "airflow-provider", "airbyte", "airflow", "integration" ]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Environment :: Web Environment",
+ "Intended Audience :: Developers",
+ "Intended Audience :: System Administrators",
+ "Framework :: Apache Airflow",
+ "Framework :: Apache Airflow :: Provider",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Topic :: System :: Monitoring",
+]
+requires-python = "~=3.9"
+dependencies = [
+ "apache-airflow>=2.9.0",
+ "airbyte-api>=0.52.0",
+]
Review Comment:
How is the dependency generated when is it not contained in provider.yaml anymore?!?
##########
providers/src/airflow/providers/edge/provider.yaml:
##########
@@ -14,8 +14,6 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
-
-# yaml-language-server: $schema=../../provider.yaml.schema.json
Review Comment:
Ups, good catch... VSCode user revealed :-D - seems the path is from the structure before providers were extracted from airflow folder. Thanks for the cleanup!
##########
airflow/models/abstractoperator.py:
##########
@@ -44,7 +44,7 @@
if TYPE_CHECKING:
from collections.abc import Mapping
- import jinja2 # Slow import.
+ import jinja2 # Slow imports.
Review Comment:
I'd propose to make this to another PR. Seems unrelated.
##########
airflow/new_provider.yaml.schema.json:
##########
Review Comment:
Why actually do we need a new schema? I just saw the diff is mainly coming from removed dependencies... but where are these now defined? (if pyproject.toml is generated...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "perry2of5 (via GitHub)" <gi...@apache.org>.
perry2of5 commented on code in PR #45259:
URL: https://github.com/apache/airflow/pull/45259#discussion_r1907790731
##########
providers/airbyte/pyproject.toml:
##########
@@ -0,0 +1,73 @@
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# NOTE! THIS FILE IS AUTOMATICALLY GENERATED AND WILL BE OVERWRITTEN!
+
+# IF YOU WANT TO MODIFY THIS FILE, YOU SHOULD MODIFY THE TEMPLATE
+# `pyproject_TEMPLATE.toml.jinja2` IN the `dev/breeze/src/airflow_breeze/templates` DIRECTORY
+[build-system]
+requires = ["flit_core==3.10.1"]
+build-backend = "flit_core.buildapi"
+
+[project]
+name = "apache-airflow-providers-airbyte"
+version = "5.0.0"
+description = "Provider package apache-airflow-providers-airbyte for Apache Airflow"
+readme = "README.rst"
+authors = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+maintainers = [
+ {name="Apache Software Foundation", email="dev@airflow.apache.org"},
+]
+keywords = [ "airflow-provider", "airbyte", "airflow", "integration" ]
+classifiers = [
+ "Development Status :: 5 - Production/Stable",
+ "Environment :: Console",
+ "Environment :: Web Environment",
+ "Intended Audience :: Developers",
+ "Intended Audience :: System Administrators",
+ "Framework :: Apache Airflow",
+ "Framework :: Apache Airflow :: Provider",
+ "License :: OSI Approved :: Apache Software License",
+ "Programming Language :: Python :: 3.9",
+ "Programming Language :: Python :: 3.10",
+ "Programming Language :: Python :: 3.11",
+ "Programming Language :: Python :: 3.12",
+ "Topic :: System :: Monitoring",
+]
+requires-python = "~=3.9"
+dependencies = [
+ "apache-airflow>=2.9.0",
+ "airbyte-api>=0.52.0",
+]
Review Comment:
Perhaps a stupid question, but after the pyproject.toml is first generated, why does it need to be regenerated? No need to answer unless it is going to be generally relevant to provider developers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Move first provider (airbyte) to a separate project [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #45259:
URL: https://github.com/apache/airflow/pull/45259#discussion_r1907731701
##########
airflow/new_provider.yaml.schema.json:
##########
Review Comment:
Good idea (to use different naming here) .
Yes. It's all about dependencies. PyProject toml is generated ***with the exception** of dependencies. What we do - we read the dependencies from `pyproject.toml` first -> and then generate the pyproject.toml again with all the updated things (such as version of the provider).
This is for example to accomodate the links to the provider's URLs - directly linking to the docs of the provider in the right version - that's why `pyproject.toml` needs to be regenerated anyway. And dependencies is really the only things that we are expected to be maintained by contributors, so it actually makes sense to let them update directly in `pyproject.toml` - rather than have another source of truth.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org