You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "romsharon98 (via GitHub)" <gi...@apache.org> on 2024/08/18 15:36:50 UTC
[PR] Airflow Core Provider [airflow]
romsharon98 opened a new pull request, #41564:
URL: https://github.com/apache/airflow/pull/41564
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1733269695
##########
airflow/example_dags/example_branch_day_of_week_operator.py:
##########
Review Comment:
Agree some easiest solution would be better , and help us when we move forward down the line.
Was refactoring bash operator to standard provider, and encountered few pre commit checks failed. i have added them to exclude block as a workoaround.
<img width="1214" alt="image" src="https://github.com/user-attachments/assets/857942b3-93cc-4e9c-9823-82f96f50201d">
<img width="1086" alt="image" src="https://github.com/user-attachments/assets/8ee93371-cd31-4b33-a4c2-8cc944a31ae6">
Appreciate some solution or suggestion for these failures. Once the current PR merged i will raise the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308793842
> we are missing few things...
>
> 1. we need to deprecate the existed operators in core
> 2. we are missing all the operators and sensors we have in core (currently it's only for time operators)
> 3. I am not sure what about the `hooks` folder?
@eladkal me and romsharon discussed , will be adding rest of the operators and sensors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1732142780
##########
airflow/example_dags/example_branch_day_of_week_operator.py:
##########
Review Comment:
> you probably need to move this example to the provider
as @potiuk said in the upper comment
> the examples should be embedded and shown in airflow when "examples" are enabled - I think we should not (at least not now) to develop a mechanism to read examples for Airflow from providers.
however I get a weird import error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308790425
One more thing - at least for now, "standard" should be added to list of preinstalled providers in `hatch_build.py`
```
PRE_INSTALLED_PROVIDERS = [
"common.compat",
"common.io",
"common.sql",
"fab>=1.0.2",
"ftp",
"http",
"imap",
"smtp",
"sqlite",
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1733254694
##########
airflow/example_dags/example_branch_day_of_week_operator.py:
##########
Review Comment:
I agree. I think we should favour the easiest solution to make this work. If leaving it in core is the one then so be it. We can always move them later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1733211889
##########
airflow/example_dags/example_branch_day_of_week_operator.py:
##########
Review Comment:
But we do have examples for providers.. they are just not in `providers/*/examples` they are in the system tests.
We don't want `providers/*/examples` paths.. we moved all of them to the system test path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1731164155
##########
airflow/example_dags/example_branch_datetime_operator.py:
##########
Review Comment:
you probably need to move this example to the provider
##########
airflow/example_dags/example_branch_day_of_week_operator.py:
##########
Review Comment:
you probably need to move this example to the provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1733251520
##########
airflow/example_dags/example_branch_day_of_week_operator.py:
##########
Review Comment:
Yeah. But currently the "examples" in core are automatically enabled when you choose to show them - and I think those "standard provider" examples should also show up - because otherwise, when we move all "core" operators to providers, there will be no more examples to show in "airflow".
So I am ok with moving the examples to "standard provider" - but there we should figure out how to show them when we choose "show examples = True" (and we likely do not want to show all the few 100 system tests we have from other providers). Also those examples are part of the "core" documentation currently, and we should find them to be prominently visible in the docs - without having to go to "standard provider" documentation (I think) - just to make them better discoverable (currently I can imagine you know you should go to Google provider docs when you are looking for google examples - but people might not be aware they should look at standard provider when they are looking for PythonVirtualenv example - and the doc pages describing those should **likely** be in Airflow "core" docs.
So for me - this is a sign that "standard" provider's examples should be treated somewhat differently, but I am open to hear what others think about it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308975730
> also not sure but should I change somewhere in the code to tell that this provider is mandatory and will auto installed?
That's the preinstalled_providers - but it will only work after it's been released and it will likely change with Airflow 3 so for now - no need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2351515949
@romsharon98 just checking is the changes required in here https://github.com/apache/airflow/blob/main/docs/apache-airflow/operators-and-hooks-ref.rst?plain=1#L91 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1748058063
##########
generated/provider_dependencies.json:
##########
@@ -1249,6 +1249,16 @@
"excluded-python-versions": [],
"state": "ready"
},
+ "standard": {
+ "deps": [
+ "apache-airflow>=2.10.0"
Review Comment:
```suggestion
"apache-airflow>=2.8.0"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1748057936
##########
dev/breeze/src/airflow_breeze/global_constants.py:
##########
@@ -509,13 +509,13 @@ def get_airflow_extras():
{
"python-version": "3.8",
"airflow-version": "2.8.4",
- "remove-providers": "cloudant fab",
+ "remove-providers": ["fab", "standard"],
Review Comment:
I think we should aim to get standard to be **not** removed. Not sure why we are removing it now - but we agreed that 2.8 should be the "min-version" for standard provider, so the tests should be adapted to work for those versions as well..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2352241084
> @romsharon98 just checking is the changes required in here https://github.com/apache/airflow/blob/main/docs/apache-airflow/operators-and-hooks-ref.rst?plain=1#L91 ?
good catch! I will modify it 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1733299228
##########
airflow/example_dags/example_branch_day_of_week_operator.py:
##########
Review Comment:
> Appreciate some solution or suggestion for these failures. Once the current PR merged i will raise the changes.
I think if we agree that "examples" should stay in the core, then the pre-commit should be changed to allow imports from "apache.airlfow.providers.standard" - but not for any other provider import.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308799044
> > Yes, but I think this should be done as separate PR to v2-10-test. I think we do not want to back-port that one to v2-10-test - and we should simply do it there (after we release standard as provider). I am not sure we want to back-port the whole PR here to v2-10-test (providers are not build nor tested there anyway).
>
> Then I propose this PR to remove the files from main that we migrated to the provider. The PR to v2-10-test would be just to replace the classes with deprecation notes after we release the provider.
Please correct me if i understood this, on v2-10-test deprecation updates required for all the changes we are doing on main?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1733541596
##########
airflow/example_dags/example_branch_day_of_week_operator.py:
##########
Review Comment:
Thanks Jarek :) , makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Core Provider [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1722351435
##########
docs/apache-airflow-providers-core/index.rst:
##########
@@ -0,0 +1,97 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+``apache-airflow-providers-core``
+===========================================
+
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Basics
+
+ Home <self>
+ Changelog <changelog>
+ Security <security>
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: System tests
+
+ System Tests <_api/tests/system/providers/core/index>
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Guides
+
+ Connection types <connections>
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Resources
+
+ Example DAGs <https://github.com/apache/airflow/tree/providers-core/|version|/tests/system/providers/core/iceberg>
+ PyPI Repository <https://pypi.org/project/apache-airflow-providers-core/>
+ Installing from sources <installing-providers-from-sources>
+ Python API <_api/airflow/providers/core/index>
+
+
+
+.. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN AT RELEASE TIME!
+
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Commits
+
+ Detailed list of commits <commits>
+
+
+apache-airflow-providers-core package
+------------------------------------------------------
+
+
+Release: 1.0.0
+
+Provider package
+----------------
+
+This package is for the ``core`` provider.
+All classes for this package are included in the ``airflow.providers.core`` python package.
+
+Installation
+------------
+
+You can install this package on top of an existing Airflow 2 installation via
+``pip install apache-airflow-providers-core``.
+For the minimum Airflow version supported, see ``Requirements`` below.
+
+Requirements
+------------
+
+The minimum Apache Airflow version supported by this provider package is ``2.7.0``.
+
+================== ==================
+PIP package Version required
+================== ==================
+``apache-airflow`` ``>=2.7.0``
Review Comment:
changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308793428
Thanks @romsharon98 , will be adding rest of the operators and sensors to this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308913875
Interesting. The errors we see are very useful :).
1) examples are part of the documentation of "main" airflow" - and I am afraid they will have to remain as examples in "airflow" not moved to provider, especially because point 2)...
2) the examples should be embedded and shown in airflow when "examples" are enabled - I think we should not (at least not now) to develop a mechanism to read examples for Airflow from providers.
3) The tests should also mock the moved packages.
```
<module 'airflow.sensors.time_delta' from '/usr/local/lib/python3.8/site-packages/airflow/sensors/time_delta.py'> does not have the attribute 'sleep'
FAILED tests/providers/standard/time/sensors/test_time_delta.py::TestTimeDeltaSensorAsync::test_wait_sensor[True] - AttributeError: <module 'airflow.sensors.time_delta' from '/u
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1748057312
##########
airflow/providers/standard/CHANGELOG.rst:
##########
@@ -0,0 +1,39 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+.. NOTE TO CONTRIBUTORS:
+ Please, only add notes to the Changelog just below the "Changelog" header when there are some breaking changes
+ and you want to add an explanation to the users on how they are supposed to deal with them.
+ The changelog is updated and maintained semi-automatically by release manager.
+
+``apache-airflow-providers-standard``
+
+
+Changelog
+---------
+
+1.0.0
+.....
+
+Initial version of the provider.
Review Comment:
We shoudl make sure to add a note for every deprecation we added vs. the original providers. Basically this changeliog shouid describe what has changed here comparing to the operators/sensors/hooks we had in `Airflow 2`.
The format of this should be like "this and this deprecated thing has been removed, you need to do this and that to fix it" (i.e. actionable decription of the change).
##########
airflow/providers/standard/CHANGELOG.rst:
##########
@@ -0,0 +1,39 @@
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+.. NOTE TO CONTRIBUTORS:
+ Please, only add notes to the Changelog just below the "Changelog" header when there are some breaking changes
+ and you want to add an explanation to the users on how they are supposed to deal with them.
+ The changelog is updated and maintained semi-automatically by release manager.
+
+``apache-airflow-providers-standard``
+
+
+Changelog
+---------
+
+1.0.0
+.....
+
+Initial version of the provider.
Review Comment:
We should make sure to add a note for every deprecation we added vs. the original providers. Basically this changeliog shouid describe what has changed here comparing to the operators/sensors/hooks we had in `Airflow 2`.
The format of this should be like "this and this deprecated thing has been removed, you need to do this and that to fix it" (i.e. actionable decription of the change).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Core Provider [airflow]
Posted by "nevcohen (via GitHub)" <gi...@apache.org>.
nevcohen commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1722031567
##########
docs/apache-airflow-providers-core/index.rst:
##########
@@ -0,0 +1,97 @@
+
+ .. Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ .. http://www.apache.org/licenses/LICENSE-2.0
+
+ .. Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+``apache-airflow-providers-core``
+===========================================
+
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Basics
+
+ Home <self>
+ Changelog <changelog>
+ Security <security>
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: System tests
+
+ System Tests <_api/tests/system/providers/core/index>
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Guides
+
+ Connection types <connections>
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Resources
+
+ Example DAGs <https://github.com/apache/airflow/tree/providers-core/|version|/tests/system/providers/core/iceberg>
+ PyPI Repository <https://pypi.org/project/apache-airflow-providers-core/>
+ Installing from sources <installing-providers-from-sources>
+ Python API <_api/airflow/providers/core/index>
+
+
+
+.. THE REMAINDER OF THE FILE IS AUTOMATICALLY GENERATED. IT WILL BE OVERWRITTEN AT RELEASE TIME!
+
+
+.. toctree::
+ :hidden:
+ :maxdepth: 1
+ :caption: Commits
+
+ Detailed list of commits <commits>
+
+
+apache-airflow-providers-core package
+------------------------------------------------------
+
+
+Release: 1.0.0
+
+Provider package
+----------------
+
+This package is for the ``core`` provider.
+All classes for this package are included in the ``airflow.providers.core`` python package.
+
+Installation
+------------
+
+You can install this package on top of an existing Airflow 2 installation via
+``pip install apache-airflow-providers-core``.
+For the minimum Airflow version supported, see ``Requirements`` below.
+
+Requirements
+------------
+
+The minimum Apache Airflow version supported by this provider package is ``2.7.0``.
+
+================== ==================
+PIP package Version required
+================== ==================
+``apache-airflow`` ``>=2.7.0``
Review Comment:
Required `>=2.8.0`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308795806
> Yes, but I think this should be done as separate PR to v2-10-test. I think we do not want to back-port that one to v2-10-test - and we should simply do it there (after we release standard as provider). I am not sure we want to back-port the whole PR here to v2-10-test (providers are not build nor tested there anyway).
Then I propose this PR to remove the files from main that we migrated to the provider.
The PR to v2-10-test would be just to replace the classes with deprecation notes after we release the provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308797132
> Then I propose this PR to remove the files from main that we migrated to the provider.
This is what the PR does.
<img width="767" alt="Screenshot 2024-08-25 at 13 46 45" src="https://github.com/user-attachments/assets/a2090679-334f-4f79-8d87-a775d17bde13">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2309566421
Lets wait with merge till we understand the impact of https://github.com/apache/airflow/pull/41727#discussion_r1730836865
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308800207
> Please correct me if i understood this, on v2-10-test deprecation updates required for all the changes we are doing on main?
No - only those that are supposed to affect DAG authors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308798626
Ah! I didn't see changes to `airflow/core` folder so I assumed it was missed.
Not very friendly UX design from GitHub
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2309602342
> Lets wait with merge till we understand the impact of [#41727 (comment)](https://github.com/apache/airflow/pull/41727#discussion_r1730836865)
relevant code was reverted. We are good to go here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1733028882
##########
airflow/example_dags/example_branch_day_of_week_operator.py:
##########
Review Comment:
The import error - I explained in Slack - we migtht (likely) make the provider to be only Airflow 2.10+ compatible to allow people to change the imports in 2.10 as part of "recommended" migration process.
I believe @ashb planned to get the old imports to continue to work with https://docs.python.org/3/glossary.html#term-meta-path-finder with deprecation) - I will let him comment on it.
But for the reasons explained above (examples are really part of "core" airflow and changing providers to contribute examples when "show examples" are enabled might be an overkill and since this provider is going to be "preinstalled" - I think adding a feature to read examples from providers in this case might be quite an overkill - but maybe it makes sense if you think that @eladkal -> it **could** be done possibly by just adding "providers/*/examples" to the paths when DAGs are searched. We would likely also have to figure out how to filter out "examples/system tests" in this case, because then "show examples" might led to a huge set of examples visible.
Also in this case we should move all the documentation for the standard operators to "provider" documentation.
But maybe that's a good idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308803425
BTW. We **will** have to make the old "standard" providers still work in Airflow 3 I **think** so we will have to make the old operator imports work (likely with the mechanism mentioned by @ashb - `sys.meta_path`) - otherwise it will `require` many dags to change. I think we should apply the same mechanism as we will do for importing BaseOperator etc. in Airflow 3 (and still raise deprecation warning).
@ashb ? Do you agree ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308794930
Ah .. so no ... We should not add it to preinstalled just **yet** - until we release it (sorry @romsharon98 -> see the failure on openapi client).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308792633
1. we need to deprecate the existed operators in core
> Yes, but I think this should be done as separate PR to v2-10-test. I think we do not want to back-port that one to v2-10-test - and we should simply do it there (after we release `standard` as provider). I am not sure we want to back-port the whole PR here.
2. we are missing all the operators (currently it's only for time operators)
> I think those should be separate, smaller PRs ?
3. I am not sure what about the hooks folder?
> I think it's the same ^^ - some should be moved but separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308790670
We might change where it is added in Airflow 3, but for now it should be added here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "gopidesupavan (via GitHub)" <gi...@apache.org>.
gopidesupavan commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308984153
so for time related things its placed under time/ , for other operators and sensors am placing like this .
standard/core/operators/
standard/core/sensors/
is this fine? or any other suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2308975222
> Interesting. The errors we see are very useful :).
>
> 1. examples are part of the documentation of "main" airflow" - and I am afraid they will have to remain as examples in "airflow" not moved to provider, especially because point 2)...
> 2. the examples should be embedded and shown in airflow when "examples" are enabled - I think we should not (at least not now) to develop a mechanism to read examples for Airflow from providers.
> 3. The tests should also mock the moved packages.
>
> ```
> <module 'airflow.sensors.time_delta' from '/usr/local/lib/python3.8/site-packages/airflow/sensors/time_delta.py'> does not have the attribute 'sleep'
> FAILED tests/providers/standard/time/sensors/test_time_delta.py::TestTimeDeltaSensorAsync::test_wait_sensor[True] - AttributeError: <module 'airflow.sensors.time_delta' from '/u
> ```
revert the example dags to be under `main` airflow.
should I add a link in index.yaml for those example dags?
also not sure but should I change somewhere in the code to tell that this provider is mandatory and will auto installed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1748058154
##########
airflow/providers/standard/provider.yaml:
##########
@@ -0,0 +1,50 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+---
+package-name: apache-airflow-providers-standard
+name: Standard
+description: |
+ Airflow Standard Provider
+state: not-ready
+source-date-epoch: 1718603992
+
+# note that those versions are maintained by release manager - do not update them manually
+versions:
+ - 1.0.0
+
+dependencies:
+ - apache-airflow>=2.10.0
Review Comment:
```suggestion
- apache-airflow>=2.8.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2335161119
LGTM in general. Tests are failing and need to be fixed, and I added few comments that should be resolved before merging that one. Also I think we should figure out (but that might be next) how to get example_dags to display them in Airflow 3 from provider (likely change/extend from where they are loaded) - but this should be a separate PR, likely when we have few more examples moved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 commented on PR #41564:
URL: https://github.com/apache/airflow/pull/41564#issuecomment-2335097142
I think that after the [consensus](https://lists.apache.org/thread/1ltpczo121tgk2pzx31bgsk0zj4s5o6x) we can remove the request change @eladkal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 merged PR #41564:
URL: https://github.com/apache/airflow/pull/41564
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Airflow Standard Provider [airflow]
Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 commented on code in PR #41564:
URL: https://github.com/apache/airflow/pull/41564#discussion_r1763023160
##########
dev/breeze/src/airflow_breeze/global_constants.py:
##########
@@ -509,13 +509,13 @@ def get_airflow_extras():
{
"python-version": "3.8",
"airflow-version": "2.8.4",
- "remove-providers": "cloudant fab",
+ "remove-providers": ["fab", "standard"],
Review Comment:
I remember why I did it and kept the min version to be 2.10.
We talked about it in slack: https://apache-airflow.slack.com/archives/C06K9Q5G2UA/p1724737809230079
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org