You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/22 04:56:19 UTC
[GitHub] [airflow] potiuk opened a new pull request, #27196: Move min airflow version to 2.3.0 for all providers
potiuk opened a new pull request, #27196:
URL: https://github.com/apache/airflow/pull/27196
As of October 11 our providers are supposed to be compatible with Airflow 2.3+ and all code for backwards compatibility with Airflow 2.2 can be removed now.
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1287786665
Actually, the DBApi one we could do before already after we split-out common.sql.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1287674610
I'd love @uranusjr if you take a look as well. The biggest part of pre-2.3 was removal of dttm from "XCom.get_value" - mostly in operator links.
Also I am not sure if what I've done is correct. I assumed that after migration to 2.3 each task instance will have a "task_instance_key" and we will be able to use task instance key and `get_value` even for task instances created in previous Airflow versions (I believe migration takes care about it).
I would love your comments here @uranusjr if I understood correctly that we should genereally skip all the dttm usage and rely on task_instance_key being always present.
I have found however one case where it is not as straightforward and dttm is still used and I am not sure if we want to do anything about it:
```
class ExternalDagLink(BaseOperatorLink):
"""
Operator link for ExternalTaskSensor and ExternalTaskMarker.
It allows users to access DAG waited with ExternalTaskSensor or cleared by ExternalTaskMarker.
"""
name = 'External DAG'
def get_link(self, operator, dttm):
ti = TaskInstance(task=operator, execution_date=dttm)
operator.render_template_fields(ti.get_template_context())
query = {"dag_id": operator.external_dag_id, "execution_date": dttm.isoformat()}
return build_airflow_url_with_query(query)
```
In all the other cases I converted 'get_link" to
```
def get_link(
self,
operator: BaseOperator,
*,
ti_key: TaskInstanceKey,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27196:
URL: https://github.com/apache/airflow/pull/27196#discussion_r1002355371
##########
.pre-commit-config.yaml:
##########
@@ -344,7 +344,7 @@ repos:
# This check might be removed when min-airflow-version in providers is 2.2
- id: check-airflow-2-2-compatibility
name: Check that providers are 2.2 compatible.
- entry: ./scripts/ci/pre_commit/pre_commit_check_2_2_compatibility.py
+ entry: ./scripts/ci/pre_commit/pre_commit_check_provider_airflow_compatibility.py
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1288220568
All right - seems I get it alll green - @uranusjr - is it possible you take a look at the XCom.get_value() changes before I merge this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1287789216
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1288175286
This one also should cut the "compatibility" test by more than hour. Currently just installing latest providers on airflow 2.2 takes > 1 hour because `pip` tries to resolve dependencies and it tries very hard to do so (this is because of the complex dependency resolution caused by "old" airflow and "new" providers). This is -yet another reason why the 12 months compatibility for providers is a good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1287785768
> We also have:
> https://github.com/apache/airflow/blob/b254a9f4bead4e5d4f74c633446da38550f8e0a1/airflow/providers/oracle/hooks/oracle.py#L428-L441
Ah cool. Indeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1287789324
(also done the same for Trino, Snowflake and Databricks)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
uranusjr commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1288479503
Just did, I think you did it correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1288879908
> Deferrable Operators only work starting Airflow 2.2
Indeed. Thanks @eladkal for all the extra checks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on code in PR #27196:
URL: https://github.com/apache/airflow/pull/27196#discussion_r1002355416
##########
airflow/providers/jira/provider.yaml:
##########
@@ -36,7 +36,7 @@ versions:
- 1.0.0
dependencies:
- - apache-airflow>=2.2.0
+ - apache-airflow>=2.3.0
Review Comment:
I wille get to that PR shortly, but let's get it changed here in the meantime for consistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
eladkal commented on code in PR #27196:
URL: https://github.com/apache/airflow/pull/27196#discussion_r1002353341
##########
.pre-commit-config.yaml:
##########
@@ -344,7 +344,7 @@ repos:
# This check might be removed when min-airflow-version in providers is 2.2
- id: check-airflow-2-2-compatibility
name: Check that providers are 2.2 compatible.
- entry: ./scripts/ci/pre_commit/pre_commit_check_2_2_compatibility.py
+ entry: ./scripts/ci/pre_commit/pre_commit_check_provider_airflow_compatibility.py
Review Comment:
Need to change pre commit id and name
##########
airflow/providers/jira/provider.yaml:
##########
@@ -36,7 +36,7 @@ versions:
- 1.0.0
dependencies:
- - apache-airflow>=2.2.0
+ - apache-airflow>=2.3.0
Review Comment:
I opened a PR to drop this provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1287778721
We also have:
https://github.com/apache/airflow/blob/b254a9f4bead4e5d4f74c633446da38550f8e0a1/airflow/providers/oracle/hooks/oracle.py#L428-L441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1287634806
We need also update
https://github.com/apache/airflow#release-process-for-providers
With new date
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1287635669
> We need also update https://github.com/apache/airflow#release-process-for-providers With new date
Right. Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
eladkal commented on PR #27196:
URL: https://github.com/apache/airflow/pull/27196#issuecomment-1287922404
Ahh we can also handle:
https://github.com/apache/airflow/blob/06acf40a4337759797f666d5bb27a5a393b74fed/airflow/providers/databricks/triggers/databricks.py#L30
https://github.com/apache/airflow/blob/06acf40a4337759797f666d5bb27a5a393b74fed/airflow/providers/google/cloud/triggers/cloud_composer.py#L31
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #27196: Move min airflow version to 2.3.0 for all providers
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #27196:
URL: https://github.com/apache/airflow/pull/27196
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org