You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "yehoshuadimarsky (via GitHub)" <gi...@apache.org> on 2024/05/31 19:20:33 UTC
[PR] bump minimum boto3 1.34.52 for AWS provider [airflow]
yehoshuadimarsky opened a new pull request, #39983:
URL: https://github.com/apache/airflow/pull/39983
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
The change introduced here (https://github.com/apache/airflow/pull/39903) doesn't work unless boto3 is using version 1.34.52 or higher. In that version, they added `ecsPropertiesOverride` to the API of `submitJob` ([source](https://github.com/boto/botocore/compare/1.34.51...1.34.52#diff-40ff9627f8663b754f0b417aa4ddba080099f0468a87528eb2d71612e92b8b88)).
Note - I didn't update the versions of these 2 packages, even though the comment said to, because the version 1.34.52 doesn't exist for them
```yaml
- mypy-boto3-appflow>=1.33.0
- mypy-boto3-redshift-data>=1.33.0
```
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "yehoshuadimarsky (via GitHub)" <gi...@apache.org>.
yehoshuadimarsky commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2147671327
> Actually. If you don't mind - bumping the version to .54 is probably way better approach - boto3 .54 has been releaed 3 months ago and we already have .106 in our constraints, so this is a bascically "no-op" change.
>
> And while it's a bit worrying that our tests do not catch it, i am not particularly picky here - we only go "up" with the minimum version, so it's quite ok if you just bump the version in provider.yaml (and pre-commit will update it in generated/provider_dependencies.json).
So are you saying I should just bump the version, and even if mypy or other tests fail, we'll just ignore it? Because I don't have bandwidth to fix it.
I'm not clear if you are volunteering to fix it, or are saying to ignore it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bump minimum boto3 1.34.52 for AWS provider [airflow]
Posted by "yehoshuadimarsky (via GitHub)" <gi...@apache.org>.
yehoshuadimarsky commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2145628152
To be honest, I don't have the bandwidth now to fix all these things. I originally added the Batch PR to fix a use case that my team at work needed. I'm OK with just keeping that first PR as is, and merely adding a warning to the docs that it requires boto3 1.34.52+. I will push a new commit to this PR to make the docs only change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2148433161
Yep. That is also okl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bump minimum boto3 1.34.52 for AWS provider [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2143384514
Also see the https://github.com/apache/airflow/pull/39946/files#diff-08906ccb524962bced7b3f0c2a19d144567985f6f80077ab0e7d5bb2a6917458R95 in https://github.com/apache/airflow/pull/39946 -> for some reason, the tests there are satisified with 1.34.0, so likely we need to add tests as well to cover the 1.34.52+ check. In #39946, apparently all tests passed with 1.34.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bump minimum boto3 1.34.52 for AWS provider [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2143321046
Can you fix the static checks failures?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2146638603
Actually. If you don't mind - bumping the version to .54 is probably way better approach - boto3 .54 has been releaed 3 months ago and we already have .106 in our constraints, so this is a bascically "no-op" change.
And while it's a bit worrying that our tests do not catch it, i am not particularly picky here - we only go "up" with the minimum version, so it's quite ok if you just bump the version in provider.yaml (and pre-commit will update it in generated/provider_dependencies.json).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] bump minimum boto3 1.34.52 for AWS provider [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2142918974
> Note - I didn't update the versions of these 2 packages, even though the comment said to, because the version 1.34.52 doesn't exist for them
I suspect that's going to lead to mypy failures, no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2148438512
aiobotocore has been out limiting factor for a while, they pin botocore to a pretty old version. A couple weeks ago they released a new version with much newer botocore support. I'm going to see what it will take to get this bumped properly, and we can go from there.
related: https://github.com/apache/airflow/pull/40052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "yehoshuadimarsky (via GitHub)" <gi...@apache.org>.
yehoshuadimarsky commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2147888672
Following an offline conversation with @o-nikolas and @eladkal, decided to follow an approach similar to [the Operator in AWS Bedrock](https://github.com/aws-mwaa/upstream-to-airflow/blob/651a6d6a686c5091c04a2f709297d6e8e6171f90/airflow/providers/amazon/aws/operators/bedrock.py#L802-L819), where new features of boto3 aren't enforced, but rather if an error shows up because of it, we add more context to it.
(Plus keeping the existing change in the docstring to warn that the ECS param only works with boto3 1.34.52+)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "yehoshuadimarsky (via GitHub)" <gi...@apache.org>.
yehoshuadimarsky commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2147690995
You missed the beginning of this PR, where I indeed bumped the version, all the stuff broke, and I wasn't able to fix it. Just need this to unblock releasing the new version of the Amazon provider, I need it for my team at my day job at Amazon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "yehoshuadimarsky (via GitHub)" <gi...@apache.org>.
yehoshuadimarsky commented on code in PR #39983:
URL: https://github.com/apache/airflow/pull/39983#discussion_r1626272094
##########
airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -323,6 +330,17 @@ def submit_job(self, context: Context):
try:
response = self.hook.client.submit_job(**trim_none_values(args))
+ except botocore.exceptions.ParamValidationError as error:
+ if (
+ 'Unknown parameter in input: "ecsPropertiesOverride"' in str(error)
+ ) and self.ecs_properties_override:
+ self.log.error(
+ "You are attempting to use ecsPropertiesOverride and the Boto3 API returned an "
+ "error message which may indicate the need to update botocore to do this. \n"
+ "Support for using ecsPropertiesOverride was added in botocore 1.34.52 and you are using botocore %s",
Review Comment:
I can clarify the message if you want
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #39983:
URL: https://github.com/apache/airflow/pull/39983#discussion_r1626268822
##########
airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -323,6 +330,17 @@ def submit_job(self, context: Context):
try:
response = self.hook.client.submit_job(**trim_none_values(args))
+ except botocore.exceptions.ParamValidationError as error:
+ if (
+ 'Unknown parameter in input: "ecsPropertiesOverride"' in str(error)
+ ) and self.ecs_properties_override:
+ self.log.error(
+ "You are attempting to use ecsPropertiesOverride and the Boto3 API returned an "
+ "error message which may indicate the need to update botocore to do this. \n"
+ "Support for using ecsPropertiesOverride was added in botocore 1.34.52 and you are using botocore %s",
Review Comment:
botocore or boto3?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #39983:
URL: https://github.com/apache/airflow/pull/39983#issuecomment-2147686267
Why do you think my will fail ? I believe just bumping the min version in those two files will **just work**. But if there will be error, then yes - we expect you fix them. But it can take quite so e time so there is no hurry - you can do it at your leisure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "yehoshuadimarsky (via GitHub)" <gi...@apache.org>.
yehoshuadimarsky commented on code in PR #39983:
URL: https://github.com/apache/airflow/pull/39983#discussion_r1626278154
##########
airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -323,6 +330,17 @@ def submit_job(self, context: Context):
try:
response = self.hook.client.submit_job(**trim_none_values(args))
+ except botocore.exceptions.ParamValidationError as error:
+ if (
+ 'Unknown parameter in input: "ecsPropertiesOverride"' in str(error)
+ ) and self.ecs_properties_override:
+ self.log.error(
+ "You are attempting to use ecsPropertiesOverride and the Boto3 API returned an "
+ "error message which may indicate the need to update botocore to do this. \n"
+ "Support for using ecsPropertiesOverride was added in botocore 1.34.52 and you are using botocore %s",
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "ferruzzi (via GitHub)" <gi...@apache.org>.
ferruzzi merged PR #39983:
URL: https://github.com/apache/airflow/pull/39983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Doc-only: mention minimum boto3 1.34.52 for AWS provider when using Batch `ecs_properties_override` [airflow]
Posted by "yehoshuadimarsky (via GitHub)" <gi...@apache.org>.
yehoshuadimarsky commented on code in PR #39983:
URL: https://github.com/apache/airflow/pull/39983#discussion_r1626270666
##########
airflow/providers/amazon/aws/operators/batch.py:
##########
@@ -323,6 +330,17 @@ def submit_job(self, context: Context):
try:
response = self.hook.client.submit_job(**trim_none_values(args))
+ except botocore.exceptions.ParamValidationError as error:
+ if (
+ 'Unknown parameter in input: "ecsPropertiesOverride"' in str(error)
+ ) and self.ecs_properties_override:
+ self.log.error(
+ "You are attempting to use ecsPropertiesOverride and the Boto3 API returned an "
+ "error message which may indicate the need to update botocore to do this. \n"
+ "Support for using ecsPropertiesOverride was added in botocore 1.34.52 and you are using botocore %s",
Review Comment:
Technically it's botocore, that's where all the APIs are defined. Boto3 is just a more user-friendly wrapper on top of boto3, I think they keep their versions synchronized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org