You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "shahar1 (via GitHub)" <gi...@apache.org> on 2023/08/26 19:50:43 UTC
[GitHub] [airflow] shahar1 opened a new pull request, #33786: Validate operators init
shahar1 opened a new pull request, #33786:
URL: https://github.com/apache/airflow/pull/33786
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
closes: #29069
### Concept
We want to add a pre-commit that checks for templated field logic checks within the constructor of providers' operators to avoid issues like #27328. Currently, I defined "invalid logic" as follows (I would be happy for fine-tuning):
- If statements that their content does not include calls to `warnings.warn` [1] nor `raise` statements [2].
- Return statements - I saw one randomly in one of the providers and removed it.
- Loop statements - I didn't notice existing ones, but just in case someone goes wild :)
### Notes
[1] Excluded due to deprecation warnings, which often appear in the constructors.
[2] I assume that we would like to avoid exceptions as well, at least those that are related to input validation - but we need to consider what to do in case of exceptions that are related to deprecated params.
### Implementation
As of the time I sent this PR, it includes two commits:
- One that includes the pre-commit Python script and its YAML definition in `pre-commit-config.yaml` - I excluded files that should be fixed in later PR; see next.
- Another that includes modifications to specific operators, either to align with the requirements immediately or by inserting a `FIXME` comment to do it later (in case the complexity seems too high to address at the moment). If the reviewers deem it necessary, I can create a separate pull request for this. Regardless, I'll create a new issue for the necessary fixes.
### TODOs
- [ ] Open an issue for fixing untreated constructors.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent templated field logic checks in __init__ of operators automatically [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1782099531
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1714489727
> @potiuk @eladkal The PR is still in the oven, yet I have a couple of questions:
>
> 1. Can I assume that the name of a template argument (e.g., `foo`) is the same as its respective attribute (`self.foo`)? Otherwise, checking if it's assigned to the appropriate attribute becomes a very hard task (instead of simply recognizing pattern like `self.foo = foo`, which is easier to isolate from the code). I know it is quite strict, but I see no other choice.
Fine for me. We are here for avoiding accidental mistakes not 100% accuracy, and it's fair to assume self.x = x is something that people use (I guess) in most cases. We can likely run some checks on existing codebase to turn it into actual datapoints (see below)
> 2. To what depth should I try diving in when checking the usages of the template argument in the scope of this PR? Python's grammar is pretty [rich](https://docs.python.org/3/library/ast.html#abstract-grammar) - Does it sound reasonable to limit it to conditions or/and loops at this point?
Wahtever helps with big percentage of use cases. and part of the implementation is to decide what is the low-effort/high coverage of cases. This is part of the difficulty - to make some assumptions that make sense. I don't think there is easy yes/no asnswer - but maybe running the check on existing codebase might give the answer on how often various patterns are used ? That could give a better datapoint that wild guessing I believe?
> 3. Please take a look at the limitations that I mentioned in the PR's content - can we live with them for now?
I think without knowing some datapoints - anyone can at most guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent templated field logic checks in __init__ of operators automatically [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1868306978
> @shahar1 I see that the pre-commit found some invalid usages which is awesome.
> I suggest to handle them in separated PR(s) then once all merged you can rebase this one and it will be green
I need some help from the community to implement all required fixes :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent templated field logic checks in __init__ of operators automatically [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1868308640
>
@shahar1 I see that the pre-commit found some invalid usages which is awesome.
I suggest to handle them in separated PR(s) then once all merged you can rebase this one and it will be green
I need some help from the community to implement all required fixes :)
Absolutely. The best thing to do in this case is to somehow extract all those cases automatically and build an issue with
- [ ] problem 1
- [ ] problem 2
... and raise it to devlist. and mark it as a good first issue.
We have quite a number of the "code quality freaks" that woul likely jump into it and start fixing those. Myself including :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1694491786
> One that includes the pre-commit Python script and its YAML definition in pre-commit-config.yaml - I excluded files that should be fixed in later PR; see next.
> Another that includes modifications to specific operators, either to align with the requirements immediately or by inserting a FIXME comment to do it later (in case the complexity seems too high to address at the moment). If the reviewers deem it necessary, I can create a separate pull request for this. Regardless, I'll create a new issue for the necessary fixes.
I think it should be approahed differently.. This PR should fail. It should not have exclusioins. It should only be merged when we (or others) fixed all the problems. This is what we do normally. Then this PR open should serve as reminder and incentivisation to fix all the places. So I think in general, the sequence should be reverse:
1) make sure (it does not have to be you - you can ping other people to do it) to fix alll the places
2) continue rebasing this PR seeing less and less problems as you iterate
3) merge it once all the problems are fixed.
This is the aproach we've used for other cases that were similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shahar1 commented on a diff in pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #33786:
URL: https://github.com/apache/airflow/pull/33786#discussion_r1320694908
##########
airflow/providers/google/cloud/operators/vertex_ai/custom_job.py:
##########
@@ -1111,10 +1129,19 @@ def __init__(
self,
*,
script_path: str,
+ region: str,
requirements: Sequence[str] | None = None,
+ dataset_id: str | None = None,
+ impersonation_chain: str | Sequence[str] | None = None,
**kwargs,
) -> None:
- super().__init__(**kwargs)
+ # FIXME: allow assignments in the super() constructor
Review Comment:
I need to figure out how to deal with attributes that are initialized at the parent's level -
Let's say that the parent has a mandatory template field `foo`, it will be init. by `super().__init__(foo=foo)`. As of the current implementation, I also need to set it in the class itself to detect it in the pre-commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1694731041
I am not sure if that's the right set of validations. I do not find loops or if/return particularly troubling. What I was really hoping for was a check that will check for "typical" errors made in operators that are not often caught during review, but they have some negative side effects.
I am not sure if we want to limit operator's authors from raising warnings/running returns etc. This is far too much IMHO. We do not want to tell the authors "this is how you shoudl do your logic in constructor" - we want to make them aware when they made an obvious mistake. Generally I think we should only run validations that are very clearly violating some not even best but "reasonbly only possible" practice in Airflow.
I think the rule of thumb should be "You should be able to tell the author EXACTLY how to correct their code and explain why it is wrong". And it should be a mistake that "newbie" contributors are likely to make.
The issue https://github.com/apache/airflow/issues/29069 was really about this one:
* when you are using templated arg for something else than just settng it as templated field - Generally speaking the only thing you should do for templated field is you should set it as field. The reason is that this field will not have the right value before "execute" command is fired.
otherwise having some logic for non-templated fields is perfectly fine. I would not want us to be a "dictactor" to tell people how to write their constructors.
And there are few other things that are OBVIOUSLY wrong:
* when you have templated_filed defined for operator but it is not set in the constructor as field, the field is not really templated - it will be missing when templated fields are processed. This is obviously wrong
* when you try to instantiate Hook in the operator's constructor, that's also obviously wrong. You should never do that, because such Hook instantiation will also happen in the DAGFileProcessor when the DAG is parsed.
I'd start with these three rules, rather than trying to present some constructs that are not obviously wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent templated field logic checks in __init__ of operators automatically [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1868306737
@shahar1 I see that the pre-commit found some invalid usages which is awesome.
I suggest to handle them in separated PR(s) then once all merged you can rebase this one and it will be green
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on a diff in pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #33786:
URL: https://github.com/apache/airflow/pull/33786#discussion_r1320685064
##########
airflow/providers/google/cloud/operators/vertex_ai/custom_job.py:
##########
@@ -1111,10 +1129,19 @@ def __init__(
self,
*,
script_path: str,
+ region: str,
requirements: Sequence[str] | None = None,
+ dataset_id: str | None = None,
+ impersonation_chain: str | Sequence[str] | None = None,
**kwargs,
) -> None:
- super().__init__(**kwargs)
+ # FIXME: allow assignments in the super() constructor
Review Comment:
What is this note about?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent templated field logic checks in __init__ of operators automatically [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1872147699
> > @shahar1 I see that the pre-commit found some invalid usages which is awesome.
> > I suggest to handle them in separated PR(s) then once all merged you can rebase this one and it will be green
>
> > I need some help from the community to implement all required fixes :)
>
> Absolutely. The best thing to do in this case is to somehow extract all those cases automatically and build an issue with
>
> * [ ] problem 1
> * [ ] problem 2
>
> ... and raise it to devlist. and mark it as a good first issue.
>
> We have quite a number of the "code quality freaks" that would likely jump into it and start fixing those. Myself including :D
Eventually, I created this list manually in #36484 :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent templated field logic checks in __init__ of operators automatically [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1902071627
I think we can also improve the process here as it takes longer to finish the list of operators to handle.
@shahar1 you can add exclusion list of operators to the pre-commit (the ones that are not ready). This should give us green build here thus we can merge this PR and enjoy the protection it brings.
In the mean time @romsharon98 can continue to work on removing items from the exclusion list (we won't need to maintain https://github.com/apache/airflow/issues/36484 any longer as the exclusion list will be on the pre-commit itself)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shahar1 commented on pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1712935380
@potiuk @eladkal The PR is still in the oven, yet I have a couple of questions:
1. Can I assume that the name of a template argument (e.g., `foo`) is the same as its respective attribute (`self.foo`)? Otherwise, checking if it's assigned to the appropriate attribute becomes a very hard task (instead of simply recognizing pattern like `self.foo = foo`, which is easier to isolate from the code). I know it is quite strict, but I see no other choice.
2. To what depth should I try diving in when checking the usages of the template argument in the scope of this PR? Python's grammar is pretty [rich](https://docs.python.org/3/library/ast.html#abstract-grammar) - Does it sound reasonable to limit it to conditions or/and loops at this point?
3. Please take a look at the limitations that I mentioned in the PR's content - can we live with them for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent templated field logic checks in __init__ of operators automatically [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1902090286
> d exclusion list of operators to the pre-commit (the ones that are not ready). This should give us green build here thus we can merge this PR and enjoy the protection it brings.
Done: https://github.com/apache/airflow/blob/de6a4e0e68f5464ae2a921f8501dce2440f52beb/.pre-commit-config.yaml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shahar1 commented on pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1712189882
> The PR is too big, IMHO it should be split in small PRs before adding the pre-commit hook.
Yes, I'll split it - the files from the providers were here temporarily only for demonstration.
As I got the original idea quite wrong, the current implementation is rather excessive. Therefore, I converted it to a draft - I'll convert it back to "ready" after I fix the pre commit and separate the relevant providers to another PR (if it would still be necessary).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #33786:
URL: https://github.com/apache/airflow/pull/33786#discussion_r1320342248
##########
airflow/providers/amazon/aws/operators/ecs.py:
##########
@@ -496,15 +496,12 @@ def __init__(
self.tags = tags
self.awslogs_group = awslogs_group
self.awslogs_stream_prefix = awslogs_stream_prefix
- self.awslogs_region = awslogs_region
+ self.awslogs_region = awslogs_region or self.region
self.awslogs_fetch_interval = awslogs_fetch_interval
self.propagate_tags = propagate_tags
self.reattach = reattach
self.number_logs_exception = number_logs_exception
- if self.awslogs_region is None:
- self.awslogs_region = self.region
Review Comment:
Empty region_name is invalid region
```
import boto3
session = boto3.session.Session(region_name="")
session.client("sts").get_caller_id()
# ValueError: Invalid endpoint: https://sts..amazonaws.com
```
##########
airflow/providers/amazon/aws/operators/ecs.py:
##########
@@ -496,15 +496,12 @@ def __init__(
self.tags = tags
self.awslogs_group = awslogs_group
self.awslogs_stream_prefix = awslogs_stream_prefix
- self.awslogs_region = awslogs_region
+ self.awslogs_region = awslogs_region or self.region
self.awslogs_fetch_interval = awslogs_fetch_interval
self.propagate_tags = propagate_tags
self.reattach = reattach
self.number_logs_exception = number_logs_exception
- if self.awslogs_region is None:
- self.awslogs_region = self.region
Review Comment:
Empty region_name is invalid region
```python
import boto3
session = boto3.session.Session(region_name="")
session.client("sts").get_caller_id()
# ValueError: Invalid endpoint: https://sts..amazonaws.com
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #33786:
URL: https://github.com/apache/airflow/pull/33786#discussion_r1320342248
##########
airflow/providers/amazon/aws/operators/ecs.py:
##########
@@ -496,15 +496,12 @@ def __init__(
self.tags = tags
self.awslogs_group = awslogs_group
self.awslogs_stream_prefix = awslogs_stream_prefix
- self.awslogs_region = awslogs_region
+ self.awslogs_region = awslogs_region or self.region
self.awslogs_fetch_interval = awslogs_fetch_interval
self.propagate_tags = propagate_tags
self.reattach = reattach
self.number_logs_exception = number_logs_exception
- if self.awslogs_region is None:
- self.awslogs_region = self.region
Review Comment:
Empty region_name is invalid region
```python
import boto3
session = boto3.session.Session(region_name="")
session.client("sts").get_caller_identity()
# ValueError: Invalid endpoint: https://sts..amazonaws.com
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shahar1 commented on pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1694495915
> > One that includes the pre-commit Python script and its YAML definition in pre-commit-config.yaml - I excluded files that should be fixed in later PR; see next.
> > Another that includes modifications to specific operators, either to align with the requirements immediately or by inserting a FIXME comment to do it later (in case the complexity seems too high to address at the moment). If the reviewers deem it necessary, I can create a separate pull request for this. Regardless, I'll create a new issue for the necessary fixes.
>
> I think it should be approached differently.. This PR should fail. It should not have exclusioins. It should only be merged when we (or others) fixed all the problems. This is what we do normally. Then this PR open should serve as reminder and incentivisation to fix all the places. So I think in general, the sequence should be reverse:
>
> 1. add PR that fails and reminds others when they cross the border
> 2. make sure (it does not have to be you - you can ping other people to do it) to fix alll the places
> 3. continue rebasing this PR seeing less and less problems as you iterate
> 4. merge it once all the problems are fixed.
>
> This is the aproach we've used for other cases that were similar.
Sounds good to me :)
I removed the exceptions, yet I kept the second commit for inspiration - I'd be happy for feedback regarding the logic that should (not) be in the constructors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] shahar1 commented on pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1694741468
> I am not sure if that's the right set of validations. I do not find loops or if/return particularly troubling. What I was really hoping for was a check that will check for "typical" errors made in operators that are not often caught during review, but they have some negative side effects.
>
> I am not sure if we want to limit operator's authors from raising warnings/running returns etc. This is far too much IMHO. We do not want to tell the authors "this is how you shoudl do your logic in constructor" - we want to make them aware when they made an obvious mistake. Generally I think we should only run validations that are very clearly violating some not even best but "reasonbly only possible" practice in Airflow.
>
> I think the rule of thumb should be "You should be able to tell the author EXACTLY how to correct their code and explain why it is wrong". And it should be a mistake that "newbie" contributors are likely to make.
>
> The issue #29069 was really about this one:
>
> * when you are using templated arg for something else than just settng it as templated field - Generally speaking the only thing you should do for templated field is you should set it as field. The reason is that this field will not have the right value before "execute" command is fired.
>
> otherwise having some logic for non-templated fields is perfectly fine. I would not want us to be a "dictactor" to tell people how to write their constructors.
>
> And there are few other things that are OBVIOUSLY wrong:
>
> * when you have templated_filed defined for operator but it is not set in the constructor as field, the field is not really templated - it will be missing when templated fields are processed. This is obviously wrong
> * when you try to instantiate Hook in the operator's constructor, that's also obviously wrong. You should never do that, because such Hook instantiation will also happen in the DAGFileProcessor when the DAG is parsed.
>
> I'd start with these three rules, rather than trying to present some constructs that are not obviously wrong.
Thanks for the detailed feedback - now I have a better sense of the issue.
Drafting this PR for now and I'll try fixing it according to these rules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent templated field logic checks in __init__ of operators automatically [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #33786:
URL: https://github.com/apache/airflow/pull/33786#issuecomment-1858814207
@potiuk @hussein-awala @Taragolis @eladkal
I tried my best to come up with assumptions that make sense and develop a useful pre-commit that will help us regulating the templated fields.
I refactored the pre-commit and added some documentation (both docstrings and RST).
There should be failing checks, as not all of the existing providers currently comply with the new pre-commit. However, it shouldn't be too hard to fix what's needed (considering that these are the only limitations).
Let me know what you think, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on a diff in pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on code in PR #33786:
URL: https://github.com/apache/airflow/pull/33786#discussion_r1320302570
##########
airflow/providers/amazon/aws/operators/ecs.py:
##########
@@ -496,15 +496,12 @@ def __init__(
self.tags = tags
self.awslogs_group = awslogs_group
self.awslogs_stream_prefix = awslogs_stream_prefix
- self.awslogs_region = awslogs_region
+ self.awslogs_region = awslogs_region or self.region
self.awslogs_fetch_interval = awslogs_fetch_interval
self.propagate_tags = propagate_tags
self.reattach = reattach
self.number_logs_exception = number_logs_exception
- if self.awslogs_region is None:
- self.awslogs_region = self.region
Review Comment:
I wonder why the condition was `is None`, should empty string be accepted?
##########
airflow/providers/amazon/aws/operators/sagemaker.py:
##########
@@ -666,6 +660,12 @@ def expand_role(self) -> None:
config["ExecutionRoleArn"] = hook.expand_role(config["ExecutionRoleArn"])
def execute(self, context: Context) -> dict:
+ if self.action_if_job_exists not in ("increment", "fail", "timestamp"):
+ raise AirflowException(
+ f"Argument action_if_job_exists accepts only 'timestamp', 'increment' and 'fail'. \
+ Provided value: '{self.action_if_job_exists}'."
+ )
+
Review Comment:
With this change, instead of warning the user in dev time to fix the param, you move it to runtime. Personally I prefer to keep it in init :thinking:
##########
airflow/providers/amazon/aws/operators/sagemaker.py:
##########
@@ -1026,6 +1021,11 @@ def _create_integer_fields(self) -> None:
]
def execute(self, context: Context) -> dict:
+ if self.action_if_job_exists not in {"timestamp", "increment", "fail"}:
+ raise AirflowException(
+ f"Argument action_if_job_exists accepts only 'timestamp', 'increment' and 'fail'. \
+ Provided value: '{self.action_if_job_exists}'."
+ )
Review Comment:
same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on a diff in pull request #33786: Prevent templated field logic checks in __init__ of operators automatically
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #33786:
URL: https://github.com/apache/airflow/pull/33786#discussion_r1320349216
##########
airflow/providers/qubole/operators/qubole.py:
##########
Review Comment:
Qubole is suspended provider.
##########
airflow/providers/amazon/aws/operators/emr.py:
##########
@@ -1011,6 +1011,7 @@ def __init__(
deferrable: bool = conf.getboolean("operators", "default_deferrable", fallback=False),
**kwargs,
):
+ # FIXME: Handle if statement(s) in execute() method to comply with validate-operators-init pre-commit
if waiter_check_interval_seconds is NOTSET:
Review Comment:
I'm sure that this not required fixed,`NOTSET` it is special sentinel which should not provided by user manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Prevent templated field logic checks in __init__ of operators automatically [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal merged PR #33786:
URL: https://github.com/apache/airflow/pull/33786
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org