You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "aritra24 (via GitHub)" <gi...@apache.org> on 2025/04/08 07:21:31 UTC
[PR] Add airflowctl codegen [airflow]
aritra24 opened a new pull request, #48914:
URL: https://github.com/apache/airflow/pull/48914
closes: [#47509](https://github.com/apache/airflow/issues/47509)
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2044659211
##########
airflow-ctl/pyproject.toml:
##########
@@ -127,7 +122,7 @@ use-standard-collections=true # list[] not List[]
use-subclass-enum=true # enum, not union of Literals
use-union-operator=true # 3.9+annotations, not `Union[]`
-url = "http://0.0.0.0:28080/openapi.json"
+input = "../airflow-core/src/airflow/api_fastapi/core_api/openapi/v1-generated.yaml"
Review Comment:
I just renamed that one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2041197302
##########
task-sdk/dev/generate_task_sdk_models.py:
##########
@@ -40,7 +40,6 @@
sys.path.insert(0, AIRFLOW_CORE_SOURCES_PATH.as_posix())
sys.path.insert(0, str(AIRFLOW_ROOT_PATH)) # make sure setup is imported from Airflow
-
Review Comment:
Is this intentional? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2801281386
> Regarding the above problem. Maybe we should consider supporting 3.10 from the beginning of airflowctl.
Fine for 3.10+ only
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl merged PR #48914:
URL: https://github.com/apache/airflow/pull/48914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2041167489
##########
airflow-ctl/src/airflowctl/api/datamodels/generated.py:
##########
@@ -136,7 +129,7 @@ class BulkDeleteActionConnectionBody(BaseModel):
entities: Annotated[
list[str], Field(description="A list of entity id/key to be deleted.", title="Entities")
]
- action_on_non_existence: BulkActionNotOnExistence | None = "fail"
+ action_on_non_existence: BulkActionNotOnExistence | None = BulkActionNotOnExistence.FAIL
Review Comment:
Why? Such modificatiosn of generated code are very, very prone to future errors - codegen generation might change any time and someone will have to dig it out and find out how to modify the resuling code to fix the issue - and since that will happen randomly and usually will break someone's PR, that might be super confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2037217728
##########
airflow-ctl/src/airflowctl/api/datamodels/generated.py:
##########
@@ -136,7 +129,7 @@ class BulkDeleteActionConnectionBody(BaseModel):
entities: Annotated[
list[str], Field(description="A list of entity id/key to be deleted.", title="Entities")
]
- action_on_non_existence: BulkActionNotOnExistence | None = "fail"
+ action_on_non_existence: BulkActionNotOnExistence | None = BulkActionNotOnExistence.FAIL
Review Comment:
Seems like it. I think we can exclude it for now since it is not a thing we are implementing and can find a automated way as a next step
##########
.pre-commit-config.yaml:
##########
@@ -1345,6 +1345,13 @@ repos:
pass_filenames: false
files: ^airflow-core/src/airflow/api_fastapi/execution_api/.*\.py$
require_serial: true
+ - id: generate-airflowctl-datamodels
+ name: Generate Datamodels for AirflowCTL
+ language: python
+ entry: uv run -p 3.12 --no-dev --no-progress --active --group codegen --project apache-airflow-ctl --directory airflow-ctl/ datamodel-codegen
+ pass_filenames: false
+ files: ^airflow-core/src/airflow/api_fastapi/execution_api/.*\.py$
Review Comment:
```suggestion
files: ^airflow-core/src/airflow/api_fastapi/core_api/.*\.py$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2806916687
> The license is auto added by one of the pre-commit rules... So I think that's fine 🤔 But also need to ignore the generated files from the manual mypy run since that's what we do in the ci.
>
> @potiuk I didn't want to add a blanket ignore of the `*generated.py` suffix in `mypy_folder.py` in case it unintentionally skips checking one of the other files. What are your thoughts?
I have excluded it from MyPy here https://github.com/apache/airflow/pull/49250/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R1192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2807629582
Unfortunately I'm without a laptop for the next 24 hrs. I can push the change post that 😕
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2805076261
@bugraoz93 / @potiuk do you think this is good now or any concerns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2785486733
I wasn't sure if there was a template file for the license which I could use for custom header instead of putting the license in directly. Also one drawback is it loses the generated-by codegen line. I don't think there's an option to append the header instead of replacing 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2041192270
##########
airflow-ctl/pyproject.toml:
##########
@@ -126,8 +127,11 @@ use-schema-description=true # Desc becomes class doc comment
use-standard-collections=true # list[] not List[]
use-subclass-enum=true # enum, not union of Literals
use-union-operator=true # 3.9+annotations, not `Union[]`
+custom-formatters = 'misc.simple_code_formatter'
Review Comment:
Let's update the comments above, too if anyone wants to run manually in local
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2800080669
> I think it's ready for review. Need to fix up the mypy failure still. I manually added the ignore assignment. Will look into that tomorrow, but the rest should be good I believe.
We should also include automation `auth_generated` for our login models.
https://github.com/apache/airflow/blob/0ca0f17996c86efb292cf5b10181944c67e3b862/airflow-ctl/pyproject.toml#L112
https://github.com/apache/airflow/blob/main/airflow-ctl/src/airflowctl/api/datamodels/auth_generated.py
>@bugraoz93 I used the airflow-core/src/airflow/api_fastapi/core_api/openapi/v1-generated.yaml to not have to start the server just to do the codegen? But I guess it's creating a bunch more models than it should? I'm not too sure but my assumption was the generated yaml file should be the same api exposed by the server and thus we should've gotten the same output? Also this is causing some tests to fail which also confuses me a bit. 🤔 Do you have any thoughts?
It seems like Python 3.9 doesn't like the annotation, but we have `from __future__ ...` there. Checking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2044736841
##########
airflow-ctl/pyproject.toml:
##########
@@ -127,7 +122,7 @@ use-standard-collections=true # list[] not List[]
use-subclass-enum=true # enum, not union of Literals
use-union-operator=true # 3.9+annotations, not `Union[]`
-url = "http://0.0.0.0:28080/openapi.json"
+input = "../airflow-core/src/airflow/api_fastapi/core_api/openapi/v1-generated.yaml"
Review Comment:
oops, I'll update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2044550398
##########
airflow-ctl/pyproject.toml:
##########
@@ -20,7 +20,7 @@ name = "apache-airflow-ctl"
dynamic = ["version"]
description = "Apache Airflow command line tool for communicating with an Apache Airflow, using the API."
readme = { file = "README.md", content-type = "text/markdown" }
-requires-python = ">=3.9, <3.13"
+requires-python = ">=3.10, <3.13"
Review Comment:
removed this
##########
.pre-commit-config.yaml:
##########
@@ -1357,6 +1357,13 @@ repos:
pass_filenames: false
files: ^airflow-core/src/airflow/api_fastapi/execution_api/.*\.py$
require_serial: true
+ - id: generate-airflowctl-datamodels
+ name: Generate Datamodels for AirflowCTL
+ language: python
+ entry: uv run -p 3.12 --no-dev --no-progress --active --group codegen --project apache-airflow-ctl --directory airflow-ctl/ datamodel-codegen
+ pass_filenames: false
+ files: ^airflow-core/src/airflow/api_fastapi/core_api/.*\.py$
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2785499365
Pushed the generated file in a separate commit, which will fail the ci due to reverted default values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2037217728
##########
airflow-ctl/src/airflowctl/api/datamodels/generated.py:
##########
@@ -136,7 +129,7 @@ class BulkDeleteActionConnectionBody(BaseModel):
entities: Annotated[
list[str], Field(description="A list of entity id/key to be deleted.", title="Entities")
]
- action_on_non_existence: BulkActionNotOnExistence | None = "fail"
+ action_on_non_existence: BulkActionNotOnExistence | None = BulkActionNotOnExistence.FAIL
Review Comment:
Seems like it. I think we can exclude it for now since it is not a thing we are implementing and can find an automated way as a next step
Edit: exclude from mypy checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2041143762
##########
airflow-ctl/src/airflowctl/api/datamodels/generated.py:
##########
@@ -136,7 +129,7 @@ class BulkDeleteActionConnectionBody(BaseModel):
entities: Annotated[
list[str], Field(description="A list of entity id/key to be deleted.", title="Entities")
]
- action_on_non_existence: BulkActionNotOnExistence | None = "fail"
+ action_on_non_existence: BulkActionNotOnExistence | None = BulkActionNotOnExistence.FAIL
Review Comment:
Oh... 🥲. I think we do need the script anyways though cos we need to add the mypy ignore line so might as well also add the license? What's your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2807527900
Yeah. Should be excluded in pre-commit config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2799867817
@bugraoz93 I used the `airflow-core/src/airflow/api_fastapi/core_api/openapi/v1-generated.yaml` to not have to start the server just to do the codegen? But I guess it's creating a bunch more models than it should? I'm not too sure but my assumption was the generated yaml file should be the same api exposed by the server and thus we should've gotten the same output? Also this is causing some tests to fail which also confuses me a bit. 🤔 Do you have any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2042561972
##########
task-sdk/dev/generate_task_sdk_models.py:
##########
@@ -40,7 +40,6 @@
sys.path.insert(0, AIRFLOW_CORE_SOURCES_PATH.as_posix())
sys.path.insert(0, str(AIRFLOW_ROOT_PATH)) # make sure setup is imported from Airflow
-
Review Comment:
Oops :p
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2042900248
##########
airflow-ctl/pyproject.toml:
##########
@@ -20,7 +20,7 @@ name = "apache-airflow-ctl"
dynamic = ["version"]
description = "Apache Airflow command line tool for communicating with an Apache Airflow, using the API."
readme = { file = "README.md", content-type = "text/markdown" }
-requires-python = ">=3.9, <3.13"
+requires-python = ">=3.10, <3.13"
Review Comment:
We need to do more than change this to lose support for `3.9`, so let's keep the `3.10+` PR separate. I created a PR for fixing the issue with ExtraLinks and excluding the entire package from MyPY for some Enum conversions. Let's rebase after that one is merged.
#49250
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2807464560
Static check failed on license comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2037601265
##########
.pre-commit-config.yaml:
##########
@@ -1345,6 +1345,13 @@ repos:
pass_filenames: false
files: ^airflow-core/src/airflow/api_fastapi/execution_api/.*\.py$
require_serial: true
+ - id: generate-airflowctl-datamodels
+ name: Generate Datamodels for AirflowCTL
+ language: python
+ entry: uv run -p 3.12 --no-dev --no-progress --active --group codegen --project apache-airflow-ctl --directory airflow-ctl/ datamodel-codegen
+ pass_filenames: false
+ files: ^airflow-core/src/airflow/api_fastapi/execution_api/.*\.py$
Review Comment:
Ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2041167132
##########
airflow-ctl/src/airflowctl/api/datamodels/generated.py:
##########
@@ -136,7 +129,7 @@ class BulkDeleteActionConnectionBody(BaseModel):
entities: Annotated[
list[str], Field(description="A list of entity id/key to be deleted.", title="Entities")
]
- action_on_non_existence: BulkActionNotOnExistence | None = "fail"
+ action_on_non_existence: BulkActionNotOnExistence | None = BulkActionNotOnExistence.FAIL
Review Comment:
We can just ignore all mypy errors comingfrom this package - we do it in few other generated packages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2800112818
> > I think it's ready for review. Need to fix up the mypy failure still. I manually added the ignore assignment. Will look into that tomorrow, but the rest should be good I believe.
>
> We should also include automation `auth_generated` for our login models.
>
> https://github.com/apache/airflow/blob/0ca0f17996c86efb292cf5b10181944c67e3b862/airflow-ctl/pyproject.toml#L112
>
>
> https://github.com/apache/airflow/blob/main/airflow-ctl/src/airflowctl/api/datamodels/auth_generated.py
> > @bugraoz93 I used the airflow-core/src/airflow/api_fastapi/core_api/openapi/v1-generated.yaml to not have to start the server just to do the codegen? But I guess it's creating a bunch more models than it should? I'm not too sure but my assumption was the generated yaml file should be the same api exposed by the server and thus we should've gotten the same output? Also this is causing some tests to fail which also confuses me a bit. 🤔 Do you have any thoughts?
>
> It seems like Python 3.9 doesn't like the annotation, but we have `from __future__ ...` there. Checking.
@aritra24 This part `class ExtraLinksResponse(RootModel[dict[str, str | None] | None]):` threws.
``` Python
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
```
This occurs because the syntax str | None is not supported in Python 3.9. This union operator (|) for type hints was introduced in Python 3.10. Although from __future__ import annotations postpones the evaluation of type annotations, it doesn't enable the use of the | operator for type unions in earlier Python versions.​ Since we are calling the class in tests, it throws the error.
We can exclude some models in Custom Formatter but that could cause some models to be missing and want to be used in aitflowctl. So we can also find out why this model is coming from `RootModel` and not from `from airflow.api_fastapi.core_api.base import BaseModel` :thinking: Because if it would use BaseModel, we won't get this problem at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2032566140
##########
airflow-ctl/src/airflowctl/api/datamodels/generated.py:
##########
@@ -136,7 +129,7 @@ class BulkDeleteActionConnectionBody(BaseModel):
entities: Annotated[
list[str], Field(description="A list of entity id/key to be deleted.", title="Entities")
]
- action_on_non_existence: BulkActionNotOnExistence | None = "fail"
+ action_on_non_existence: BulkActionNotOnExistence | None = BulkActionNotOnExistence.FAIL
Review Comment:
I made these changes by hand, I wasn't sure why the mypy was failing, I guess because `# mypy: disable-error-code="assignment"` this line was removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2041201308
##########
airflow-ctl/pyproject.toml:
##########
@@ -126,8 +127,11 @@ use-schema-description=true # Desc becomes class doc comment
use-standard-collections=true # list[] not List[]
use-subclass-enum=true # enum, not union of Literals
use-union-operator=true # 3.9+annotations, not `Union[]`
+custom-formatters = 'misc.simple_code_formatter'
Review Comment:
Also there is a TODO we can also delete that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2044550006
##########
.pre-commit-config.yaml:
##########
@@ -1357,6 +1357,13 @@ repos:
pass_filenames: false
files: ^airflow-core/src/airflow/api_fastapi/execution_api/.*\.py$
require_serial: true
+ - id: generate-airflowctl-datamodels
+ name: Generate Datamodels for AirflowCTL
+ language: python
+ entry: uv run -p 3.12 --no-dev --no-progress --active --group codegen --project apache-airflow-ctl --directory airflow-ctl/ datamodel-codegen
+ pass_filenames: false
+ files: ^airflow-core/src/airflow/api_fastapi/core_api/.*\.py$
+ require_serial: true
Review Comment:
Added under the same pre-commit step
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2805988746
The license is auto added by one of the pre-commit rules... So I think that's fine 🤔 But also need to ignore the generated files from the manual mypy run since that's what we do in the ci.
@potiuk I didn't want to add a blanket ignore of the `*generated.py` suffix in `mypy_folder.py` in case it unintentionally skips checking one of the other files. What are your thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2807395697
merged main, should have resolved all the conflicts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2794737017
I think it's ready for review. Need to fix up the mypy failure still. I manually added the ignore assignment. Will look into that tomorrow, but the rest should be good I believe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2041167182
##########
airflow-ctl/src/airflowctl/api/datamodels/generated.py:
##########
@@ -136,7 +129,7 @@ class BulkDeleteActionConnectionBody(BaseModel):
entities: Annotated[
list[str], Field(description="A list of entity id/key to be deleted.", title="Entities")
]
- action_on_non_existence: BulkActionNotOnExistence | None = "fail"
+ action_on_non_existence: BulkActionNotOnExistence | None = BulkActionNotOnExistence.FAIL
Review Comment:
See pyproject.toml `ignore_errors`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2041179655
##########
airflow-ctl/src/airflowctl/api/datamodels/generated.py:
##########
@@ -136,7 +129,7 @@ class BulkDeleteActionConnectionBody(BaseModel):
entities: Annotated[
list[str], Field(description="A list of entity id/key to be deleted.", title="Entities")
]
- action_on_non_existence: BulkActionNotOnExistence | None = "fail"
+ action_on_non_existence: BulkActionNotOnExistence | None = BulkActionNotOnExistence.FAIL
Review Comment:
@potiuk Thanks! I am okay with excluding the license. I saw we have one in `task-sdk`. That's why I was thinking maybe we can do something similar here.
@aritra24, I agree with Jarek that this package will break at some point without us making any changes. Let's use `ignore_errors` for this package and ignore everything rather than chasing and fixing specific rules. There won't be any code piece under this package other than generated Pydantic objects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2807357914
Merged the PR. Small conflict to solve :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2043298810
##########
airflow-ctl/pyproject.toml:
##########
@@ -20,7 +20,7 @@ name = "apache-airflow-ctl"
dynamic = ["version"]
description = "Apache Airflow command line tool for communicating with an Apache Airflow, using the API."
readme = { file = "README.md", content-type = "text/markdown" }
-requires-python = ">=3.9, <3.13"
+requires-python = ">=3.10, <3.13"
Review Comment:
Ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2041110765
##########
airflow-ctl/src/airflowctl/api/datamodels/generated.py:
##########
@@ -136,7 +129,7 @@ class BulkDeleteActionConnectionBody(BaseModel):
entities: Annotated[
list[str], Field(description="A list of entity id/key to be deleted.", title="Entities")
]
- action_on_non_existence: BulkActionNotOnExistence | None = "fail"
+ action_on_non_existence: BulkActionNotOnExistence | None = BulkActionNotOnExistence.FAIL
Review Comment:
@aritra24 @bugraoz93 -> it's perfectly fine to have no licence in generated code.
We just have to exclude such generated files in .rat-excludes and in the pre-commits that are adding licence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2041187106
##########
airflow-ctl/src/airflowctl/api/datamodels/generated.py:
##########
@@ -136,7 +129,7 @@ class BulkDeleteActionConnectionBody(BaseModel):
entities: Annotated[
list[str], Field(description="A list of entity id/key to be deleted.", title="Entities")
]
- action_on_non_existence: BulkActionNotOnExistence | None = "fail"
+ action_on_non_existence: BulkActionNotOnExistence | None = BulkActionNotOnExistence.FAIL
Review Comment:
Ack, that's fair. I'll update it to just use the regular code gen. Also if you have any thoughts on the usage of the yaml file as input instead of the api server let me know. 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2800126988
>@aritra24 This part class ExtraLinksResponse(RootModel[dict[str, str | None] | None]): threws.
>
>TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'
This occurs because the syntax str | None is not supported in Python 3.9. This union operator (|) for type hints was introduced in Python 3.10. Although from future import annotations postpones the evaluation of type annotations, it doesn't enable the use of the | operator for type unions in earlier Python versions.​ Since we are calling the class in tests, it throws the error.
>
>We can exclude some models in Custom Formatter but that could cause some models to be missing and want to be used in aitflowctl. So we can also find out why this model is coming from RootModel and not from from airflow.api_fastapi.core_api.base import BaseModel 🤔 Because if it would use BaseModel, we won't get this problem at all.
>
>Edit:
Found the answer: https://github.com/apache/airflow/pull/44277/files#r1856765103
Regarding the above problem. Maybe we should consider supporting 3.10 from the beginning of `airflowctl`.
@potiuk @kaxil @jedcunningham What do you think? There is a reason why we aren't using BaseModel and are using RootModel in the ExtraLinks. So, we can bump into similar issues in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "aritra24 (via GitHub)" <gi...@apache.org>.
aritra24 commented on PR #48914:
URL: https://github.com/apache/airflow/pull/48914#issuecomment-2815671912
@potiuk @bugraoz93 resolved all the ci failures 🎉 let me know if any other changes you think is needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add airflowctl codegen [airflow]
Posted by "bugraoz93 (via GitHub)" <gi...@apache.org>.
bugraoz93 commented on code in PR #48914:
URL: https://github.com/apache/airflow/pull/48914#discussion_r2042834351
##########
.pre-commit-config.yaml:
##########
@@ -1357,6 +1357,13 @@ repos:
pass_filenames: false
files: ^airflow-core/src/airflow/api_fastapi/execution_api/.*\.py$
require_serial: true
+ - id: generate-airflowctl-datamodels
+ name: Generate Datamodels for AirflowCTL
+ language: python
+ entry: uv run -p 3.12 --no-dev --no-progress --active --group codegen --project apache-airflow-ctl --directory airflow-ctl/ datamodel-codegen
+ pass_filenames: false
+ files: ^airflow-core/src/airflow/api_fastapi/core_api/.*\.py$
+ require_serial: true
Review Comment:
Could you please add the same things for auth datamodels?
```suggestion
require_serial: true
- id: generate-airflowctl-auth-datamodels
name: Generate Datamodels for AirflowCTL
language: python
entry: uv run -p 3.12 --no-dev --no-progress --active --group codegen --project apache-airflow-ctl --directory airflow-ctl/ datamodel-codegen --input=../airflow-core/src/airflow/api_fastapi/auth/managers/simple/openapi/v1-generated.yaml --output=src/airflowctl/api/datamodels/auth_generated.py
pass_filenames: false
files: ^airflow-core/src/airflow/api_fastapi/auth/managers/simple/datamodels/.*\.py$
require_serial: true
```
##########
.pre-commit-config.yaml:
##########
@@ -1357,6 +1357,13 @@ repos:
pass_filenames: false
files: ^airflow-core/src/airflow/api_fastapi/execution_api/.*\.py$
require_serial: true
+ - id: generate-airflowctl-datamodels
+ name: Generate Datamodels for AirflowCTL
+ language: python
+ entry: uv run -p 3.12 --no-dev --no-progress --active --group codegen --project apache-airflow-ctl --directory airflow-ctl/ datamodel-codegen
+ pass_filenames: false
+ files: ^airflow-core/src/airflow/api_fastapi/core_api/.*\.py$
Review Comment:
```suggestion
files: ^airflow-core/src/airflow/api_fastapi/core_api/datamodels/.*\.py$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org