You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2023/10/28 14:11:52 UTC
[PR] Fix errors in common.io release documentation generation with prevention [airflow]
potiuk opened a new pull request, #35236:
URL: https://github.com/apache/airflow/pull/35236
The common.io provider release caused a few problems when preparing the new wave of the providers:
* it missed link to security.rst (which is generated automatically anticipating SBOM generation in the near future)
* it did not have proper min-version set in MIN_AIRFLOW_VERSION_EXCEPTIONS
* it had single-backtick description that resulted in generated commit and changelog code - since it failed documentation building
All those errors have not been caught at PR time. This PR fixes them and introduces prevention from similar mistakes - future providers with similar errors will get nice errors generated in CI at "docs publishing step".
* if some generated file is missing in the index user is asked to add the files in the index
* if there are `backtick` errors they will be reported and pointed at
* if there are liniting problems with generated .rst files they will be reported
* the "min airflow version" is now taken directly from provider.yaml not from exception list - but only when it is bigger than current min version. This way, we will also not forget to remove the provider from the list of exceptions when we bump min version above the exception version
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix errors in common.io release documentation generation with prevention [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35236:
URL: https://github.com/apache/airflow/pull/35236#issuecomment-1783897920
> @potiuk Thanks for the heads-up, I'll address all those issues in the LLM ops providers before merging. :)
You will have no other choice now 😁 ... Your PRs will not get green :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix errors in common.io release documentation generation with prevention [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35236:
URL: https://github.com/apache/airflow/pull/35236#issuecomment-1784118677
> @potiuk I'm not against those checks, just as someone contributing I just long for a way that my IDE catches those errors before I commit. It's just a drag to find out that you need to refactor some work, because pre-commit hooks don't allow you to commit it.
Interesting idea. We are IDE agnostic, but pre-commit is as-close as it can get to being "OS/IDE"-agnostic.
But... Your comment gave me an idea.
Taking your stuff literally, maybe a good idea will be to add a pre-commit extension that will watch for file changes (as specified in pre-commit hook configuration and will run the pre-commits in the background. Especially that many of those pre-commits are not only telling you what's wrong but also fixing it for you.
That would be a REALLY nice IDE plugin if someone would like to do it, and I think entirely possible to implement for both PyCharm. VSCode. Even automagically correcting your code as you save the files possibly. The whole idea about Pre-commit is that they should be **fast** - specifically if you keep them running a number of them (like mypy one) use the cache, but most of them should be **really** fast when you change individual files.
Let me capture it as an interesting task to implement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix errors in common.io release documentation generation with prevention [airflow]
Posted by "bolkedebruin (via GitHub)" <gi...@apache.org>.
bolkedebruin commented on PR #35236:
URL: https://github.com/apache/airflow/pull/35236#issuecomment-1784065299
@potiuk I'm not against those checks, just as someone contributing I just long for a way that my IDE catches those errors before I'd like to commit. It's just a drag to find out that you need to refactor some work, because pre-commit hooks don't allow you to commit it.
Lately, I stumbled upon the inclusive language check. Sure, makes sense but it doesn't when the API you are using uses the 'banned' words and then you need to figure what to do.
So not against it, just the experience isn't as much fun :-).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix errors in common.io release documentation generation with prevention [airflow]
Posted by "utkarsharma2 (via GitHub)" <gi...@apache.org>.
utkarsharma2 commented on PR #35236:
URL: https://github.com/apache/airflow/pull/35236#issuecomment-1783897696
@potiuk Thanks for the heads-up, I'll address all those issues in the LLM ops providers before merging. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix errors in common.io release documentation generation with prevention [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35236:
URL: https://github.com/apache/airflow/pull/35236#issuecomment-1783825435
Might also prove useful when we will be merging the 5 new ML providers as it catches a lot more potential "publishing" errors - also improves management of exceptional "min-airflow" versions - basically we do not have to remember about removing exceptions for them any more:
Those errors will be reported in the CI at the "doc publishing" verification step once this is merged.
Example erros:
Example sucesses:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix errors in common.io release documentation generation with prevention [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35236:
URL: https://github.com/apache/airflow/pull/35236#issuecomment-1783840939
Actually seems that linting found quite a few problems across multiple dosc of ours :). Those are minor issues (but worth fixing I think
I will disable it tempoarily to allow the release to proceed without those and prepare a separate PR with fixes to old docs/generation process (the problem is in templates we have)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix errors in common.io release documentation generation with prevention [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35236:
URL: https://github.com/apache/airflow/pull/35236#issuecomment-1783827561
Related #35233
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix errors in common.io release documentation generation with prevention [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35236:
URL: https://github.com/apache/airflow/pull/35236#issuecomment-1784121500
Issue created and marked as "good first issue" - possibly good issue for someone who has experience with writing such plugins. https://github.com/apache/airflow/issues/35253
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix errors in common.io release documentation generation with prevention [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #35236:
URL: https://github.com/apache/airflow/pull/35236#issuecomment-1783830793
Also cc: @bolkedebruin -> re, our discussion - this is why we keep on adding more and more checks like that and it's "difficutl" to add a new provider.
But if we do not do that, it makes release manager's work way harder when things break at the release time. While we are slowing down contributions, it makes release management WAY simpler (and on top of that we are quite certain (barring some new exceptional cases like that) that when release manager attempts to release the software it will **just** work ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix errors in common.io release documentation generation with prevention [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #35236:
URL: https://github.com/apache/airflow/pull/35236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org