You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "potiuk (via GitHub)" <gi...@apache.org> on 2025/11/10 11:24:09 UTC
[PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
potiuk opened a new pull request, #58138:
URL: https://github.com/apache/airflow/pull/58138
According to https://www.apache.org/legal/src-headers.html#faq-binaries all our released source packages and binaries should have both LICENCE and NOTICE files - even if we do not have any 3rd-party software embedded.
This PR updates it accross the board and makes sure that all our distributions contain both LICENCE and NOTICE file and that they are added as "licence" information to all our distribution packages when they are built.
Inspired by #58131
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3512670414
Random failure. Merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511357545
Unfortunately - the https://www.apache.org/legal/src-headers.html#faq-binaries is very clear that the licence should be in the produced binaries. I think it's a serious legal requirement, and pyspark is not doing it **right** . And yes for me this is always kinda "stupid" that in all Android apps (.apk files) not only you have licences for all the libraries included extracted from those binaries but also there is this completely unparseable "Licences" in "About" - that you should be able to see as a user. This is legal stuff and a little of the "red-tape" we have here.
small comment - as ASF security committee - we have 5-10 reports a day called "licence confusion" when someone complains that Apache had taken over their phone and doing nasty things - as a proof showing screenshots from that licence screen in their apps. 😱
> This also means I'd prefer it not symlinked either.
Yeah that was my second thought. The problem is that IF we want it dynamically added, then as far as I can tell - we cannot use a PEP-standard backend any more - without anything that wraps around and creating this file. And this is the simplest build backend. And we **cannot** use anything outside of the "source" tree of the project.
I tried various things and I could not make it work otherwise -according to the PEP standards for pyproject.toml
1. Tried parent inclusion
```
license-files = ['../../LICENSE', '../../NOTICE']
```
This is the error that flit reports.
```
Config error: Invalid glob pattern for [project.license-files]: '../../LICENSE'. Pattern must not contain '..'
```
This is in-line with the expectation of pyproject.toml "glob patterns" specification:
https://packaging.python.org/en/latest/specifications/glob-patterns/
> Parent directory indicators (..) MUST NOT be used.
2. Tried to remove the files and see what happens when they are dynamically generated:
```
Config error: No files found for [project.license-files]: 'LICENSE' pattern
```
This is in-line with the same glob-patterns definition:
> MUST raise an error if any individual user-specified pattern does not match at least one file.
3. We could probably try to have build backend to generate those files dynamically - but this complicates A LOT the build process for providers and it basically means that either we have to use some 3rd-party hatchling plugins (and switch to those). And I think this is far too "complex" to make it really part of our build. If we don't have this plugin, our users will stop having the possibility of building our providers by just `python -m build .` - or for example will not be able to install providers via Git URL - probably some other things will be more "complex" as well.
I honestly have no other ideas how to do it to be compliant with the standards.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3515912648
https://github.com/apache/airflow/pull/58191
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511385003
> Ah yes, if we can't include `../LICENSE` then I would prefer symlinking over a copy.
Yeah - the whole `pyproject.toml` standard is designed in the way (and I think it's very correct) to assume that you need only whatever you have "below" but no assumptions about what you have in the "parent" directories.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511388055
Let me try with symlinks :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511379002
Ah yes, if we can't include `../LICENSE` then I would prefer symlinking over a copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58138:
URL: https://github.com/apache/airflow/pull/58138#discussion_r2510862255
##########
airflow-core/pyproject.toml:
##########
@@ -34,7 +34,9 @@ build-backend = "hatchling.build"
name = "apache-airflow-core"
description = "Core packages for Apache Airflow, schedule and API server"
readme = { file = "README.md", content-type = "text/markdown" }
-license-files.globs = ["LICENSE", "3rd-party-licenses/*.txt", "NOTICE"]
+license-expression = "Apache-2.0"
+license = "Apache-2.0"
Review Comment:
Ah... yeah.. my bad ... I **tried** it and left it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #58138:
URL: https://github.com/apache/airflow/pull/58138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511429753
Just my view -- happy to defer to others on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #58138:
URL: https://github.com/apache/airflow/pull/58138#discussion_r2510872651
##########
airflow-core/pyproject.toml:
##########
@@ -34,7 +34,9 @@ build-backend = "hatchling.build"
name = "apache-airflow-core"
description = "Core packages for Apache Airflow, schedule and API server"
readme = { file = "README.md", content-type = "text/markdown" }
-license-files.globs = ["LICENSE", "3rd-party-licenses/*.txt", "NOTICE"]
+license-expression = "Apache-2.0"
+license = "Apache-2.0"
Review Comment:
Yeah all good also with hatch after removing it:
```
Metadata-Version: 2.4
Name: apache-airflow-core
Version: 3.2.0
Summary: Core packages for Apache Airflow, schedule and API server
Project-URL: Bug Tracker, https://github.com/apache/airflow/issues
Project-URL: Documentation, https://airflow.apache.org/docs/
Project-URL: Downloads, https://archive.apache.org/dist/airflow/
Project-URL: Homepage, https://airflow.apache.org/
Project-URL: Release Notes, https://airflow.apache.org/docs/apache-airflow/stable/release_notes.html
Project-URL: Slack Chat, https://s.apache.org/airflow-slack
Project-URL: Source Code, https://github.com/apache/airflow
Project-URL: LinkedIn, https://www.linkedin.com/company/apache-airflow/
Project-URL: Mastodon, https://fosstodon.org/@airflow
Project-URL: Bluesky, https://bsky.app/profile/apache-airflow.bsky.social
Project-URL: YouTube, https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA/
Author-email: Apache Software Foundation <de...@airflow.apache.org>
Maintainer-email: Apache Software Foundation <de...@airflow.apache.org>
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58138:
URL: https://github.com/apache/airflow/pull/58138#discussion_r2510169562
##########
airflow-ctl/NOTICE:
##########
Review Comment:
Do we really need a notice file that is "empty"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511572569
Doing some cleanups still.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58138:
URL: https://github.com/apache/airflow/pull/58138#discussion_r2510838574
##########
airflow-core/pyproject.toml:
##########
@@ -34,7 +34,9 @@ build-backend = "hatchling.build"
name = "apache-airflow-core"
description = "Core packages for Apache Airflow, schedule and API server"
readme = { file = "README.md", content-type = "text/markdown" }
-license-files.globs = ["LICENSE", "3rd-party-licenses/*.txt", "NOTICE"]
+license-expression = "Apache-2.0"
+license = "Apache-2.0"
Review Comment:
I thought we didn't need both? Or is this the difference between flit and hatchling?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3515912456
Hah.. I looked again at that PR - and I THOUGHT I also deleted the deeply nested LICENCE files (under "src/airflow" in this PR - apparently it was there at some point of time when I did it - and the PR was much smaller overall - just moving files. But apparently when I rebased it, this got lost and the files are still there. This one delets those old files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511549823
Unfortunately - could not make symlinks to work :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511107660
-1 (non-blocking) to duplicating it.
My personal view is that these should be copied in a dist preparation time -- it's much clearer for users when browsing the repo to find everything at the top level rather "oh each subproject has it's own license, now I have to check each one individually" where as seeing just a single license file would be much less cognitive load -- and closer to the truth, that everything in the repo _must_ be the same license. This also means I'd prefer it not symlinked either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3515874314
> Sorry, late to the "party" and if symlinks are hard and relative parent folders not workin... the positive side is that in the GIT backend identical files are de-duplicated, just they bloat up during checkout...
Yeah. I am not too worried about those repeated licence NOTICE files - they are lying around. Due to git deduplication it's even far less of a problem than the same licence header repeated in **every single source file** :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511551560
But I cleaned LICENCE files that were already present where they were not really necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "ashb (via GitHub)" <gi...@apache.org>.
ashb commented on code in PR #58138:
URL: https://github.com/apache/airflow/pull/58138#discussion_r2510169562
##########
airflow-ctl/NOTICE:
##########
Review Comment:
Do we really need a notice file that is "empty"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3512674645
### Backport successfully created: v3-1-test
<table>
<tr>
<th>Status</th>
<th>Branch</th>
<th>Result</th>
</tr>
<tr>
<td>✅</td>
<td>v3-1-test</td>
<td><a href="https://github.com/apache/airflow/pull/58150"><img src="https://img.shields.io/badge/PR-58150-blue" alt="PR Link"></a></td>
</tr>
</table>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3514361680
Sorry, late to the "party" and if symlinks are hard and relative parent folders not workin... the positive side is that in the GIT backend identical files are de-duplicated, just they bloat up during checkout...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511024421
Note! Potentially we could have use symbolic links or dynamically copying the LICENCE and NOTICE files. That seems like a LOT of duplication.
However, if we want to have the providers to be "truly" standalone and posssibly later to separate them out - we should not use any "common" files from "airflow" in the "build configuration" of the providers. The licence and NOTCE files are VERY unlikely to be changed (except a global search/replace for 2025) - so there is no **real** need for DRY here. In this case WET (Write Everything Twice) seems to be better approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Convert all airflow distributions to be compliant with ASF requirements [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #58138:
URL: https://github.com/apache/airflow/pull/58138#issuecomment-3511361182
I honestly think either copying or symlinking those is the only "feasible" way. And I am good with either approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org