You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/10/11 17:56:08 UTC
[GitHub] [airflow] potiuk opened a new issue #11425: Automate Provider versioning changes with semver
potiuk opened a new issue #11425:
URL: https://github.com/apache/airflow/issues/11425
We agreed that provider package versioning needs to be done with semver. We need to work out the way on how this automated versioning would work from those perspectives:
* contributor
* maintainer
* release manager
Short documentation about this would be great, before we implement that, so that we know what could be automated and what should be manual work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-708460612
> - [ ] decide how we release sources - all together or as separate set of sources per package
Per provider, and containing just the sources for that provider (i.e. `airflow/providers/X/*` -- essentially what we would get if doing ` setup.py sdist`) -- the whole point of separating them out is to be able to have different release cycles for different providers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-749984154
This has been spli into separate issues #13271 #13272 #13273 #13274 #13275 #13276 #13277 so I am closing this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11425: Automate Provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-706746571
> @ashb - since you seem to be the biggest proponent of the Semver proposal, I think it would be great that you describe the process or at least propose it. There is no big hurry with that, as this will only be needed when we eventually release the final 2.0 packages together with 2.0 release, but it would be great to at least think and propose how the provess will look like from those three perspectives and where we can automate what.
>
> It would be great to discuss it upfront before someone attempts to implement it. Would you be ok with leading that part ?
>
Sure! I'll start on this on Monday or Tuesday
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11425: Automate Provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-706742978
@ashb - since you seem to be the biggest proponent of the Semver proposal, I think it would be great that you describe the process or at least propose it. There is no big hurry with that, as this will only be needed when we eventually release the final 2.0 packages together with 2.0 release, but it would be great to at least think and propose how the provess will look like from those three perspectives and where we can automate what.
It would be great to discuss it upfront before someone attempts to implement it. Would you be ok with leading that part ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-708321163
Once agreed the process here we will implement it in #11427
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #11425:
URL: https://github.com/apache/airflow/issues/11425
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-749661371
> * We can use the existing version field in the provide.yaml for this can't we?
Yep. We currently have the 'versions' field which is an array of versions and we use it to keep list of available versions of the provider (we can use it to select version in the generated docs like in Airflow docs). Sorted in descending order, so we can add the right version as the first entry at the moment we prepare for release.
```
"versions": {
"description": "List of available versions in Pypi. Sorted descending according to release date.",
"type": "array",
"items": {
"type": "string"
}
},
```
> * I think for the 1.0.0 I did `providers/` (rather than `providers-`) prefix. We should be consistent, and either continue to use `/` or go back and recreate the 1.0.0. Same for upgrade-check tags.
We have to recreate it anyway because we want to have a separate tag for each provider (yeah a lot of tags). But I am also fine with keeping it as:
```
providers-google/1.0.0
providers-apache-hive/1.0.0
```
I think this is a nice convention to separate version with '/'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-749664948
And one more thing. I believe for the new versions of providers we can completely skip the list of added/moved operators in README. Those are useful for Backport providers and possibly for 1.0.0 version, but I think for all the future releases of providers the "ADDITIONAL_INFO.md" content (or renamed to UPDATING) is much more important, and the documentation we have covers the list of classes etc. much better. So we should have link to the documentation here and list of changes from ADDITIONAL_INFO.
I think list of commits in each version is useful though. The fact that you can get from the README directly to the commit and see the description of it is really nice (but there is no need to show all the commits which were there for the 1.0.0 version).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-749653446
Sounds good. Two minor comments
1. We can use the existing version field in the provide.yaml for this can't we?
2. I think for the 1.0.0 I did `providers/` (rather than `providers-`) prefix. We should be consistent, and either continue to use `/` or go back and recreate the 1.0.0. Same for upgrade-check tags.
Otherwise in total agreement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-708470439
> > * [ ] decide how we release sources - all together or as separate set of sources per package
>
> Per provider, and containing just the sources for that provider (i.e. `airflow/providers/X/*` -- essentially what we would get if doing ` setup.py sdist`) -- the whole point of separating them out is to be able to have different release cycles for different providers.
Yep. Agree this is probably the best way. It will be also easier to version the source packages then. We'd have to remove providers from Airflow sdist completely, but that's easy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-749664948
And one more thing. I believe for the new versions of providers we can completely skip the list of added/moved operators in README. Those are useful for Backport providers and possibly for 1.0.0 version, but I think for all the future releases of providers the "ADDITIONAL_INFO.md" content (or renamed to UPDATING) is much more important, and the documentation we have covers the list of classes etc. much better. So we should have link to the documentation here and list of changes.
I think list of commits in each version is useful though. The fact that you can get from the README directly to the commit and see the description of it is really nice (but there is no need to show all the commits which were there prior the 1.0.0 version).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-749647496
Hello everyone here. I wanted to propose some vision of the 'separate provider' versioning that we should implement as part of cleanup and release Google Provider (we are already planning to release 2.0.0 provider for Google as it will contain breaking changes due to non-compatible Python API 2.0 libraries. So we will have a chance to test the extreme version,
My proposal:
1) We should add 'current version' of the library to provider.yaml. This might be the top "version" name in the list of versions simply. This way we avoid duplication.
2) For every released version of provider we add a tag in the repo (currently this means 60 tags): Example: 'providers-google-1.0.0'.
3) We slightly change the way we generate release notes. We will simply get the list of commits from the `providers-google-<PREVIOUS_VERSION>` tag.
4) We remove all the generated CHANGES_* and README_* files completely and instead of keeping them in the repo, we will generate them automatically when the documentation is generated and place it as part of the 'per-provider' documentation.
5) Similarly as we currently do with Airflow's UPDATING.md, we keep the information about breaking changes in the ADDIITONAL_INFO.md (which is already there for some packages). This additional info is automatically part of README.md. We should make sure that any breaking changes or new features added are reflected in the ADDITIONAL_INFO.md during PR Review. This should be author's responsibility to describe it and reviewer's responsibility to point out that it should be updated when we see new feature/backwards incompatible changes. We should add them in "master" version and just before release we decide what is the new version and we update it in a separate commit. We can even rename ADDITIONAL_INFO to UPDATING to keep consistency or rename it to something different. I am open for proposals. This will also become a better "readable" description of what has changed between different versions.
6) We will add a tool/view/smth to show each provider together with ADDITIONAL_CHANGES and list of affecting commit in a single summary page, so that we can decide when to release which providers.
I think this is a minimal but pretty comprehensive set of things that we have to implement in order start releasing providers independently and more frequently and if we agree that this looks good, I can split it into issues and start implementing it.
Let me know what you think @kaxil @ashb @mik-laj @turbaszek @XD-DENG others interested :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-749647496
Hello everyone here. I wanted to propose some vision of the 'separate provider' versioning that we should implement as part of cleanup and release Google Provider.
We are already planning to release 2.0.0 provider for Google as it will contain breaking changes due to non-compatible Python API 2.0 libraries. So we will have a chance to test the extreme version,
My proposal:
1) We should add 'current version' of the library to provider.yaml. This might be the top "version" name in the list of versions simply. This way we avoid duplication.
2) For every released version of provider we add a tag in the repo (currently this means 60 tags): Example: 'providers-google-1.0.0'.
3) We slightly change the way we generate release notes. We will simply get the list of commits from the `providers-google-<PREVIOUS_VERSION>` tag.
4) We remove all the generated CHANGES_* and README_* files completely and instead of keeping them in the repo, we will generate them automatically when the documentation is generated and place it as part of the 'per-provider' documentation.
5) Similarly as we currently do with Airflow's UPDATING.md, we keep the information about breaking changes in the ADDIITONAL_INFO.md (which is already there for some packages). This additional info is automatically part of README.md. We should make sure that any breaking changes or new features added are reflected in the ADDITIONAL_INFO.md during PR Review. This should be author's responsibility to describe it and reviewer's responsibility to point out that it should be updated when we see new feature/backwards incompatible changes. We should add them in "master" version and just before release we decide what is the new version and we update it in a separate commit. We can even rename ADDITIONAL_INFO to UPDATING to keep consistency or rename it to something different. I am open for proposals. This will also become a better "readable" description of what has changed between different versions.
6) We will add a tool/view/smth to show each provider together with ADDITIONAL_CHANGES and list of affecting commit in a single summary page, so that we can decide when to release which providers.
I think this is a minimal but pretty comprehensive set of things that we have to implement in order start releasing providers independently and more frequently and if we agree that this looks good, I can split it into issues and start implementing it.
Let me know what you think @kaxil @ashb @mik-laj @turbaszek @XD-DENG others interested :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #11425: Describe provider versioning changes with semver
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #11425:
URL: https://github.com/apache/airflow/issues/11425#issuecomment-749664948
And one more thing. I believe for the new versions of providers we can completely skip the list of added/moved operators in README. Those are useful for Backport providers and possibly for 1.0.0 version, but I think for all the future releases of providers the "ADDITIONAL_INFO.md" content (or renamed to UPDATING) is much more important, and the documentation we have covers the list of classes etc. much better. So we should have link to the documentation here and list of changes from ADDITIONAL_INFO.
I think list of commits in each version is useful though. The fact that you can get from the README directly to the commit and see the description of it is really nice (but there is no need to show all the commits which were there prior the 1.0.0 version).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org